Overview
I have approximately 5000 unit tests across ~100 submodules, running using JUnit Jupiter 6. The tests are configured for JUnit to run them in parallel, which it does using the ForkJoinPool. The tests are well-behaved, only writing to a RAM-backed tmpfs on per-test temporary directories in /tmp, and use dependency injection rather than static mocks, so they don't benefit much from the isolation provided by forking.
The service modules use Mockito for mocking. Mockito takes approximately 2 seconds to initialize, per test fork. With threading, this is manageable, but testForked is likely paying that cost multiple times.
Surprisingly, the default testParallelism provided by Mill actually slows down my test suite even on a domain model package.
Looking at the profiling results, it looks like with testParallelism=true, Mill parallelizes across forked JVMs rather than threads. While this is great for isolation, my tests don't need that level of isolation, so I seem to be paying a startup cost for no gain.
If this is not already the case, I'd love to be able to parallelize my tests using threads rather than forks.
JUnit Jupiter Configuration
All unit test modules have the following configuration, which enables JUnit to run both methods and classes in parallel, using the same number of threads as CPU cores:
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.config.dynamic.factor = 1
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = concurrent
Benchmarks
I ran some benchmarks to compare build systems, following the same protocol (5 runs to warm; median time of 3 runs) as https://mill-build.org/mill/comparisons/performance.html.
Tools
Processor: Intel Core i7-12800HX (24 cores)
Java: OpenJDK 25
Maven
Version: 3.9.9
Command: mvn test -T24
Mill
Version: 1.1.5
Command: ./mill <submodule>.test or ./mill __.test
Gradle
Version: 9.4.0
Command ./gradlew :<submodule>:test --rerun and ./gradlew test --rerun
Mill takes 5+ runs to warm the JVM and get full speed with testParallelism enabled, and is about half as fast until then. Gradle reaches full speed after 1 run, and is only about 50% slower on the first run.
Modules
Domain model module: 1266 unit tests, no Mockito usage.
Service module: 474 tests, Mockito used
Results
| Build target |
Maven |
Mill (-j24, testParallelism=false) |
Mill (-j24 testParallelism=true |
Gradle |
| Domain model module |
4.690s |
7.493s |
8.879s |
1.652s |
| One module (uses Mockito) |
6.326s |
6.431s |
10.413s |
3.686s |
| Full project |
93s |
33.535s |
44.748s |
24.943s |
While most of the test samples were quite close, Mill with testParallelism=true on the full project test run varied widely, from 41.924s to 61s.
The gradle numbers are similar to what I see in IntelliJ, running all tests in a submodule.
I also tried using testLocal with testParallelism=true. Unfortunately, my domain model module ran into an error with this mode (I'll raise another issue once I can isolate it), so I only have numbers for the service module, which ran in 6.731s.
Overview
I have approximately 5000 unit tests across ~100 submodules, running using JUnit Jupiter 6. The tests are configured for JUnit to run them in parallel, which it does using the ForkJoinPool. The tests are well-behaved, only writing to a RAM-backed tmpfs on per-test temporary directories in
/tmp, and use dependency injection rather than static mocks, so they don't benefit much from the isolation provided by forking.The service modules use Mockito for mocking. Mockito takes approximately 2 seconds to initialize, per test fork. With threading, this is manageable, but
testForkedis likely paying that cost multiple times.Surprisingly, the default
testParallelismprovided by Mill actually slows down my test suite even on a domain model package.Looking at the profiling results, it looks like with
testParallelism=true, Mill parallelizes across forked JVMs rather than threads. While this is great for isolation, my tests don't need that level of isolation, so I seem to be paying a startup cost for no gain.If this is not already the case, I'd love to be able to parallelize my tests using threads rather than forks.
JUnit Jupiter Configuration
All unit test modules have the following configuration, which enables JUnit to run both methods and classes in parallel, using the same number of threads as CPU cores:
Benchmarks
I ran some benchmarks to compare build systems, following the same protocol (5 runs to warm; median time of 3 runs) as https://mill-build.org/mill/comparisons/performance.html.
Tools
Processor: Intel Core i7-12800HX (24 cores)
Java: OpenJDK 25
Maven
Version: 3.9.9
Command:
mvn test -T24Mill
Version: 1.1.5
Command:
./mill <submodule>.testor./mill __.testGradle
Version: 9.4.0
Command
./gradlew :<submodule>:test --rerunand./gradlew test --rerunMill takes 5+ runs to warm the JVM and get full speed with
testParallelismenabled, and is about half as fast until then. Gradle reaches full speed after 1 run, and is only about 50% slower on the first run.Modules
Domain model module: 1266 unit tests, no Mockito usage.
Service module: 474 tests, Mockito used
Results
While most of the test samples were quite close, Mill with
testParallelism=trueon the full project test run varied widely, from 41.924s to 61s.The gradle numbers are similar to what I see in IntelliJ, running all tests in a submodule.
I also tried using
testLocalwithtestParallelism=true. Unfortunately, my domain model module ran into an error with this mode (I'll raise another issue once I can isolate it), so I only have numbers for the service module, which ran in 6.731s.