Comparing Parallel Test Execution in JUnit 5, Gradle, and Maven

JUnit 5, Gradle, and Maven: All offer parallel execution of unit tests. With so many options, I wondered whether it makes a difference what I pick. It does.

Capabilities of the Contenders

JUnit 5

Parallel test execution is an experimental feature of JUnit 5 that has been available since version 5.3. Once enabled, you can define whether you want to run top-level test classes or methods sequentially, concurrently or any combination thereof. JUnit 5 runs all tests in a single JVM using a ForkJoinPool. There are various knobs to control how many tests should be run in parallel.

Gradle

Gradle can execute tests in parallel, too. It works with JUnit 5, JUnit 4, TestNG and probably other frameworks. maxParallelForks is used to enable (or disable) parallel test execution and control how many tests should be run in parallel. Any value bigger than 1 enables test parallelism. Compared to the parallel test execution built into JUnit 5, Gradle can only run test classes in parallel, not individual test methods. Each test class is assigned to a single worker process (a forked JVM). In other words: Gradle runs all test methods of a single test class sequentially in the same JVM that is not shared with any other test.

tasks.withType(Test).configureEach {
    // Creates half as many forks as there are CPU cores.
    maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1
}

Maven

Maven’s Surefire plug-in can execute tests in parallel as well. Its capabilities differ by testing framework: JUnit 4, TestNG. However, there is no dedicated support for parallel testing within the same JVM in Maven for JUnit 5. You can use the parallel test execution built into JUnit 5 instead.

It is important to note that all of Maven’s options mentioned so far achieve concurrency within the same JVM as JUnit 5. If you want that Maven runs each test in a separate process, you need the option forkCount with a value higher than 1. It works with all test frameworks supported by Surefire, including JUnit 5.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!-- other options -->

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M7</version>
                <configuration>
                    <!-- Creates half as many forks as there are CPU cores. -->
                    <forkCount>0.5C</forkCount>
                    <reuseForks>true</reuseForks>
                </configuration>
            </plugin>
            <!-- other plug-ins -->
        </plugins>
    </build>
</project>

Concurrency Within the Same JVM Is Dangerous

In summary, there is one big difference between the parallel test execution of JUnit 5, Gradle, and Maven: Whether the concurrently running tests share a single JVM or not. That has a massive impact on your tests.

In short, once your tests share a single JVM, your tests become multi-threaded code which can cause all kinds of issues. For example, changing the default locale in one method impacts all other running tests. Or a single test does no longer have its Testcontainers Container for itself when using the recommended patterns.

JUnit 5 supports several synchronisation mechanisms to prepare your tests for parallel execution within the same JVM. But for me, that requires too much thought and work for very little gain¹ if you can use multiple workers in Gradle and Maven without requiring any² code changes.