Generating test coverage for Sonar with maven, surefire and jacoco

At work we’re using Sonar to keep our code quality under control. One integral part of code quality is test coverage and Sonar offers coverage metrics in the UI. However, the Sonar docs on code coverage are a bit sparse (at best) and don’t tell you the exact steps to run for getting coverage with a Maven build.

With a bit of googling I was eventually able to find the correct steps to get coverage. The key to success is the Jacoco Maven plugin. In a first Maven execution we let jacoco record coverage statistics while running the unit tests:

mvn clean org.jacoco:jacoco-maven-plugin:prepare-agent install

In the build log you see the jacoco-maven-plugin in action (we’ll get back to this in a while):

[INFO] --- jacoco-maven-plugin:0.8.5:prepare-agent (default-cli) @ sonar-code-coverage ---
[INFO] argLine set to -javaagent:/.m2/repository/org/jacoco/org.jacoco.agent/0.8.5/org.jacoco.agent-0.8.5-runtime.jar=destfile=/sonar-code-coverage/target/jacoco.exec

Then in a second Maven execution we’ll pick up those coverage statistics and convert them to XML so that Sonar can include them in its analysis:

mvn org.jacoco:jacoco-maven-plugin:report org.sonarsource.scanner.maven:sonar-maven-plugin:sonar

Hooray! Coverage info in Sonar.

The trouble starts when you need to pass custom arguments through surefire down to the JVM that will run the tests. This is done through the <argLine> configuration parameter of maven-surefire-plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <argLine>-Xmx128m</argLine>
    </configuration>
</plugin>

Looking back at the output of jacoco-maven-plugin above you’ll notice that we’re setting exactly the same <argLine> parameter as jacoco-maven-plugin does. If you don’t take extra steps only one configuration can win - and that’s the configuration you put into your pom. You have to run mvn -X to actually see this, though:

[DEBUG] Forking command line: /bin/sh -c cd /sonar-code-coverage && /opt/openjdk-bin-8.252_p09/jre/bin/java -Xmx128m -jar /sonar-code-coverage/target/surefire/surefirebooter9099169912642065145.jar /sonar-code-coverage/target/surefire 2020-05-12T07-39-04_225-jvmRun1 surefire3709014631518520641tmp surefire_01706858247537265191tmp

Note how the -javaagent:... argLine is not included in the command line arguments of the JVM that’s forked to run the tests.

A way to fix this is through creative use of Maven build properties. Configure the jacoco-maven-plugin to set a different property than argLine:

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <configuration>
        <propertyName>surefire.argLine.sonar</propertyName>
    </configuration>
</plugin>

Next, configure maven-surefire-plugin to include that property into its <argLine>:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <argLine>-Xmx128m @{surefire.argLine.sonar}</argLine>
    </configuration>
</plugin>

Note that you have to use late evaluation of the property because it’s not present when the Maven reactor starts but only after jacoco-maven-plugin has run.

Just one last hack is required to make the build work even if you don’t run jacoco coverage as part of your build. Specify an empty property surefire.argLine.sonar in your project’s properties so that it always exists and can be expanded.

For further reference I’ve put a demo project up on github.