Benchmark Java codes with JMH
javaJMH (Java Microbenchmark Harness) is a benchmark tool for JVM languages. If you execute small codes naively, the performance can be better than actual due to optimization such as JIT compile that is not performed in the real case that whole codes are large. JMH prevent the optimization from being applied, so it seems that it can benchmark accurately.
Create a project according to the README and try to run the benchmark.
$ mvn archetype:generate \
-DinteractiveMode=false \
-DarchetypeGroupId=org.openjdk.jmh \
-DarchetypeArtifactId=jmh-java-benchmark-archetype \
-DgroupId=org.sample \
-DartifactId=test \
-Dversion=1.0
$ tree
.
├── aaaa
└── test
├── pom.xml
└── src
└── main
└── java
└── org
└── sample
└── MyBenchmark.java
$ mvn clean verify
$ java -jar target/benchmarks.jar
...
# Benchmark mode: Throughput, ops/time
# Benchmark: org.sample.MyBenchmark.testMethod
# Run progress: 0.00% complete, ETA 00:08:20
# Fork: 1 of 5
# Warmup Iteration 1: 3341043347.302 ops/s
# Warmup Iteration 2: 3413334610.874 ops/s
# Warmup Iteration 3: 3435333707.240 ops/s
# Warmup Iteration 4: 3411134300.511 ops/s
# Warmup Iteration 5: 3357532714.229 ops/s
Iteration 1: 3400156491.349 ops/s
Iteration 2: 3386226994.749 ops/s
Iteration 3: 3447242449.844 ops/s
Iteration 4: 3445778472.937 ops/s
Iteration 5: 3445982939.981 ops/s
# Run progress: 20.00% complete, ETA 00:06:41
# Fork: 2 of 5
...
Benchmark Mode Cnt Score Error Units
MyBenchmark.testMethod thrpt 25 3392960046.446 ± 27900374.591 ops/s
Annotate @Benchmark to the method, run @BenchmarkMode benchmarks, and output with @OutputTimeUnit unit. Benchmarks can have states with @State and @Setup(Level.Trial) initializes it only once per fork. You can return values or pass them to Blackhole.consume() to avoid getting rid of codes due to dead codes.
package org.sample;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import java.util.concurrent.TimeUnit;
public class MyBenchmark2 {
@State(Scope.Thread)
public static class MyState {
public int v;
}
@Benchmark
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
public void some(Blackhole blackhole, MyState state) {
state.v = state.v * -1 + 1;
String notusedValue = "" + state.v;
blackhole.consume(notusedValue);
}
}
The required dependencies are the following two. Uber-jar including org.openjdk.jmh.Main as a main class is built.
<dependencies>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>${jmh.version}</version>
</dependency>
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>${jmh.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${uberjar.name}</finalName>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.openjdk.jmh.Main</mainClass>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
<filters>
<filter>
<!--
Shading signed JARs will fail without this.
http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar
-->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
Or, you can call it directly with passing classpath.
<!-- mvn exec:exec -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<executable>java</executable>
<arguments>
<argument>-classpath</argument>
<classpath />
<argument>org.openjdk.jmh.Main</argument>
</arguments>
</configuration>
</plugin>