You. Hello everyone, and thanks for tuning in to my talk at this year's Conf 42 Java conference. Today I will talk about benchmarking the warmer performance of hotspot VM, crawl VM and OpenJ nine. Let's get started. So, my name is Frank Kriegl. I live in Heidelberg, Germany, and I've been working as a Java developer since almost four years, and currently I'm also finishing my master of science in parallel to my regular employment, which is also where this talk originates from. Recently I had to write research paper and I was simply curious to learn more about jvms in general and saw this as a chance to deepen my knowledge. And that's why this talk's subtitle is also a learner's journey. So here's today's agenda. First, I will start with a brief introduction. Then I will set the baseline with some information bits about JVM internals. Next I will talk about my learnings when I try to compare the warmup performance of the three different JVM. So I will spend some words on the pitfalls of creating good or bad benchmark tests. Next I will describe my test adapt for benchmarking the warmup performance, and also mention some configurations I made for the jvms on the test. Finally, I'd like to present you my test results and for sure also give some interpretation on them. In the end, I will draw a short conclusion. The goal of my talk is actually to motivate you to start with java micro benchmarking on your own. So I hope that in the end of this presentation you will have a basic understanding of JVMs and are ready to get started with your own java or JVM benchmark measurements. So let's talk about the what and why. What is warmup actually? Usually, warmup is defined as the number of iterations the JVM needs to increase the speed of method execution between the first and the nth invocation by applying JIT compiler optimizations on the bytecode. Okay, that's the definition, and now let me show you what it actually means. So in this chart you see on the vertical axis the time per operation, and on the horizontal axis the number of iterations and warmup is the part here. From the first iteration, it takes quite long to complete one operation for the JVM one method execution, and over time it's getting faster. So after 200 iterations it's much faster than the first iteration. And this decline in execution time is called warm up. Next, I'd like to answer the question why? I would like to compare this. So mainly out of curiosity, to be honest. But there are for sure also some actual reasons. Like I was searching on the Internet and I only found little research in this area. So the most interesting article I found was don't get caught in the cold warmup your jvm and it's but hot tub which is a new JVM implementation to use pre cared JVM to avoid the warmup overhead. And so I thought, okay, why not doing my own research? I just wanted to see how much the jvms I wanted to test how they differ in the method, warmup speed, and eventually if there is a difference. Well, there is a difference and what it looks like between JIt compilers and Aot compilers, and the code they produce. Okay, next it's about setting the baseline. So on this slide you see a picture of the Java heap structure. The example is for the hotspot VM, and the Java heap is actually separated into several parts. So you see here there is the young generation and the old generation. And the young generation itself consists of Eden space and two survivor spaces. So how does memory allocation happen in the JVM? New memory is always allocated in the Eden space, and as soon as just before the Eden space fills up, some minor garbage collection occurs and the objects are transferred into survivor spaces. One survivor space is always free and the other one is occupied. And if one survivor space is running full, minor garbage collection will just swap these spaces, clear up unused objects or unreferenced objects, and the survivors will stay there. If there are long living objects that survive several minor garbage collection cycles, it could actually happen that some major garbage collection occurs and these objects are then transferred into the old generation space, also called tenured space. Next, and I already mentioned it, I briefly touched the topic of garbage collection. So meanwhile there exists, I think, seven garbage collection algorithms, at least in the version of Java eleven. Thing is that garbage collection can have unwanted side effects in performance testing, so you better try to eliminate that. Luckily, there's the Java enhancement proposal 318, which is about epsilon, a no op garbage collector. I linked it here, can read the details if you like. And that's actually a garbage collection algorithm which will always allocate memory but never freed up again. Next, it's about JIT versus Aot compilation. So as you might know, Java code is pre compiled to Java bytecode, which will then be run on any JVM. The JIT compiler, just in time compiler is first doing some profiling on the bytecode and then it will apply optimizations like method inlining, branch prediction, loop unrolling, dead code elimination, and many more. And it will also only compile parts of the bytecode to machine code because it has to decide which parts of the code need to be optimized. Then on the other hand, there's the head of time compiler, which will just directly compile all the bytecode to machine code when JVM starts up. So for the jig compiler since JDK eight, there are actually five levels of tit compilation. At least that's what applies for the hotspot VM. The first level is just about interpreting bytecode, so it's level zero, and the JVM will not compile anything at all, but just run as an interpreter. And after a few iterations the chit compiler will make use of its first compiler. It's the C one compiler, also called client compiler, and produce some simple c one compile code. So we talk about level one, two three compilations, which are all done by this C one compiler. After about 10,000 invocations, code will eventually become marked as hot, and then it will become subject to level four compilations, which is then done by the c two compiler. This is cared a server compiler and it will do some much better optimization with your Java bytecode. Okay, next we continue with Java micro benchmarking. My lessons learned so I was not sure in the beginning how to start my learners journey. I searched on the Internet and found that there are existing benchmark suites like spec JVM 2008, which is from 2008, and the Decapo benchmark suite, which was first released in 2009. While the last maintenance release of the cared benchmark suite is almost two years ago, which was eight months before the release of JDK eleven, for me they felt quite outdated, so I didn't want to use them for that reason. Also, not all benchmark tests were working with the targeted Java version eleven, so I was actually trying to use them, but failed. And finally the output format. The measurement units did not suit or run it in a suitable format, which I could use for further analyzing the collected data. So simply using some out of the box benchmark suites did not work for me. So I came up with the idea of writing my own benchmark. You have to know, writing a good benchmark is not easy. There are two fault categories. On the one hand, there are conceptual flaws when designing a micro benchmark, which I will show you an example in a minute, and on the other hand, there are contextual effects when running it. Here is an example for a conceptual flaw. On the left hand side we have the method create arrayupto and the method that code elimination, which will invoke the first method to create an array with the length of 21,000 containing values from one to 21,000. The array is then processed and all the values are accumulated into the result variable, but this variable is actually never returned, so the calculation result is not used at all. If we then execute this for like 18,000 times, invoke system current time millies before and after the method invocation, we could calculate the duration it takes to execute that code elimination method by subtract the start value from the end value. But here's the issue. When running the code, the JVM will first just interpret your method and eventually collect some profiling data on it and figure out that the result of the method is actually never used because it's never returned. So at some point in time the JVM will just eliminate this invocation and you'll see that in your output that at some point in time the execution time will just drop to almost zero milliseconds because what you measure is just the time between invoking system current time release the first time and the second time. But there is JMH to the rescue, so conceptual flaws can mostly be avoided by using frameworks like JMH. JMH is the Java benchmarking house, and it is a tool that was created with the intention to help developers in avoiding common pitfalls when writing and executing Java benchmarks. So it's actually quite handy to use it. But also you have to be careful what you're doing. And here you can see one of my first tries where I was using JMH to write my own benchmark. I actually asked for some feedback on Twitter and got none, but didn't stop me from continuing my learning journey. You can see there are two things I'd like to point out here. One thing is that JMH provides you with black holes, which you can use to consume some objects in your benchmark. So this will make sure that the code is not eliminated by the JVM. You could also just return that or print it to system out that will have the same effect, but there are black holes then. Second is that you should also consider warm up and there's an annotation at fork and you can specify the number of forks which you want to execute. So how often the benchmark test should be executed in standalone jvms and also warmup iterations to actually avoid warmup when benchmarking your code. But in my case I wanted to measure warmup, so I set this to zero to get some observation. I tried several different approaches to write some good benchmark tests. I tried to reuse existing benchmark tests from the Dakarpo or spec JVM suite, but that all didn't work out for me. But in the end I ended up with a sudoku backdracking algorithm, which turned out to be working quite well for my case. So you can find that code on GitHub. I will not go into details there, but this is the code I used to benchmark the JVM warmup performance. So here's my test environment. I did all the benchmarking on a virtual machine, which is not optimal, but I tried to compensate that with multiple test runs. See that in a minute. So the operating system is a Ubuntu version 2064 bits, and I had eight virtual cpu cores based on AMD Opturam processor. There were eight gigs of ram available, no swap configured, and a storage of eight gig hard drive disk. My test setup I decided to execute my benchmark tests with 21,000 iterations to also see some effect when a method gets marked as hot. Every one consisted of 20 forks, which means that JMH will spawn up 20 independent jvms to not accidentally make use of already pre cared code. Then I executed twelve runs at different days and daytimes to eliminate these contextual effects I would face in a virtual environment. When you multiply all these numbers, 21,000 iterations in 20 forks and twelve runs, you get 5.4 million sudoku solved per JVM. Always the same sudoku though the JVM parameters. I did not touch much because I wanted to take the approach of simulating a daily user who would just throw code the JVM at the JVM and run it. Besides two exceptions, the one is that I was using the no operation garbage collector epsilon or respective other ones for the other jvms, and also the pretouch memory option, which I will explain in a minute. So here are my jvms under test I decided for the tried and trusted hotspot VM where I used an OpenJDK 64 bit build from adopt OpenjDK. And as you can see I also configured some alias for every jvm which I could use later and just shorten the amount of text on my slides. So secondly, I went for GraalVM, which is a polyglode VM. I used the community edition for my benchmark testing in version 22. And last but not least, Opengenine as an enterprise JVM, which actually promises to have better performance on its website than hotspot VM. We'll talk about that later. Yeah, with this test setup, I started my measurements. So let's take a look at the runtime flex which I used to execute my benchmark. This one is for hotspot VM, and let's go through the lines step by step. So here I specify the benchmark target, which is my backtracking algorithm, and this is just some syntax given by JMH. The next line I will have to provide some JVM arguments for JMH that it will use for every fork it spawns up to execute the benchmark. I used a configuration of 5gb of heap and also provided the flag heap dump on out of memory error to just show me if my JVM crashes. Next you see some double x flags like unlock experimental VM options, which I need to make use of the Epsilon garbage collector, which I mentioned earlier to avoid garbage collection interrupting my measurements. And then there's also the always pre touch option which will claim physical memory from the operating system right at the beginning rather than on the fly. So this would also eliminate some interference by the JVM when it would find out that it needs more memory. This flag will just tell JMH where to store the measurement output and in which format, so it can output things in adjacent format and also others. And last line, I specify the number of iterations, which is 21,000 per fork. I run 20 forks and the timeout is just set to 360 minutes, which is very high, but just didn't want to let JMH time but and abort my measurements. Okay, and the last line I just wanted to collect the output of my program into a log file. The runtime flags for GraalVM look quite similar. For one small exception, I did not find any no operation garbage collection algorithm for GraalVM in this version. So I made use of a workaround. I set the max new size parameter for libcall compiler to a number which is higher than the actually available memory for the heap, which makes the JVM create a huge young generation, but no old generation space in the heap. So what would occur here is that actually no garbage collection can occur, or before it would occur, the JVM would run out of memory. So it's important to have enough memory for your benchmark tests available. Opengen nine has also a slight difference here. I unfortunately found that the Linux version of Openj nine does not offer a pretouch option. So this one will claim memory on the fly if it needs more from the operating system. Okay, that was the setup. And now I would already like to share some test results. Here you see the overall chart which I generated out of the collected data from the benchmarking of hotspot vm. It's on the vertical axis, again the time per operation in nanoseconds. And on the horizontal axis, the number of iterations up to 21,000. If we now zoom in a little, you can see that there is a light red colored background of the warmup graph. And I call this light colored graph the scatter shade because this actually represents the scattering of the different fox individual data points of any given time slice. So they are the interquartile ranges, q one to q three, and the red line is the median value of the execution time. So on this slide, I again zoomed in to the first thousand executions. And here you can actually see that there's already in the beginning a significant drop in the execution time. There are several things we can observe here. First of all, we see that the scatter shade is tightly following the median curve and also narrowing over time. So that shows that the execution time is generally declining. Next, the median curve is also tending to be at lower bound of the interquartile ranges of the scatter shade. Which allows the conclusion that data points between the median and q three quartile under spread compared to the range from q one to the median. Which makes absolutely sense because there's a physical lower bound when executing and this behavior and the scatter shades can also be observed for the other JVM charts for GraalVM and Openj nine. Here we have the chart for GraalVM for the first thousand benchmark iterations. Both GraalVM and hotspot actually have this sudden decline at around 100 executions where the execution time significantly drops. There's not only in the median curve, but also in the scatter shade this significant decline. And we also see that at this point, the q three boundary. So the upper part of the scatter shade will eventually fall below the q one boundary of previous data points. So I tried to visualize that with this red bar. You see that here the under bound of the scatter shade is below the lower bound. That's another view on the GraalVM warmup chart between iteration 6000, 406,800 GraalVM actually shows this bump. And I did not dig into details here because I didn't have a good profile at hand. However, I think it would be definitely interesting to investigate this anomaly. If you have any guess what this bump is about, please let me know. So the blue chart is for openj nine. Again, we look at the first thousand iterations for this benchmark. You can already see that the warmup chart of openj nine looks somewhat different than the others. So first of all, there is no sudden decline at the mark of 100 iterations, but instead there are some spikes in the execution time for single iterations. You see that here are some spikes, and also later on they're getting less over time, but they are always present. I was thinking, okay, maybe these spikes could be cared by the missing pretouch option, which is not available in open gen nine for Linux. To find out if this behavior could be, or the spikes could be attributed to the missing pretouch option. I would have expected to observe similar behavior for the other two jvms when I disabled the always pretouch option for them. So therefore I made another measurement series with GraalVM and hotspot having the always pretouch option disabled. But the warmup charts looked the same. There were no spikes for GraalVM or hotspot. There were no hints for my suspicion, which leads to the conclusion that in my test setup, fetching actual memory from the operating system had only minor or even no effect on the measurement series. And these spikes in the warmup graph of opengenine cannot directly be attributed to the fetching memory from the operating system. Okay, so up to now we just had a look at each JVM individually. Now I'd like to continue to compare them. To get started, I just talk about the average execution times. So on the left hand side you see a histogram which includes the execution times for opengen nine, hotspot and GraalVM, all in JIT compiler mode. You can see that the histogram for hotspot and GraalVM looks quite similar, and opengenine describes a rather different curve. However, they all have this tail to the right. The average execution time for hotspot and GraalVM is around 0.4 milliseconds. Graalvm seems to be a little bit faster, and Openj nine is following tightly at almost 0.5 milliseconds. Then I also made some measurements where I enabled the Aot compiler for opengenine, and this one turned out to be faster than opengenine in Jit code, but still slower on average than GraalVM or hotspot. So that's also what you see here on the right hand side in the chart. Purple curve is opengenine in Aot mode. It's faster than OpenJ nine in Jit mode overall. Okay, let's dig deeper. One interesting thing to observe in the warmup charts is the amount of time, the number of iterations it takes to speed up the method execution from five milliseconds to 0.5 milliseconds. I'm talking in the unit of milliseconds, because that's easier to pronounce. But just don't get confused by the scales here. It's still nanoseconds on the chart. The first red bar is at five milliseconds, the second one is at 2.5 milliseconds, and the third one is at 0.5 milliseconds. So for this blue chart, which represents openj nine, it takes 150 iterations to gain a 90% performance improvement within the first iterations of the benchmark test. So in numbers, this means that for every next execution, the JVM can execute the method 0.3 nanoseconds faster than the previous operation. If we take a look for this KPI at hotspot, we see that the negative slope is not as steep as in open G nine, and we can also prove that by calculating it. So, reaching the lower bound of 0.5 milliseconds from the beginning, where we start at five milliseconds, takes around 700 executions of the benchmark method. So we can say that with every next execution, the JVM or hotspot can speed up the method execution by zero point 63 nanoseconds per operation compared to the previous operation. Which means that during the first few iterations where warmup takes place, hotspot Vm is 4.6 times slower than open gen nine. If we compare all the three jvms together, you will see that Opengen nine will only win the race within the first few hundred iterations. But if we compare that after around 600 iterations, we'll see that the blue chart is above the green and red chart of hotspot and gravm, which means that in the end, Openj nine will be slower than its opponents. But just right at the beginning, it's warming up faster. I also promised to shortly talk about JIT compilers versus Aot compilers, and for that I made some measurements with OpenJ nine jit mode, which is the blue graph again. And in AOT mode, which is the purple graph here you can see the flags you need to provide to enable the Aot mode on open genine, and you can easily spot that right from the beginning. The Open Geni Aot compiler starts at its maximum performance and executes the code always in the same speed, while the jig compiler will take up on that after a few hundred executions again. So having all these nice looking charts is quite cool actually. But I also wanted to know what's actually happening there. Why is the warm up as it is, and what's causing it? So for that I found ditchwatch which is a block analyzer and visualizer for the hotspot jig compiler, and it's a really cool tool actually. You can enable it with these flags if you provide these runtime flags on your jvm. However, you have to know that this will have a negative impact on performance, so do not do that during your benchmarking, but just afterwards to investigate. And the output file, which is a XML log file you can just load into jitwatch afterwards. Then jitwatch will show you the compilations for every single method. So here's an example for compilation list of the method solve integer array, which is one of the methods in my Sudoku benchmark tests. You see actually that there are some c one compilations happen and happening, and also some c two compilations also on stack replacements, but all only after 20 seconds, which is actually like half of the time the benchmark test runs. So this is way beyond the initial warm up we saw, and actually they do not have a lot of effect on the execution time anymore. So I was wondering what else would then cause the warmup in the initial 1000 iterations if all these compilations shown by Jitbotch kick in much later. So while Jitbotch is a useful tool to visualize the actions of the JIT compiler, I encountered discrepancy between the compilations shown by Jitbotch and the JIT compiler actions locked on the terminal by providing the runtime flex print compilation and print inlining. So the terminal lock output showed several inlining operations taking place already during the first iterations of the benchmark execution. These inlining operations also fit to the warmup charts where we see a steep decline over the first few hundred or thousand iterations. So these inland operations cared probably the main driver for the fast decline in the warm up graphs we've just seen. The difference between the XML compilation log file used by Jitwatch and the compilation log output on the terminal can actually be explained by the fact that there's a limitation in the log compilation option, which leads to the fact that these inline decisions made by the c one compiler early on are not included in the XML log file which is used by Jitwatch. You can read the details here where I provided a link to the OpenJDK wiki. Okay, now I'd like to draw short conclusion and also have some additional remarks to my benchmark measurements. First of all, all the benchmark measurements I made were done for JDK version eleven for all the mentioned jvms. I did not perform measurements in any other JDK version. Secondly, the benchmark measurements I conducted in October and November 2020. So meanwhile there are new versions of the jvms, so it would be interesting to also take a look at them. Yeah, and here are also my final thoughts. As just said, graalvm version 21 was recently released. It now comes with the espresso JVM, which is a JVM fully written in Java. It's Java on truffle, if you know what that means. Yeah, maybe I find the time to also do some warm up performance benchmarking on the espresso JVM. The second thought that comes to my mind is that Opengenine's benefit is definitely its Aot mode. So it's performing better in Aot mode, at least in my measurements. But I'm asking myself, why don't they make this the default configuration if they also advertise with it that they are faster than the hotspot VM? Last but not least, I think there are many other JVM that also deserve to be benchmark on warmup performance because they become more and more important in the world of different JDK releases. For example, the Amazon Krata JVM or Alibaba Dragonwell. All right, that was my presentation. I hope you like it. It was the first talk I ever held in public, and if you want to check out my references or take a look at the source code, you can find many more details on my blog post about that topic, which is linked here. If you have any questions about my measurements, my talk, my learnings, or want to discuss something, yeah, just send me an email. Here's my contact information and thank you for tuning in. See you.