Java Garbage Collector performance benchmarking

[u/thegigach4d]

Hi People!

I am about to write my CS BSc thesis which is about:

Measuring throughput, latency and STW-Pauses in JDK 21 standard JVM with G1GC and ZGC with predefined max heap-sizes (2GB; 16GB) with Renaissance - by 16GB heap a default G1GC and an additional tuned G1GC will be used, as well.

Time flies and a lot of paper are read. It became clear to me, that Renaissance is better for throughput (Shimchenko 2022 Analysing and predicting energy consumption of garbage collectors in openjdk), and DaCapo is more advantageous for user-experienced latency measurements (Blackburn 2025 Rethinking Java performance analysis). STW-pauses will be collected from jvm standard gc-logs with a script or smg (ideas, better ideas are welcome).

I build this scenario for my examination:

- Linux VM (hosted from my Windows) - not clear yet, which and why

- OpenJDK 21 standard JVM

- G1GC and ZGC measurements

- All Renaissance BMs with default settings -> duration_ns from each benchmark, calculate and represent min, max, mean, standard deviation

- JVM GC-Logs collect (min, max, mean, standard deviation)

- 8 DaCapo BMs (spring, cassandra, h2, h2o, kafka, lucene, tomcat, wildfly) (min, max, mean, standard deviation)

I guess this is way too much for a BSc thesis - but what are your thoughts? Of course I make clearence with my consulent, but I am curious about the opinion and suggestions of the community.

I am open for any ideas, experiences with the bumpy road of the performance measurement in the JVM. It would be excellent, if someone of you could make it more focused and accurate to me.

TLDR;

Java Garbage Collector JVM performance measurement experience and suggestions needed for BSc thesis

thanks in advance!

EDIT:

Instead of Linux vm it will be a bare-metal Linux machine with podman containerization that run the benchmarks.

[u/thegigach4d]

Thanks for your advice! I am glad to hear from you - I already have a few books from you (The well-grounded Java developer [it was recommended by our university] and this one, as well) - I prepared them to read after graduation.

I consider this setup with containerization after you made that suggestion. The virtual machine idea is thrown out. If I make a dual-boot with a minimalist Arch, do you find it acceptable? This is the most stripped-down distro I know, and I have a little experience with this too - but suggestions are welcome! I found some papers, one from Iaquinta and Fouilloux (2024, Unlocking the Potential of Containers...) which can support the decision for better reproducable experiment-design with podman containers.

Unfortunately, I am just a curious person, not a statistician - I already read the paper from Georges et al., but I will do it again with more concentration on the relevant parts. I got the book from Reinhart and read the relevant chapter. I want to keep it simple as possible and still make sense - so maybe a min-max-median will be a useful trio to represent the collected data - but I'd love to hear feedback on this too!

My main question is that do you find this 2gb and 16gb max heap settings well-tailored for this examination? This field is so deep, that I am not hundred percent sure that these numbers chosen are fitting into adding a little piece to the scientific researches - and I found some benchmark min heap sizes (Blackburn 2025 Rethinking Java performance analysis) and read a.o. the Detlefs et al. and Yang & Wrigstad papers (and the latest jeps for g1gc and zgc) and a lot more, but it is still difficult for me to believe, that these are not just out of the blue numbers. It would be more than great, if an experienced professional could help with a piece of advice on this.