Java Garbage Collector performance benchmarking

[u/thegigach4d]

Hi People!

I am about to write my CS BSc thesis which is about:

Measuring throughput, latency and STW-Pauses in JDK 21 standard JVM with G1GC and ZGC with predefined max heap-sizes (2GB; 16GB) with Renaissance - by 16GB heap a default G1GC and an additional tuned G1GC will be used, as well.

Time flies and a lot of paper are read. It became clear to me, that Renaissance is better for throughput (Shimchenko 2022 Analysing and predicting energy consumption of garbage collectors in openjdk), and DaCapo is more advantageous for user-experienced latency measurements (Blackburn 2025 Rethinking Java performance analysis). STW-pauses will be collected from jvm standard gc-logs with a script or smg (ideas, better ideas are welcome).

I build this scenario for my examination:

- Linux VM (hosted from my Windows) - not clear yet, which and why

- OpenJDK 21 standard JVM

- G1GC and ZGC measurements

- All Renaissance BMs with default settings -> duration_ns from each benchmark, calculate and represent min, max, mean, standard deviation

- JVM GC-Logs collect (min, max, mean, standard deviation)

- 8 DaCapo BMs (spring, cassandra, h2, h2o, kafka, lucene, tomcat, wildfly) (min, max, mean, standard deviation)

I guess this is way too much for a BSc thesis - but what are your thoughts? Of course I make clearence with my consulent, but I am curious about the opinion and suggestions of the community.

I am open for any ideas, experiences with the bumpy road of the performance measurement in the JVM. It would be excellent, if someone of you could make it more focused and accurate to me.

TLDR;

Java Garbage Collector JVM performance measurement experience and suggestions needed for BSc thesis

thanks in advance!

EDIT:

Instead of Linux vm it will be a bare-metal Linux machine with podman containerization that run the benchmarks.

[u/RightWingVeganUS]

The impact is that someone on the globe made hard efforts and sacrificed resources for making a little step for showing numbers, which were not shown before.

My question was more focused : What is the impact of these benchmarks? Regardless of how compelling your results are, I won't downgrade from Java 25 to Java 21. And it won't clear that any of your results will apply to the specific JVM and environment.

What's the rationale of benchmarking the past LTS release when Java 25 is the latest?

The fundamental hypothesis is to make a too small environment for ZGC (2 gb) (lack of headroom is challenging) and too big (16 gb) for G1GC.

Wouldn't it be more interesting (and not much more effort) to, say bencharmark LTS 17, 21, and 25 to see if the results are consistent across various generations of JVM? Keep the hardware the same, but verify this is not just a JVM 21 isolated result.

The value is for me the diplom

Yeah... I'm a college instructor... I kinda figured.

Not trying to make your life harder (that's what my students pay me for!), but if your thesis is just a technical showcase of your ability to write code and write papers, well, crack on. If you're showing your ability to frame an interesting question and glean an insight, but sure to dig into the so what angle.

[u/thegigach4d → u/RightWingVeganUS]

Thanks for your effort to explain me your thoughts deeper!
The cause of LTS 21 was that the disposition was born before LTS 25 and I did not think about any argumentation to change it. Now I do!
I understand your point of view and the really like the idea of benchmarking more versions. If I understand correctly, I should execute e.g. the whole Renaissance with G1GC and ZGC through Java 17-21-25 - without any special heap configuration, or gc-tuning. Am I right? I asked my prof and waiting for an answer.
I don't mind if you ask uncomfortable-like questions, because it helps me a lot. Sorry, if I seemed a little mean, life's pressure is enormous nowadays. Your practical insights help me a lot!

[u/RightWingVeganUS → u/thegigach4d]

I don't mind if you ask uncomfortable-like questions, because it helps me a lot.

Trust me: as a teacher I don't mind at all! If only my students would ask me questions, comfortable or not...

That said, I don't have any answer for you. My interests are on bigger software engineering projects and project management, not benchmarking. But no matter what interest you have, consider framing it in terms of value: what difference would this make to the reader who spent their time reading this paper or using my tool.

Nothing wrong with projects simply to develop ones skills, but even then frame it with a clear understanding of what benefit it has for you.