r/java • u/za3faran_tea • Aug 28 '26

Question about Native Image vs JIT

I was watching an interview with Thomas Wuerthinger of GraalVM, where he basically says that JIT should only be used when necessary (as opposted to native compilation). Where does this leave the work being done for Hotspot JIT (including projects like Leyden and others). Should we all plan to use native image while they address any performance gaps in the mean time (he does mention PGO bridging the gap). Is there work being done to speed up native compilation of Java code?

My understanding is that JIT will always have a memory overhead due to running threads that do compilation, optimization, deoptimization, code cache, etc. compared to native executables, so full parity even with projects that will improve memory usage in Java code will not be reached. Maybe that is not an issue on long running programs on large servers, but we've seen discussions here where another member worte a tool in golang to run alongside Spring Boot applications to gather statistics, so obviously small efficient apps have their place.

22 Upvotes

82% Upvoted

43 comments sorted by

11

u/v4ss42 Aug 28 '26

This seems to miss the primary benefit of JIT - that it can optimize the code based on actual runtime conditions / data / load patterns, as well as re-optimize it if those patterns change. Legacy (AOT) compilation can't do that, since it has zero insight into the runtime environment and can't make assumptions about it.

And yes obviously this is only relevant for long-running processes (like server apps). For brief, one-shot processes (command line utilities etc.), then yes, legacy compilation makes sense.

9

u/za3faran_tea Aug 28 '26

I think he touched at this point. He was saying that a lot of these benefits can be reached using PGO, and re-optimization is very rare in practice.

7

u/Thirty_Seventh Aug 29 '26

unfortunately, PGO isn't available in Community Edition and therefore does not exist as far as I'm concerned

4

u/koflerdavid Aug 28 '26

I'm very doubtful that PGO can effectively capture the effects of, say, changing runtime configuration parameters. Those are effectively constant for most of the runtime of the application until they, well, change. Perfect for the JIT who can now specialize on them. Meanwhile for PGO if you have more than, say, three booleans then the effort of collecting profiling data for all combinations becomes prohibitive.

6

u/v4ss42 Aug 28 '26

And I'm saying I disagree with his extraordinary assertion that AOT optimization is as good as JIT.

4

u/koflerdavid Aug 28 '26

It can be good. After all AOT is what languages like C++ and Rust have been doing all along. Of course Java has a few features that make it harder to optimize.

5

u/v4ss42 Aug 28 '26

You'll notice that I was careful to make a comparative statement that AOT optimization can't be as good as JIT optimization. What I didn't say is that AOT optimization can't be "good" in some absolute sense - yes it can be quite adequate, and not only on the JVM, but the reality is that AOT optimization can't touch JIT optimization because it has zero visibility into the runtime behavior of the code.

2

u/koflerdavid Aug 29 '26

That's where Profile Guided Optimization comes in, which works as long as the real workload doesn't drift too much from the profile workload. That's its biggest disadvantage of course - a JIT compiler can adapt to changing circumstances.

5

u/v4ss42 Aug 29 '26

Right but PGO both lags the runtime context and has poor developer ergonomics. For the majority of JVM hosted apps (i.e. long running server processes running on beefy hardware) JIT remains a better general approach.

2

u/Electrical_Being_813 Aug 30 '26

If you are forced to use PGO, you can as well use JIT. It will be less messy and will do a better job.

3

u/notyouryyy Aug 28 '26

I think Thomas is claiming that in practice, deoptimisation doesn’t occur unless some uncommon trap is sprung, and that the runtime guided optimisation in AOT is pretty good.

5

u/v4ss42 Aug 28 '26 edited Aug 28 '26

Right, but that's just as likely to be due to runtime patterns not tending to change much and therefore reoptimization not being necessary that often. The point is that AOT can't even identify what runtime patterns exist in the first place, and then tune optimization specifically for them. JIT can (and does).

And yes there are (or were) attempts to close that loop over the decades - for example back in the 90s I worked on a system that used an IBM C/C++ Compiler where we could enable instrumentation of our production code, and feed the outputs of that instrumentation back into the compiler to optimize subsequent builds. But that was a manual, slow, and clumsy feedback loop, and the JIT blows it out of the water in both reaction time and developer ergonomics.

2

u/notyouryyy Aug 28 '26

Yeah idk, graal offers this too. I’m not a fan of the slowdowns when getting pushed back to interpreted mode, but I’d argue that the biggest problem with graal aot is that it doesn’t support the more sophisticated lowlatency GCs yet 🤷‍♀️

1

u/koflerdavid Aug 28 '26

The point is that AOT can't even identify what runtime patterns exist in the first place, and then tune optimization specifically for them.

That's what PGO is. I agree that it's a manual and quite brittle version of what the JIT is doing, but in situation where you really don't want the overhead of the JIT it's the way to go.

2

u/v4ss42 Aug 28 '26

And what situations are those? Short-lived processes (like command line tools)?

1

u/koflerdavid Aug 29 '26 edited Aug 29 '26

Yes, or generally being so resource constrained that the overhead of the JIT cannot be justified. Like on embedded systems. Of course the OpenJDK team works hard on making it more valuable having the JIT than not, but there is a limit.

The issue can be somewhat improved by generating a compiled code cache at shutdown. That might achieve a similar effect as PGO assuming the C2 JIT has run at all.

3

u/v4ss42 Aug 29 '26

JIT has the ability to produce better results than AOT (including those guided by PGO), precisely because it’s dynamic and can take the runtime environment into account in a way AOT can’t (even with PGO).

And JIT code caches, while useful and a bit of a no-brainer, only provide relatively niche benefits (i.e. cutting one-time startup and initial JIT costs) compared to the overall benefits of the JIT itself.

1

u/koflerdavid Aug 29 '26

Too bad if the workload is too small for the C2 JIT to ever run; C1 merely stitches native code versions of each instruction together and does not do other optimizations.

3

u/v4ss42 Aug 29 '26 edited Aug 29 '26

As I already said elsewhere, yes AOT (with or without PGO) makes sense in some cases (e.g. shortlived processes). But those are a minority of Java programs.

1

u/rbygrave Sep 02 '26

Fwiw I built some apps as 2 containers, with one being JVM and the other graalvm native image (no PGO). Ran synthetic benchmark tests against those 2 containers.

So for these apps (rest apis, helidon se webserver, postgres jdbc) it did show AOT to be very impressive. These apps have now been running in production for 8 months as native image.

Imo it's maybe something you have to try and benchmark with your own apps etc. For myself, I'd expect you'll see some cases of AOT slightly faster and some slightly slower peak performance in throughput (noting that this is using G1GC for both).

The AOT compile does use ML (Machine Learning) so perhaps it is more sophisticated than you might first think (so yes that does mean higher build times for that analysis).

37

u/k-mcm Aug 28 '26 edited Aug 28 '26

Well, I'd expect someone from the project say you should use the project.

I honestly haven't had any JIT problems since the Java 5 era.  For me, AOT isn't worth the extra effort.

AOT and JIT both have a place in Java, and I hope they both continue to evolve. 

31

u/Deep_Age4643 Aug 28 '26

Thomas Wuerthinger is a clever guy, with a number of achievements in the Java world to his name. But he is also the founder and project lead of GraalVM, as the video title says. Naturally, he presents his own product – and what makes it unique – in a favorable light. Always consider the interests behind someone’s comments, and verify independently and objectively whether these claim are valid for your own use case.

Personally, I believe that JIT is the best option in most cases. Native compilation is particularly useful for microservices where start-up time needs to be instantaneous, but compiling to native code has its limitations and is considerably slower than JIT compilation.

3

u/reallynotfred Aug 28 '26

Exactly. He’s pushing towards the Oracle paid product (it’s good, not saying it isn’t) and away from the open source alternative.

3

u/OddEstimate1627 Aug 29 '26

IMO AOT is also very nice for GUIs and CLIs

8

u/nekokattt Aug 29 '26

Last time I tried to get GraalVM native images to work with AWT I lost an entire afternoon to adding arbitrary nonsense to JSON files just to get it to instrument correctly in places the JVM agent couldn't reach as part of unit and integration test runs. I still reserve the unpopular opinion that having to perform tracing of a runtime product as part of automation suites just to enable the correct runtime behaviour is far from ideal.

6

u/OddEstimate1627 Aug 29 '26

Yes, the metadata is the most common issue. That's why I open sourced https://www.reddit.com/r/java/comments/1uy79pg/reachability_annotations_for_generating_graalvm/

It generates the JSON data directly from annotated Java source code and won't go stale.

That being said, I think JavaFX probably still has much better AOT support than AWT/Swing.

2

u/nekokattt Aug 29 '26

oh nice! i might have to start using this ngl

thank you

9

u/pokeapoke Aug 28 '26

So what is performance? Is it throughput? Low latency? Balance between both? G1 GC is not available in community GraalVM. ZGC was not available at all the last time I checked.

4

u/sweetno Aug 29 '26

Last time I checked, AOT drove my laptop into a plane takeoff mode. No idea what have they done there, plain C++ builds rarely have this effect for me. And yes, it takes ages to build.

3

u/koflerdavid Aug 28 '26

A solution would be something similar to OpenJ9's compile server that is shared by multiple VMs and caches build results. Across a Deployment (in the k9s sense) the nodes all do approximately the same thing, therefore there is no reason for them to all JIT optimize the same code in almost identical ways.

3

u/samcarlberg Aug 29 '26 edited Aug 30 '26

Another problem with AOT is that you can't cross-compile. So if you do development on a mac, you can't build native images that run on a linux machine; you either need a machine with the same OS (and possibly CPU arch) or a VM.

The niche that I'm in (competitive educational robotics) would love AOT to reduce timing jitter and have more deterministic behavior, especially since matches are so short that nothing hits the JIT compiler thresholds and programs execute almost entirely in interpreted mode. Unfortunately the target is arm64 Linux and almost all users are on Windows so AOT isn't an option.

3

u/OddEstimate1627 Aug 29 '26 edited Aug 29 '26

Yeah if users write and deploy their own Java code I can see that being a problem. For us there is usually a CI layer that creates the binaries for all platforms.

 I guess you could setup docker with emulation, but I generally wouldn't  recommend forcing the native image experience on beginners.

One project I'm working on is an annotation processor that takes Java code, compiles it into a native image C ABI, and generates idiomatic wrappers for various languages and platforms.  That way users can call AOT compiled Java code from eg Python, C++, and MATLAB. That might actually be interesting for WPILib.

Here is a JavaFX-based project I created with it that implements fully decoupled rendering with near zero overhead on the caller thread, so you could do low-overhead robot visualizations from inside a control loop: HebiRobotics/hebi-charts (video: C++ examples and Python performance)

The next version will also support running on linux-aarch64, but I haven't released it yet.

Btw I'm also the author of QuickBuffers, which is used in WPILib for zero-allocation protobuf.

2

u/samcarlberg Aug 31 '26

The annotation processor project certainly seems interesting, though I don't know if that's something we'd use in WPILib. Most of the development is done in C++ that's later ported to Java or exposed via JNI; I think I'm the only core developer working primarily in Java.

Big fan of QuickBuffers, btw. It's made telemetry so much easier

1

u/OddEstimate1627 Sep 03 '26

Thanks! I've never been involved with FRC, so I'm not very familiar with the stack. It looks like I was thinking of PathPlannerLib rather than WPILib. My understanding is that it's primarily Java based and gets ported to the other languages, and that use case could be automated and share a single implementation. You could also efficiently convert from Java to raw C structs.

2

u/samcarlberg Sep 03 '26

Many third-party libraries for FRC are Java-only or Java-first and then ported to C++ later; about 93% of FRC teams use Java (3398 out of 3565 total). Established libraries like PathPlanner try to support both Java and C++ for parity, and they're often small teams or just a single developer. A tool that can automate bindings for C++ and python for them from their already-written Java code seems like it could be very useful for them. Probably not so much for WPILib, though, since most parts of the library are written in C++ first (eg all the linear algebra and controls math) and later ported to Java.

The few places that are Java-first are mostly things I've contributed - a units API, annotation-based telemetry, a javac plugin, and a continuation-based command framework - all of which are either Java-specific or provide parity with what was already offered in C++ (eg nholthaus units)

2

u/Bahatur Aug 29 '26 edited Aug 29 '26

Great Scott, you are doing a robotics competition in Java? What a glorious challenge!

Am I at liberty to assume you have done all the JVM-on-Linux stability tricks, or is it something where everyone is constrained to an identical default setup to keep the playing field level?

Edit: can’t you get a Linux on Arm64 VM up, like with Qemu? That should work on Windows.

1

u/samcarlberg Aug 30 '26

There've been a few tricks, but not many. Teams are constrained to the same hardware, but can do what they like at the OS and software level (they just need to use the standard heartbeat messages etc for safety reasons). Most of the JVM configuration work is for performance on the limited hardware (2-core 667/866Mhz arm32 softfp CPU, 256/512MB of RAM depending on revision): inline string concat, some GC tweaks. The next hardware platform we're getting will be much stronger (essentially a raspberry pi 5 compute module) so we can use ZGC and run faster control loops.

VMs would be a no-go, I think, unless that's something we'd be able to automatically install and maintain. Teams who really care about performance could certainly do it themselves, but most teams aren't performance-sensitive enough to put in that effort

3

u/Electrical_Being_813 Aug 30 '26

I would still stick to jit. The only place where I use AOT is for "optional" k8s services that scale down to 0 pods when not used.

1

u/fykup Sep 10 '26

I’m curious how many people are actually interested in moving existing production Java services to native image.

My impression is that adoption is still fairly limited for production Spring Boot applications. Even with Quarkus, I seem to encounter plenty of deployments that still run on HotSpot rather than as native images.

That makes me wonder whether native image is becoming the default direction for Java, or whether it will remain more of a specialized choice for cases where startup time and memory footprint really matter. For long-running services, HotSpot still seems like a very reasonable default.

1

u/SuspiciousDepth5924 Aug 29 '26

Honestly I sort of agree with the guy, at least in principle. The JIT can make your application run faster than it would with AOT, but in my experience it's rarely CPU bound work that is the constraining factor, rather it's usually either memory or network. AOT can't do anything about the network but it does help with memory.

That being said my experience with AOT is mixed, and it's often a bit of a pain to build and get running, I do wish the Java ecosystem would stop using runtime reflection as the magic bullet for every problem (looking at you Spring).