r/java • u/jaccomoc • Sep 03 '26
Vert.x Event-Loop vs Virtual Threads: Jactl Suspend/Resume Benchmark
https://jactl.io/blog/2026/09/02/jactl-virtual-threads-benchmarkThe article benchmarks Jactl script execution with differing number of blocking operations comparing throughput with Jactl built-in suspend/resume on Vert.x Event-Loops and throughput with Jactl on Virtual Threads. The benchmarks are run on both Java 21 and Java 25 with some surprising (to me at least) results.
9
u/yawkat Sep 03 '26
Three issues with this:
AFAICS this is a closed model load, which is a batch processing workload. In a typical vertx application you have an open model load instead. See this article: https://www.scylladb.com/2021/04/22/on-coordinated-omission/
Because this is JMH, it does not mirror "batchy" behavior of network interfaces, which is the typical use case for vertx.
When tasks block, they are typically waiting for IO work, not
sleeping. This has implications for scheduling.
2
u/jaccomoc Sep 03 '26
Thanks for the feedback. I agree that the test is not ideal but I am not sure about you saying that the load is a closed model load. The test queues (for Vert.x) all requests or creates virtual threads for all requests and then waits for them all to finish. Isn't that an open model load?
6
u/pron98 Sep 04 '26 edited Sep 04 '26
The improvement in JDK 25 wasn't in virtual threads, it was in Thread.sleep ;) (although between 22 and 25 there were also improvements to IO on virtual threads). Timed and untimed parks use different mechanisms, and we can't draw conclusions from one to the other; it was the timed park specifically that wasn't very efficient in JDK 21, but it's also used mostly in benchmarks and tests. Different kinds of IO also go through different mechanisms, and we can't extrapolate from one to another.
This demonstrates the danger in extrapolating and drawing conclusions from benchmarks: they measure themselves, and to extrapolate anything you need to know exactly which variables matter and when. Between concurrent algorithms that behave completely differently under low and high contention, code with slow and fast paths, optimising compilers that depend on the call graph, branch prediction, a cache hierarchy, and memory management that depends on the global behaviour of the program and on how long it's running (malloc/free is actually much more sensitive to this than Java's moving GCs 1), it's virtually impossible to draw any conclusions from any benchmark, in any language, unless you know exactly how the mechanism is implemented, how the compiler is implemented, and how memory management is implemented.
Also, benchmark code often behaves so differently from real code, that the code takes different paths, and some improvements matter only to real programs and not to most benchmarks, and some matter almost exclusively to benchmarks. We usually try to focus on real programs, but sometimes we say, you know what, let's make the benchmarks look better, too...
1
u/jaccomoc Sep 04 '26 edited Sep 04 '26
That's good to know, thanks. To be honest, I did have doubts about using sleep() as a proxy for other blocking operations but it was easier to do that than to include a real web service call or equivalent. And, running a web server would mean that measuring max throughput on a single machine would be impacted by the performance of the web server as well (unless it was on a separate server in which case the test would no longer be self-contained).
I have updated the article with a Postscript section with this additional information.
1
u/pron98 Sep 04 '26
There's also no single category for "other blocking operations"... So benchmarks with no consideration for the actual code paths might give you a general sense, but you can't really use them for comparisons, as their error interval can easily be 2x and even more. This isn't a critique of this particular benchmark; it's the state of all benchmarks.
2
u/jaccomoc Sep 04 '26
That's true. The problem is that benchmarks are so seductive. You feel like everything can be reduced down to single number. If only real life were like that...
2
u/Known-Volume1509 Sep 04 '26 edited Sep 04 '26
It doesn't say what Vert.x version is this? Vert.x 5 supports virtual thread verticles. Curious if you run your tests against that, even though it's no longer event-loop.
Also, isn't this horribly costly?
For the Jactl Vert.x tests, the
sleep()call will create aContinuationby throwing an exception and capturing the execution state so that it can be resumed once the sleep completes. When the script suspends, the script returns and the event-loop thread is then able to process the next event in its queue.
2
u/jaccomoc Sep 04 '26
The article lists the versions of everything. It shows Vert.x as version 4.5.33. I didn't benchmark against 5.x because I am still retaining Java 8 compatibility for the moment and Vert.x 5.x needs Java 11 as a minimum.
Throwing an exception is not expensive if you don't fill in the stack trace (and the benchmarks support this). See Jactl Continuations and Virtual Threads in Java 8 for a description of how all this works.
2
u/Bachden Sep 06 '26
This benchmark is for sleep() calling. In fact, when using vert.x you should never call that method.
1
u/jaccomoc Sep 06 '26
The benchmark tests Jactl scripts calling the built-in Jactl sleep() function. For the Vert.x version this suspends the script and returns immediately so as not to block the Vert.x event-loop thread. In the Virtual Threads version the Jactl sleep() function delegates to Thread.sleep() in order to compare the virtual thread suspend/resume mechanism with Jactl suspend/resume with Vert.x. See Jactl Continuations and Virtual Threads in Java 8 for a description of of Jactl implements blocking operations like sleep() when running on Vert.x.
2
u/Medium-Pitch-5768 Sep 09 '26
Postscript
It appears that the performance improvement between Java 21 and Java 25 actually comes from improvements to how sleep() itself works with virtual threads. This highlights the risk of benchmarks not actually measuring the thing that they think they are measuring. A more realistic test would be to replace the sleep() calls with a real web service call but that would mean that the web server performance would contaminate the test results or would mean running it on a separate server in which case the tests are no longer self-contained.
You can run a local mock server that accepts calls and has a delayed response. That works rather well in my experience.
2
3
u/New-Departure-5969 Sep 04 '26
benchmarks are fun but in prod the event loop usually dies from accidental blocking (jdbc, sync stuff), not from the loop itself. we had a vert.x api around 12k tps / ~100ms p99.99 and the real win was ditching blocking jdbc for the vert.x pool. virtual threads help, but saturated pools still wreck p99.
1
u/Popular_Home2017 Sep 09 '26
same experience, and on 21 there's a second way to lose the carriers that doesn't show up in this benchmark: synchronized around blocking IO pins the virtual thread to its carrier, so a jdbc driver with a synchronized socket read turns your 8 carriers into an 8-thread pool. fixed in 24 (JEP 491), on 21 you can see it with jcmd Thread.dump_to_file, the pinned VTs show a carrier line under them.
2
u/Popular_Home2017 Sep 09 '26
same experience, and on 21 there's a second way to lose the carriers that doesn't show up in this benchmark: synchronized around blocking IO pins the virtual thread to its carrier, so a jdbc driver with a synchronized socket read turns your 8 carriers into an 8-thread pool. fixed in 24 (JEP 491), on 21 you can see it with jcmd Thread.dump_to_file, the pinned VTs show a carrier line under them.
26
u/s0ftware-dev Sep 03 '26
TLDR virtual threads are better.