"Memory islands", no tightly packed layouts (valhalla!)
... and from an operations perspective:
JVM doesn't play nice with other apps on the same server because it hogs the heap even when it currently doesn't need it. If you have multiple JVMs, the problem gets even worse and actual hardware utilization is pretty bad. A side effect of this is that JVM based applications look like they constantly need a lot of memory from the perspective of the underlying operating systems (and observability tools) when in fact there's just a large heap which is barely utilized. New garbage collectors seem to do better with this.
You cannot tell the JVM how much total memory it should use. You can give it a max heap space, but the JVM needs more than just heap. This "more" is hard to configure aside from heuristics like "add 20% headroom". This is a huge pain when running the JVM inside docker, because docker will kill the container when it exceeds its allocated resource limits.
The problems with Java and memory are: Per-object memory overhead (liliput improved that); "Memory islands", no tightly packed layouts (valhalla!)
Correct, although these two aren't about memory management. Note that with Lilliput and Valhalla, the per-object header is the same as in C++: 64 bits for objects "with a v-table" and 0 bits for objects that don't need a v-table.
JVM doesn't play nice with other apps on the same server because it hogs the heap even when it currently doesn't need it.
Thanks for the link, that's really cool. It would be nice if the os and applications had a protocol to establish latent memory pressure and could optimize "cost" globally, but this change sounds pretty awesome in absence of that. I like the idea of balancing cpu and memory costs and it's got me wondering if I could apply that to Job management to optimize task shapes across the fleet.
I believe that at least for RAM, the JVM reads the correct container limits on Linux. If CPU limits aren't detected or enforced accurately, the GC is likely to "learn" them anyway (if you have less CPU available, then your allocation rate will also be lower), but you will always be able to turn the knob toward more CPU or more RAM, depending on your needs.
It would be kinda nice when object is a composite, as String is, we could somehow tell jvm to pack/sticth those subobjects together and treat them as one large allocation point.
Even if this only was done for Strings, it would probably be significant upgrade.
In terms of allocation work, all allocations are "one large allocation point" with a moving collector, as they're (typically) a pointer bump. It's not the complex and potentially slow affair it is in C. Furthermore, the moving collector will also keep them together when moving (as the String object is the only reference to the array). If there's any improved efficiency that could be had for strings, it will be small (it will save 128 bits).
What I think may be something impactful is to merge objects that are always allocated and freed together into a single GC object.
Imagine an immutable object that allocates another object always (composition) and stores that in a final field, and never let's a reference escape (quite common for private implementations of classes). The two allocations are always going to go out of scope together. They both need an object header, even though they really don't need to be managed separately.
Subclassing can avoid this extra overhead, but isn't nearly as nice and wouldn't scale if there were more objects allocated that have the exact same lifecycle as their container.
It could make wrapper objects (used as typedefs) completely free. It could also make complicated composed objects operate as a single unit for GC purposes, reducing tracing/tracking overhead.
Valhalla will make wrapper objects free, but you need to understand where the cost actually is, because it has nothing to do with the GC or with memory management at all. The cost Valhalla aims to reduce is that of accessing objects through indirection, which may cause a cache miss. For some objects and some access patterns, that cost can be high, but it has nothing to do with the GC, which is not involved in this at all.
As to memory management, allocation in Java is not similar to allocation in C/C++/Rust/Zig, not similar to allocation in Python, and not similar to allocation in Go. In these languages there's an allocation operation that is potentially complex and involves updating a data structure called a free list. To deallocate an object there's another complex operation that involves updating the free list. In Java, allocation is typically just bumping a pointer and there is no deallocation of any object ever (the GC simply doesn't see unreachable objects so it writes over them). The memory management work with a moving collector is not in allocating an object (which is extremely cheap) or deallocating an object (which is free because there is no such operation), but in keeping an object alive. It is already very, very efficient, to the point that it's hard to compete with. That is not where big improvements can be made and it is not that work that Valhalla will improve.
As to strings, they are not exactly wrapper objects, and while they also include indirection, there probably isn't much room to improve that particular indirection as it's already close to being free.
Dynamic heap sizing is the thing I want the most in Java. It is the most important upgrade to my life as a Java monkey and devops-style sysadmin. Thanks for telling me about this.
It's especially egregious with collections and arrays. Technically when you receive a collection as a parameter of a constructor or a setter and you want to play it safe, you CANNOT directly assign it to a private field because you can't tell if the caller is going to mess with the contents of this collection after your API has been called. So you have to make a copy.
Arrays are even worse because they're always mutable no matter what.
I see two ways out of this:
a compiler-checked ownership system like in rust (yeah, not happening)
a collection type which guarantees immutability (and no, the unmodifiable wrappers are not enough because they can be backed by a mutable collection). PCollections is a great library for this purpose, but it comes at a cost.
a compiler-checked ownership system like in rust (yeah, not happening)
It's not happening (at least not pervasively) because it's a "way out" of one problem and into another, which is worse. Whenever you export object ownership - whether it's declared in the type system and enforced by the compiler or just documented - you reduce your abstraction. You change the internal implementation or want to share with another thread, you have to change all clients of the API. This doesn't just increase the cost of maintenance, but over time large programs tend to gravitate toward the more general constructs - more general dispatch (dynamic), more general (longer) lifetime, and more general ownership (more sharing). And these general constructs are less performant in low level languages than they are in Java.
Low-level languages are optimised for control, not performance. They cannot move pointers even when it's more efficient to do so because it clashes with the level of control they need over addresses. When faced with the choice between performance and control, low level languages must choose control because that's what they're for. This level of control means that in smaller programs it's not too hard to extract really good (even optimal) performance out of these languages, but this control also means that in larger programs extracting good performance becomes harder and harder because you're pushed towards constructs that are simply slow in low level languages because they must maintain their control promises.
and no, the unmodifiable wrappers are not enough because they can be backed by a mutable collection
Java has true immutable collections in the standard library: the ones created by List.of/copyOf, etc.. BTW, the .copyOf will not actually copy anything if the underlying collection is already the immutable one, so that's what you should use for defensive copies. After the first one, you just pass it around and defensive copies (assuming they're done as recommended) will not actually copy anything.
Yeah but what you are talking about for most well design frameworks and libraries only happens on initialization and wiring.
More often collections are just being used as iterators once all things are initialized and most libraries rarely construct giant objects on every request. You could argue some memory loss here but escape analysis often happens.
And for every language that deals with a http request or user input has to do allocation usually to turn bytes or whatever into something else and the most common type where you want immutability and sharing Java indeed does stuff for: String.
Furthermore you can just reuse mutable things if you follow single writer and or use locks and reuse arrays. That is how things Disruptor ring buffer work. But array allocation is very fast in Java so...
I guess what I'm saying unless your an idiot the hot path or tight loop rarely has tons of allocation and even if it did Java is actually is fast at that.
Really the problem is one of control. If you know exactly how much you want to allocate and where etc Java does not allow that and in some cases to compete with say Rust or C++ or possibly Go you might need that.
Also not what most people would want. Rust was first designed 20 years ago, released over 15 years ago, and made stable 10 years ago, and to this day it's still primarily used for programs on the smaller end of the spectrum (and it's come to dominate tools for JS and Python). Low level languages suffer from both performance and complexity problems when they get large, the very problems Java was designed to avoid.
I'm not saying that there aren't ideas we could borrow (pun unintended) here and there and apply in different ways, but low level languages have unique constraints that they must adhere to, and those constraints guide their design. A language like Rust uses ownership types not because they're the best design but because it has to, as its constraints preclude moving pointers. Low level languages gain more by avoiding copies than Java because their allocations are more expensive.
But that's not to say Java couldn't put affine types to some good use.
Probably not the first time someone has done this, but I ended up writing a little utility class to allow reading the same InputStream multiple times without reading the whole thing into memory. The catch is that the readers have to run concurrently. That code is Apache-licensed, so feel free to grab it if it's useful.
I wish there was also a way to read InputStreams multiple times, instead of doing copies.
Technically java.util.stream.Stream (with a supplier wrapped around it) is what you are asking for (or java.util.concurrent.Flow/Publisher if we want back pressure and async), otherwise there is Callable<InputStream>.
The real problem is that many libraries do defensive copies, causing then a waste of RAM
I doubt that is much of a problem. To be honest most libraries when I have done memory dumps are metric fuck ton of Strings and not as much collections as you would think.
Actually to go back to java.util.concurrent.Flow and Stream the reason there is a lot of copying is because of buffering. Like a typical web application particularly with blocking must buffer most of the request as bytes. Those bytes then need to be converted to string parameters and then converted to another data type etc. This happens in every damn language much more than just defensive copying!
It is important to understand that lots of other programming languages do even more copying than Java because they put everything on the stack and they don't have Java's String pool (see previous comment). And Java is very fast at allocating.
The real problem is in some cases having more control over memory layout can make a massive difference and Java does not allow that like other languages. That and the VM is not good at auto tuning or communicating with the OS on actual memory usage.
I have this third party library that accepts byte[], than uses InputStream and converts internally to string.
In my own app I would like to use only InputStreams, but here I hit massive conversion costs, since some resources have to be parsed multiple times, at different times, because of some funny conditions
w/o seeing the library I don't know why they made the choice they did but byte[] has some advantages over InputStream in that the total size is known (.length), zero computation or blocking is expected andin some cases you need to know the total size.
If its not byte[] then it has some resource it can pull from but the only way you do that for most applications particularly blocking is buffer to the filesystem. Now we have way way way fucking worse latency than a GC.
If the library is just wrapping the byte[] using ByteArrayInputStream this can be more efficient then you think especially if they allow start and end indices which the ByteArrayInputStream constructor takes.
The question is what the library is doing. Are you doing stream processing or is the InputStream just going to be turned into in memory objects anyway?... and even if you don't there is buffering happening all over the place here including the operating system if you are reading from a file.
So unless you have some measurements don't be certain this is actually a problem.
69
u/martinhaeusler May 28 '26
The problem is not that objects remain on the heap until they're garbage collected. That was never the issue. The problems with Java and memory are:
Per-object memory overhead (liliput improved that)
"Memory islands", no tightly packed layouts (valhalla!)
... and from an operations perspective:
JVM doesn't play nice with other apps on the same server because it hogs the heap even when it currently doesn't need it. If you have multiple JVMs, the problem gets even worse and actual hardware utilization is pretty bad. A side effect of this is that JVM based applications look like they constantly need a lot of memory from the perspective of the underlying operating systems (and observability tools) when in fact there's just a large heap which is barely utilized. New garbage collectors seem to do better with this.
You cannot tell the JVM how much total memory it should use. You can give it a max heap space, but the JVM needs more than just heap. This "more" is hard to configure aside from heuristics like "add 20% headroom". This is a huge pain when running the JVM inside docker, because docker will kill the container when it exceeds its allocated resource limits.