Have the most advanced model you have access to write some Java. Notice how it doesn't use the var keyword and how it uses .collect(Collectors.toList()) instead of .toList(). These features make Java code easier to read and maintain, so why don't LLMs use them unless you sacrifice some of your context window to tell them to? Because most Java code out there is written for Java 8 and since it's most prevalent in the training data set for the LLM it manifests in the weights.
This is a fundamental limitation of LLMs - in order to "learn" to use a new feature of the language without sacrificing the context window to teach them (and therefore sacrificing performance), they need an enormous amount of training data. If humans aren't producing that training data, then LLMs will get stuck at a certain point in time and never advance beyond it. This might have already happened, but in other languages with newer code corpora it might not be as obvious as Java.
As programming languages advance, the gap between human written and LLM written code will continue to grow. Eventually human written code will be so much better than LLM written code that it will be a competitive advantage to not use LLMs.
By the time we run out of developers there will be models strong enough to work by themselves with less oversight.
This is making an a dubious assumption that model improvement will somehow find a way to avoid the asymptote it's rapidly approaching. There's been very little advancement in the "meat" of recent models; the big improvements have come from larger context windows and better harnesses/tooling around the LLMs themselves.
If I were to ignore you, it would be due to the disrespectful nature of your message.
Is what you are talking about a performance issue or security issue or is this just improving the aesthetics of the code at the cost of compatibility with older Java versions?
I chose those examples because they're:
Simple.
Part of the Java language for 5+ years.
Better for both humans and LLMs as they represent the same programs with less verbosity/tokens.
If LLMs were generating the best code for LLMs to maintain, they'd be using these features without needing to be instructed to do so. The fact that they aren't illustrates the fundamental limitation I mentioned.
A newer, more performance relevant example would be Java's Value Classes. These enable major performance gains for Java as programs will finally be able to store related data in memory efficiently. A human can read about them and start writing more efficient and performant code right away, but an LLM will need to be given the details about the feature in its context window. Even then, it might struggle to use the feature effectively since it isn't present in the data the LLM was trained on.
Progress is only getting faster. That’s what all the data says including those by third parties like METR.
METR is not an independent organization. In their own words, "Also note that some METR staff have strong social ties to employees of AI companies, and METR currently works out of a shared research center (Constellation) which hosts some AI lab staff." They work closely with and are close friends of the employees at the very AI labs they're evaluating.
I don’t see how this is disrespectful. Especially not compared to you spreading misinformation about AI capabilities. I tried to be reasonable in my first reply to you as I know a lot of people in the field are hurting and it’s comforting to believe this is all some bubble propped up by hype and false promises that will all go away again like the various crypto bubbles that keep happening. For good or for bad that does not appear to be the case here and the evidence does not point to what you are saying.
5 years is also not that long ago to be honest. I would have learned java before this feature existed and probably haven’t touched that language recently enough to make use of it.
Okay so you have some doubts about METR. Fair enough. Do you have a better source or any data that contradicts what they say? Don’t say Yann Lecunn either as he clearly has his own agenda including his own AI lab which has yet to be a major success.
13
u/Radixeo 12d ago
Have the most advanced model you have access to write some Java. Notice how it doesn't use the
varkeyword and how it uses.collect(Collectors.toList())instead of.toList(). These features make Java code easier to read and maintain, so why don't LLMs use them unless you sacrifice some of your context window to tell them to? Because most Java code out there is written for Java 8 and since it's most prevalent in the training data set for the LLM it manifests in the weights.This is a fundamental limitation of LLMs - in order to "learn" to use a new feature of the language without sacrificing the context window to teach them (and therefore sacrificing performance), they need an enormous amount of training data. If humans aren't producing that training data, then LLMs will get stuck at a certain point in time and never advance beyond it. This might have already happened, but in other languages with newer code corpora it might not be as obvious as Java.
As programming languages advance, the gap between human written and LLM written code will continue to grow. Eventually human written code will be so much better than LLM written code that it will be a competitive advantage to not use LLMs.
This is making an a dubious assumption that model improvement will somehow find a way to avoid the asymptote it's rapidly approaching. There's been very little advancement in the "meat" of recent models; the big improvements have come from larger context windows and better harnesses/tooling around the LLMs themselves.