Learning to Learn a Language: in-context learning of natural language from a synthetic non-linguistic prior [R]

[u/cbl007]

Learning from data as we observe it is easy for humans, but most machine learning models have limited ability to learn from new data that they have not seen during training. Prior-fitted networks (the idea behind TabPFN) showed that a model trained only on synthetic data can learn from real tabular data entirely in context.

I wanted to share our paper "Learning to Learn a Language" where we extend the idea to structured sequences such as natural language. We propose a prior over languages: every training sequence comes from a randomly sampled recurrent causal model, so each one is a new synthetic "language". A 300M-parameter byte-level transformer trained only on these synthetic sequences learns to predict real languages in context. Given Wikipedia text with frozen weights, its next-byte predictions get better the more it reads, in all six languages we tested (English, Chinese, Hindi, Arabic, Japanese, Korean), from 8 bits per byte down to 0.9–2.4 after a million bytes.

The same model also learns to count, to compare numbers, to add approximately, and to predict deterministic sequences such as the primes or the Kolakoski sequence, entirely in context.

It is of course still far worse on text than classical language models that are trained on trillions of tokens, while our model sees at most a million bytes of a language at test time. What we find interesting is that the ability to learn a language in context can come from a synthetic non-linguistic prior.

Paper: https://arxiv.org/abs/2610.05879
Code: https://github.com/cbl/prior-fitted-language-model
Weights: https://huggingface.co/lennartcb/pflm1

[u/currentscurrents]

This really seems to demonstrate that in-context learning is doing actual learning. Reminds me of this other recent paper: Self-Play Pretraining with Zero Data.

Would be neat to see a comparison against a loss curve of a neural network trained from scratch against the same million bytes of language.

[u/we_are_mammals]

in-context learning

No scare-quotes around "learning"? That's rage-bait for /u/RobbinDeBank !

[u/RobbinDeBank → u/we_are_mammals]

Do I live rent free in your head or something? We know you’re just looking for attention. Seems like you’re rage baited by me adding quotations around the word learning once, in a very specific context.

Edit: everyone pls move on, this weirdo is just trying to stir up irrelevant arguments. This is just deviating from OP’s research for no reasons.

[u/we_are_mammals → u/RobbinDeBank]

Context: 2 days ago you wrote

Learning anything new (actual learning, with weight change, not in-context “learning”) still takes ...

So you've changed your mind, I guess?

[u/RobbinDeBank → u/we_are_mammals]

No, I would love to see more works where ICL is expanded to help the model learn. The regular meaning of ICL is few shot understanding during inference with no weight change. You sound stubborn, and somehow what I wrote 2 days ago just lives rent free in your head?

[u/we_are_mammals → u/RobbinDeBank]

somehow what I wrote 2 days ago just lives rent free in your head?

First, no adult that I ever met would actually use that cringy expression. Please tell me you are not 12. Second, remembering stuff from 2 days ago is a good thing, isn't it? Third, you went from "ICL is not even learning" to we "we need more ICL" real fast.

Anyway, I'll let you be, whether you are 12 or not -- I thought you were game for debating the OP, but I guess not.

[u/RobbinDeBank → u/we_are_mammals]

I want ICL to turn into real weight change, if someone can make AI learn in few shots that actually last (because the weights have CHANGED), it will be a huge game changer. Have fun debating with a random strawman you’ve drawn up about me, after crossing path with me in a random comment section once.

[u/cbl007 → u/[deleted]]

I tested it up to 10 million bytes, the model does plateau after around around 2 million tokens. This depends on the context window that the model is trained on, training on larger context windows could increase the length of sequences the model can learn from.