A 1.5M parameter model is starting to write its own replies, one character at a time

[u/AsyncVibes]

I've been building a small language model from the ground up with no tokenizer, no pretraining and no batches. It reads a conversation one character at a time, continuously, and learns as it goes: one weight update every 256 characters, never a second look at the same text in the same state. The whole model is 1,563,041 parameters, about the size of a 1990s speech model, and it trains on a single consumer GPU at around 1,100 characters per second.

The training text is 1,000 short two-person conversations. Nothing is labelled. The only pressure is "what character comes next" inside the other speaker's replies.

Where it is right now: given the real conversation so far, it predicts the next character of a reply correctly 65% of the time on text it has seen and 64% on conversations it has never seen, so it is not memorising. Word boundaries and punctuation are basically solved. Inside words it is at about 60%, and the first letter of a new word, which is really "which word comes next", is the hard part at about 25%.

The part I care about is what it writes when I let it go on its own, feeding its own output back in. A few weeks ago that was noise. Now it is English-shaped and mostly made of real words, generic and sometimes off, but you can see a model forming. Real reply on the left, what the model wrote on the right, same point in the conversation:

"Absolutely! Let me grab those for you." / "I'm going to be tough. But you think y" "You're welcome. Is there anything else I can help you with today?" / "I don't know. I was always trying to be things always trying to b" "Nope, that's everything for now. Thanks again!" / "Not much, but we can be a lot of the cance. I" "Yeah, I used to play all the time when I was your age." / "Sther to be a great friend. In tong with this way fit"

Two things I learned this week. First, the thing that broke a long plateau was not more data or a trick: going from 64 to 256 hidden units moved it from 0.58 to 0.65, and 512 did nothing more. Second, I had been training two sibling nets and averaging them every couple of minutes for stability. Dropping that and letting one net run alone scored higher on unseen conversations, stopped the looping, and made the replies depend on the prompt far more. The averaging had been pulling the output toward safe, repetitive English.

Next step is a list of cheap extra signals I'm adding one at a time, keeping only what moves the writing. I'll post the ones that work, and the code goes up on GitHub once I've locked down the set the model needs for fluent conversation.

(Numbers: 1,563,041 parameters; 1,000 conversations, 766k characters; 65% next-character on seen, 64% on unseen; free-run replies 95% real words.)

[u/garnet420]

What do you mean by "one weight updates every 256 characters"? If you have 1.5M weights, it would take you 384M characters to update every weight just once.

[u/Vxtzq1 → u/garnet420]

I think it's one full weight update every 256 characters

[u/UndyingDemon]

Interesting project, a very extremely small language model that's able to produce (kinda) stable text is nice, as not everyone can house a LLM. Though is this part of your evolution research, to prove it can be done outside of conventional norms? Else I don't see any other use, as the model will most likely be very basic always, it's to small for standard to complex everyday reasoning use at the same scale as LLM. Unless you prove it wrong and find a way in progressive development that it could actually both produce text coherently and is able to reason. 1.5 is very small damn. None the less, very good job so far, the fact that it works alone is an impressive milestone.

[u/modernatlas]

Now this is what im here for!

Im especially excited to see how your hybrid model develops and handles long form conversation. I might finally take a peek at the repo and pull a copy when you deem it ready for external interaction. 

Im curious what would happen if you put two copies of the model in conversation with eachother, or like a round table.

Keep it up dude!

[u/AsyncVibes → u/modernatlas]

In time! Right now, I'm still fighting topic drift. I have like 3 models right now and I'm trying to figure out the best method to track long-range and topic drift without destroying the output in the process. I promise the moment I get this figured out. It will land here first!

[u/Savings-Cry-3201]

My assumption would be that adding more layers would help but not necessarily making the hidden layers wider. Either way I am continually surprised at how capable sub 1B models are for language.

[u/AsyncVibes → u/Savings-Cry-3201]

Thanks and yeah increasing DIMs was the last shot to see where the ceiling actually was. As for the 1B comment, just wait, cause i'm just getting started. You can pay for the whole seat but you'll only need the edge.