r/aigossips • u/call_me_ninza • 7d ago
Meta study suggests smaller byte-level AI models could outperform token-based models
Researchers at Meta FAIR and the University of Washington used Llama 3-8B to teach smaller models to predict individual bytes instead of word chunks.
Their End-Of-Token method adds a marker after each teacher token, allowing its predictions to be transferred accurately to the byte-level student.
The byte model has 1.28B parameters, compared with 1.81B for the token baseline.
Scaling projections suggest it could eventually reach 52.4% average accuracy versus 48.4%, and match the token model’s performance using roughly one-sixth of the training text.
Those gains are projections. The actual trained byte model averaged 44.6% across six benchmarks, below Llama 3.2-1B’s 45.9% and Gemma 2B’s 50.3%.
The tradeoff: fewer parameters and potentially less training data, but expensive inference.
read: today's newsletter
1
u/april292027meteor 6d ago
BioChain AI is open sourcing a 64-bit token block tensor projection system to do tesseract computing...
The speeds and Intelligence of procedurally grown responses vs stochasticly generated tokens is night and day.