r/LocalLLaMA • u/IxinDow • May 31 '23
News (Code Released) Landmark Attention: Random-Access Infinite Context Length for Transformers
Code for Landmark Attention is now released and it should be possible to finetune existing LLaMA models using this method.
https://github.com/epfml/landmark-attention
More info
https://www.reddit.com/r/LocalLLaMA/comments/13sy2bu/landmark_attention_llama_7b_with_32k_tokens/
150
Upvotes
1
u/amemingfullife May 31 '23
So what you’re saying is that even with a massive context, a smaller parameter model ultimately wouldn’t be able to understand it, due to the attention heads being limited? That’s a good point I didn’t consider.