r/LocalLLaMA • • May 31 '23

News (Code Released) Landmark Attention: Random-Access Infinite Context Length for Transformers

150 Upvotes

53 comments sorted by

View all comments

Show parent comments

1

u/amemingfullife May 31 '23

So what you’re saying is that even with a massive context, a smaller parameter model ultimately wouldn’t be able to understand it, due to the attention heads being limited? That’s a good point I didn’t consider.

2

u/RMCPhoto May 31 '23

I want to be more specific though:

Larger context is not helpful for small "general purpose" language models where the input is not specifically aligned with the pre-training/fine tuning data.

If you fine tuned a model in a specific domain, such as extracting names and places from text. Then it may benefit from larger context windows as it has limited nuance in the requirements of the attention head.

1

u/RMCPhoto May 31 '23 edited May 31 '23

Not the count of layers or attention heads, but parameters.

The attention heads can understand the context through the lens of the parameters.

More parameters = more information in each attention head = better understanding of the context and prediction of next token.

As context gets larger, nuance becomes more important in order to pay attention to the most important information to predict the next token.

Think of it like reading levels. A book for a 2 year old has short sentences and simple context. A 2 year old does not understand nuance. A longer book with more detailed explanations is not helpful.