More than 4096 context length?

[StoyanStAtanasov]

Is it possible to have larger context as this allows to do more complicated things with smaller models?
A lot of the negatives of a smaller model can be rectified by pushing more data into the context. For example: Help pages, datasheets, examples, thinking rules, longer conversations trying to fix an issue, etc.

Please excuse me if this is the wrong place to ask this question, but very rarely the context is discussed. Thanks in advance.

[jon-tow]

Sure; You just need to fine-tune to a longer context :)

[StoyanStAtanasov]

@jon-tow You are joking right?

[jon-tow]

Nope; see https://github.com/kyleliang919/Long-context-transformers

[NPap0]

1. Training code has to change
2. Data that you fine tune the model with after training has to change

So no, nothing can be done user-side to change attention span. (But maybe you can summarize blocks of text with the model so you can then feed the already summarized text as a whole to do your thing)

[mallorbc]

Pretty sure the answer is no due to how positional encoding is done.

[jon-tow]

@mallorbc The model uses RoPE - a relative attention mechanism. See https://blog.eleuther.ai/rotary-embeddings/

[mallorbc]

@jon-tow So models like GPTJ can be finetuned and generate more than their sequence length? Whenever I try to generate sequences for GPTJ I have issues. Maybe that is something else unrelated.

[Ph0rk0z]

A lora will not work for this? Would you have to retrain the model from scratch on long context as you have done?