All terms
Glossary
Context window
The amount of text a language model can consider at once for a request, measured in tokens.
Everything the model should see has to fit: system instruction, conversation, supplied documents and the expected answer. When it is full, something has to give.
Large windows do not solve the problem by themselves. Models attend to the beginning and the end more reliably than the middle, and every extra token costs money and time.
How you notice it
- Long documents get truncated.
- Cost per request grows with the amount of text.
- The model misses details that were provided.
Frequently asked
Why does a full context window get used poorly?
Because models use material at the beginning and the end more reliably than in the middle. Ten relevant excerpts produce better answers in practice than a hundred mediocre ones, and cost a tenth.