Daisy Hollman points out that while model capabilities have exploded over the past year, context window sizes have largely stagnated: the first 1 million token context windows appeared in late 2024, the most advanced models in February 2025 had 1 million tokens, and it's still the same now. This means that as tasks become more complex, engineers must be increasingly smart about what they choose to put into the context.
The context window is the set of tokens the model can see when predicting the next token, and it's the space for customization. You can't throw the entire codebase in; you must follow the zero-overhead principle of 'don't pay for what you don't use.' Every customization competes with the workspace, and the more you pile up front, the less space remains for the actual task.
KV caching imposes hard constraints: to predict the next token, all previous tokens must be consistent, causing some operations to cost 10 times more. Cursor early on tried LRU caching on Cursor Rules but quickly found the cost prohibitive, indicating the problem is more subtle than simple caching.