AI research

NVIDIA researchers say AI agents rarely need giant memories. Here is what they tested.

Published November 5, 2026

AI companies keep advertising bigger “context windows,” the amount of text a model can hold in view at once. A paper posted on November 3 by NVIDIA researchers argues most of that space is unnecessary. When two leading models worked with short, regularly trimmed memories, accuracy on coding, search and long-document tasks stayed about the same, and the authors estimate the work could be much cheaper to run. It is a preprint, not yet peer-reviewed, and the numbers are the authors’ own.

What happened

First, two terms. AI models read and write in tokens, small chunks of text; the paper uses about 1.33 tokens per English word. The context window is the most tokens a model can take in at once. GPT 5.6 Sol’s holds about 1.05 million, more than The Lord of the Rings.

Bigger is not free. Each new word a model writes is computed against everything already in view. The authors measured three open models on rented NVIDIA H100 chips and found that serving them with a 256,000-token window instead of 16,000 made each generated token 3.6 to 7 times more expensive. Retail prices hide this, they point out, because providers charge a flat rate per million tokens.

So they tested a cheaper habit called compaction. When an AI agent (a model that works through a task step by step, running commands and reading results) fills a set budget, it stops and writes a handoff note. Then a fresh conversation starts with only the original task and that note. For long documents, instead of pasting the whole thing in, the agent got the document as a file and searched it, reading only the parts it needed.

They ran Claude Opus 4.5 and GPT 5.6 Sol through 13 standard tests, with budgets between about 8,000 and 48,000 tokens. On the six multi-step tests, scores moved between 3.7 points lower and 5.6 points higher than the full-memory runs, and no more than 5 of 22 settings lost more than 2 points for either model. In the most extreme version, the agent took one action, rewrote its note and started over every time. Opus scored 70.2% on SWE-bench Verified, a test of fixing real software bugs, against 70.0% with full memory. GPT 5.6 Sol slipped from 84.4% to 82.4%.

On long-document tests, searching beat pasting. On RULER, a test of finding facts buried in very long text, Opus went from 62.5% to 95.7%.

Why it matters to a normal person

Piling more text into view is not the same as using it well. The authors found the best handoff notes were not summaries of everything that happened, but short briefs on the work still to do: the task, the plan, what counts as finished, and the facts needed next.

It also matters for running AI on your own computer: on an NVIDIA desktop machine, several local models took about twice as long per token late in a long input as early on. As an early independent write-up on sysnote notes, coding tools such as Claude Code and Codex already compact long sessions; this paper argues they could do it far more often.

What it does not mean

It does not mean big context windows are useless. The authors say their evidence does not rule out tasks that benefit from them. An agent asked to recall the exact result of an old coin toss will eventually lose it after enough trimming, unless the full history is saved somewhere searchable.

The savings are mostly an estimate. The headline figures assume providers would charge less for shorter windows, which none do today. At actual list prices in the paper’s own tables, compaction made GPT 5.6 Sol’s SWE-bench run cost more ($184 to $203 versus $168), while Opus got cheaper.

Smaller models struggled. At the tightest budget, all five open models the team ran on a desktop lost accuracy on the bug-fixing test; Qwen 3 Coder Next fell from 53.0% to 3.6%. And the “semantic state space models” the authors propose, AI trained to work this way from the start, are only an idea so far. The paper is new and has not been independently reproduced.

Sources

You don’t need long context — Peter Belcak, Saurav Muralidharan, Pavlo Molchanov and nine co-authors, NVIDIA (primary source; arXiv preprint, Nov 3, 2026; not peer-reviewed)
Project Moonshot: Local Frontier Intelligence — NVIDIA Research project page linked from the paper
NVIDIA’s “You don’t need long context”: Opus 4.5 and GPT 5.6 Sol keep their SWE-bench scores on 16K-48K windows, but the savings need new pricing — sysnote (independent analysis; Nov 4, 2026)