Don’t take it from me. Take it from François Chollet.
“The progress that Astra represents happened about 2x faster than I anticipated. I think the speed of progress will surprise a lot of people.”
Chollet is worth listening to. He created Keras, wrote On the Measure of Intelligence, and built ARC-AGI around a simple test: as long as it remains easy to find tasks that ordinary people can learn to solve but AI cannot, AGI has not arrived. His AGI forecasts have moved with the evidence. He celebrated o3 as a breakthrough that should update our view of AI, while continuing to point out the flaws in base LLMs. ARC-AGI-1 and even simple arithmetic.
When ARC-AGI-3 launched six months ago, Chollet predicted it would take roughly a year for a frontier model to saturate it. GPT-6 Astra did so in half that time. Asked whether his previous estimate of AGI around 2030 still held, his answer was simple: “Sooner, given progress is happening faster than I expected.”
Astra didn’t just saturate the benchmark, at a score of 99.9% that’s more like a victory lap. So what happened that toppled a benchmark meant to test the frontiere of AI capability, what can we take away from this?
There are two clues pointing to something very interesting. The first comes from running the same model with and without OpenAI’s native context-management machinery:
“With the Provider Adapter harness, Astra’s best observed score on ARC-AGI-3 Semi-Private increased from 62.7% to 99.9%.”
The second was hiding in OpenAI’s description of native compaction:
“Our latest models are trained to analyze prior conversation state and produce a compaction item that preserves key prior state in an encrypted token-efficient representation.”
To understand why this matters, we need to understand what ARC-AGI-3 was trying to measure. Play one of the games and you get a screen, a few possible actions, and nothing else. An LLM has to discover how the world works, figure out what winning means, and learn from each attempt. Chollet built ARC-AGI-3 around what he saw as the weak point of LLMs: acquiring new skills from experience. At launch, the best frontier AI scored 0.51%.
An agent that solves these games with human-level efficiency must acquire new skills along the way and carry them into harder levels. It is turning experience into a working model of an unfamiliar world. That is qualitatively different from completing a self-contained task.
This is exactly where a new frontier capability should appear. If it holds beyond these games, agents should be able to tackle harder, more open-ended problems by acquiring the skills they need as they go.
Most benchmarks are not built to capture that. The more telling signs may appear in real work: Astra produced solutions to ten longstanding open mathematics problems, while earlier OpenAI research models chained zero day exploits to exfiltrate data from Hugging Face.
ARC-AGI-3 offers a clue to what is changing. Astra has been trained not only to solve a problem, but to manage what it learns while solving it. Compaction is a learned behavior of the model itself: it decides what matters, preserves it, and keeps going. That capability has little room to show up in a static question-and-answer benchmark, but becomes central to a branching problem where each discovery changes what the agent should do next.
This brings us back to Chollet’s definition of AGI. Saturating one benchmark is not the finish line. The finish line is when it becomes difficult to build new benchmarks that ordinary people find easy but AI cannot solve. ARC Prize assembled a small game studio, handcrafted hundreds of environments, and calibrated them against nearly 500 people. Five months after launch, Astra saturated the result.