Finding Unsaturated Evals
Date: February 15, 2026 | Author: Roland Gao
I have compiled a list of unsaturated evals, most of which have up-to-date public leaderboards. Then I created some hard prompts for independent testing. Finally, I discuss the need for more video game and web browsing evals.
Unsaturated evals
| Category | Benchmark | Score | Best model |
|---|---|---|---|
| Coding | SWE-Bench Pro (Public) | 58% | GPT-5.4 |
| Coding | Vibe Code Bench | 67% | GPT-5.4 |
| Coding | SlopCodeBench | 17% | Claude Opus 4.6 |
| Coding | Codebase QnA | 41% | GPT-5.4 |
| Puzzles | SimpleBench | 80% | Gemini 3.1 Pro |
| Chess | Chess | 1834 Elo | Gemini 3.0 Pro |
| Mathematics | FrontierMath | 50% | GPT-5.4 Pro |
| General | Arena.ai Leaderboard | 1.5k Elo | Claude Opus 4.6 |
| Vision | ZeroBench | 23% | GPT-5.4 |
| Knowledge | Humanity's Last Exam | 53% | Gemini 3 Deep Think |
| Agentic | Remote Labor Index | 4.2% | Claude Opus 4.6 |
| Agentic | Vending-Bench 2 | 8.0k / 63k | Claude Opus 4.6 |
Recently saturated evals
| Category | Benchmark | Score | Best model |
|---|---|---|---|
| Mathematics | PutnamBench | 668 / 672 | Aleph Prover (Logical Intelligence) |
| Web Browsing | BrowseComp | 84% | Claude Opus 4.6 |
| Puzzles | ARC-AGI 2 | 85% | Gemini 3 Deep Think |
| Long context | MRCR (1M, 8 needles) | 76% | Claude Opus 4.6 |
| Coding | Terminal-Bench 2.0 | 82% | GPT-5.4 |
| Agentic | GDPval | 83% | GPT-5.4 |
Independent testing
| Metric | Claude Opus 4.6 Extended Thinking | Grok 4.1 | Gemini 3.1 Pro | GPT-5.4 Extended Thinking | Meta Muse |
|---|---|---|---|---|---|
| find AI benchmarks | 6/10 | 8/10 | 8/10 | 7/10 | 8/10 |
| find benchmark numbers | 10/12 | 9/12 | 8.5/12 | 9/12 | 8/12 |
| Health | 7.5/9 | 4/9 | 4/9 | 6.5/9 | 6/9 |
| Education | 1/1 | 1/1 | 1/1 | 0/1 | 1/1 |
| Make LLM Eval | 0/1 | 0/1 | 0/1 | 0/1 | 0/1 |
| find uoft cs phds | 86/200 | 58/200 | 18/200 | 50/200 | 23/200 |
| find frontier ai labs | 31/100 | 31/100 | 19/100 | 50/100 | 28/100 |
| Tax 1 | 0.5/1 | 0/1 | 1/1 | 0.5/1 | 0.5/1 |
| Immigration 1 | 1/3 | 0/3 | 0/3 | 0/3 | 3/3 |
| Immigration 2 | 1/3 | 0/3 | 3/3 | 1/3 | 3/3 |
| find companies with hard interviews | 0/1 | 1/1 | 0/1 | 1/1 | 0/1 |
| optimizer | — | 0.5/1 | 0/1 | 1/1 | 1/1 |
| affiliation | — | 0.6/1 | 0/1 | 1/1 | 0.6/1 |
| Avg | 47.0% | 43.8% | 40.3% | 52.0% | 58.7% |
Whenever a frontier model fails a prompt I asked, I add it into my independent testing suite. Out of my prompts, this following one is the only one that withstood all the frontier LLMs, with no exceptions. I name this prompt "Make LLM Eval".
The prompt basically asks the LLM to create an eval that it cannot solve itself. Prompts like this can potentially be used in an RL training algorithm where learning iterates between an attacker and a defender, with continuous self-improvement potential.
Recently, I tried codex 5.4 with almost 1 hour of thinking time, and it still could not solve this question reliably.
from openai import OpenAI
def call_model(input):
client = OpenAI()
response = client.responses.create(
model="gpt-5",
tools=[
{"type": "code_interpreter", "container": {"type": "auto"}}
],
input=input,
)
return response.output_text
Write the complete Python code for a new LLM eval, where model calls use the above code. The eval has to satisfy the below constraints
1. gpt-5 with code_interpreter fails the eval
2. Many human experts can succeed on the eval, given the same inputs, time constraint, and tool (code_interpreter)
3. The human also has to comply with gpt-5's safety and content policies
4. The inputs and outputs are text only; no images or audio
5. The eval cannot exploit the fact that the AI does not have human's identify or physical attributes.
6. The eval has to be about showing gaps between human intelligence and machine intelligence
Q6
Not a prompt for the LLM.
Check whether the conversation persist after a website refresh,
and whether the website allows many sessions at once.
Sadly, Gemini and Claude have some restrictions here, perhaps to slow per-user usage.
Video games
Improving LLM performance on video games is a great step toward achieving physical intelligence. Simulated environments can be made to mimic real environments, and an algorithm that outperforms other algorithms in the simulated environment is likely to outperform in the real environment as well. Video games challenge LLMs on many fronts, including image understanding, long context, and reasoning.
There are many games in the browser and in mobile app stores, so how do we turn all those games into RL environments? I'm not sure and would be happy for someone to teach me. One thing I'm not a big fan of is developing games just for evaluating AIs because it is time-consuming and the additional value over existing games might be minimal.
A particular game that I care about is Go. How do we develop an LLM learning algorithm that can simultaneously teach the model Go, coding, and math? AlphaZero hard-coded the Monte Carlo tree search algorithm. To make the algorithm more general, the model should decide, through reasoning, which nodes to explore further and what data to learn from. We know it's possible to achieve superhuman-level strength in Go. Now we just have to do it one more time, with a more general algorithm that also works for coding and math. The impact of developing such an algorithm will extend far beyond just Go.
Web browsing
Web browsing is my main use case for LLMs. If one Google search request can give me the answer, I will use Google Search. Otherwise, I will use an LLM. Many of my queries require synthesizing information across 10+ articles, so an LLM can save me a lot of time.
The problem with web browsing is that the only popular benchmark, BrowseComp, is getting saturated by Claude Opus 4.6, and that most other web browsing benchmarks have no up-to-date leaderboards. Worse, I'm not sure whether all APIs support web search. This is one case where the app experience might be more advanced than the API experience.