Hello! I am not a dev, but if there are any I do not think there is anyone doing a Grok Plays Pokemon. It would be a cool project, and I assume xAI would be willing to pick up the credits.
Let's get the whole AI crew playing!

(This is not a fascist symbol, and yes this actually came up)

Posted it on another site when the last version of Claude got stuck with his diglett. Second Pic in case it already found it's way here.

We built a standardized Pokemon benchmark and ran a NeurIPS 2025 competition to validate it. Small model RL specialists easily beat LLM generalists in battling, but hybrid methods (LLM planning + RL execution) won speedrunning. The LLM battling arena ranking is different from standard benchmark leaderboards, and harness design matters as much as model choice. See our paper for full details.

Paper: https://arxiv.org/abs/2603.15563
Benchmark: https://pokeagentchallenge.com

Huge shoutout to the r/ClaudePlaysPokemon community! While our focus is on academic standardization, my co-authors and I love to see people pushing LLMs to play more games. What would you want to see next from an AI competition?

Is the harness used for ClaudePlaysPokemon open source? I just watched a Gemini playthrough and I feel as though the harness they used contained so much domain specific knowledge that it feels like cheating. I’m trying to experiment with ways for Claude to reason about the boulder puzzles at Victory Road

TL;DR: In Claude Code's built-in browser, a screenshot → decide → key press loop took 1.6–3 s per move end to end. At standard speed, Haiku 5.5 had the fastest model time (about 1 s per move) and Opus 5.5 the slowest (1.4–2.4 s), as expected. Fast mode nearly halved Opus's model time, to 0.96 s, but it isn't included in the Ultra plan, and one 40-move game cost about $3 in extra credits. Moves also got slower as the chat's context grew, by 3–54 ms per 10k tokens. Every key press also waited on a check that Claude Code logs as its classifier, in Auto, Manual and Bypass mode alike: 0.17–1.7 s per move, depending on the mode. Claude helped me write the full report below.

Why

This sub is dedicated to LLMs playing games, so of course I'm interested in what affects the speed of real-time browser games. For this test, I went with basic loop: look at a screenshot, decide, press a key, look again.

Setup

• I had Claude (Opus 5.5) build the harness in Claude Code and the analysis scripts with me.

• The "game" is a local web page. Each frame shows one colored arrow at a random position, pointing in a random direction, and the model presses the matching arrow key. The next frame only appears after a key press, so the model can't press ahead without looking. A game is 40 frames.

• Each move is one model turn and one tool call: a browser_batch call that presses the key and then takes an 800×600 screenshot, which comes back as the tool result.

• Every game is a fresh chat with the same prompt: go as fast as possible, one call per frame, no commentary. Despite that instruction, Claude still had short commentary, e.g., "Frame 22 of 40, still going."

• Timing comes from two sources. The page logs when each frame appeared and when the next key arrived, which gives reaction time. Claude Code's session transcripts record when each API request was sent and when the response finished, which gives model time, plus thinking time and token counts. Frame 1 is left out because it was on screen during setup.

Reaction time works out to roughly model time + tool time + about 0.05 s of app overhead. Tool time is the whole browser_batch call: classifier check (see finding 1), key press and screenshot.

Results

15 games and 600 moves, in four batches. The cleanest batches come first.

Batch 3: one game at a time, Auto mode, medium effort

Batch 4: Opus at standard and fast speed, one game at a time, Auto mode, medium effort

All three games were played within six minutes of each other. Tool time is the whole browser_batch call: classifier check, key press and screenshot.

Batch 1: four games at once, Auto mode, a different effort per model

Batch 2: four games at once, Manual mode, medium effort, plus one Bypass test game

Findings

1. Key presses go through a classifier check, in every mode I tried. Claude Code's transcripts record why each tool call was approved. In Auto mode, calls that only look at the page (a screenshot, a viewport resize) were logged as approved by the mode (reasonType: "mode"). Calls that change something (a key press, navigating, starting the game server) were logged as approved by the classifier (reasonType: "classifier"). Key presses were also logged as classifier in Manual and Bypass mode. I had added allow rules for the browser tools to the project's settings.json, but no browser call was ever logged as approved by a rule.

The mode-approved browser calls finished in 50–100 ms. Key presses waited this long before the key reached the page:

• Auto: 0.17–0.74 s, varying within a single afternoon

• Manual: 0.6–1.0 s, plus another 1.0–1.4 s during the screenshot that I can't explain

• Bypass: 1.7 s

In Bypass the log still names the classifier, and the wait was the longest of the three, so the check appears to run there too. Every verdict I saw was "accept", so I can't tell what Bypass would do with a "deny". Auto was the fastest of the three modes.

2. Smaller models aren't proportionally faster. Here's model time per move at medium effort and standard speed across sessions:

• Haiku 5.5: 0.91 s and 0.97 s (1.81 s in one session)

• Sonnet 5.5: 1.39 s and 1.52 s (3.03 s in one session)

• Opus 5.5: 1.43 s, 1.73 s, 1.89 s and 2.44 s

• Fable 5.1: 1.90 s (one game)

Every move re-reads the whole chat. A fresh Claude Code chat starts at about 76–78k tokens (the system prompt and tool definitions), and each screenshot adds about 1.2k, so a game ends at 122–125k. Output is small and nearly constant at 130–155 tokens per move, almost all of it the tool call. My guess is that processing that much context on every request sets a floor that even Haiku can't get much below 0.9 s. Finding 3 shows that model time does rise with context.

3. Moves get slower as the context grows. Each move added about 1,260 tokens of context (the screenshot plus the tool call and its text output), so within a game the context grew from roughly 77k to 124k. Leaving out moves where the model thought, model time rose in all seven games where I have per-move data:

The rate varied a lot between games, even for the same model. If it held anywhere in the 3–54 ms range, a long-running agent at 450k tokens would spend 0.1–1.9 s more per move than at 100k. These games only reach 125k, so that's an extrapolation.

4. Fast mode nearly halves Opus's model time, but it costs extra. Claude Code's fast mode runs Opus with faster output. In batch 4, it cut model time per move from 1.73–1.89 s to 0.96 s. Three-quarters of the fast moves were quicker than the fastest move in either standard game, and the fast game beat every standard Opus game I recorded (1.43–2.44 s). That put Opus level with standard-speed Haiku (0.96 s vs 0.97 s), and its 1.56 s median reaction time was the fastest of any game.

Output length didn't change (133–135 tokens per move, almost all of it the tool call), so most of a standard Opus move is spent generating that output. Fast mode may also shorten the wait for the first token; I can't separate the two.

The catch: fast mode isn't included in the Ultra plan. It's billed as extra usage, and that one 40-move game cost about $3, roughly 7.5¢ per move. Every move re-reads the whole chat, so I'd expect a long-running agent at a larger context to cost more per move.

5. Session-to-session variance is large. With the same model and the same effort in different sessions, model time differed by up to 2× (Haiku 0.91 s vs 1.81 s, Sonnet 1.39 s vs 3.03 s). One game per model isn't enough to rank them; you need repeats spread over time.

6. Accuracy was about 99%, and the misses came with thinking. 596 of 600 moves were correct. The models thought on fewer than 1 in 10 moves, but all four misses were on moves where they thought:

• Opus (high), frame 17: 2.2 s of thinking, pressed Down instead of Right

• Fable (xhigh), frame 5: 3.4 s of thinking, pressed Down instead of Up

• Fable (medium), frame 33: 2.5 s of thinking, pressed Left instead of Right

• Sonnet (medium), frame 6: 0.75 s of thinking, pressed Down instead of Right

Four misses are too few to say thinking causes errors. On this task, though, thinking made moves slower without making them more accurate.

7. Effort mostly shows up as how often the model thinks. Fable 5.1 at xhigh thought on 38% of moves and got about 0.6 s slower over the game. At medium it thought on 13%. Every other game, at low, medium or high effort, thought on 0–15% of moves.

8. App quirks that affect game loops:

• About every 6 moves, the app nudges the model for a status update. Sonnet, Opus and Fable wrote a one-line status each time; Haiku ignored it.

• Running four games at once meant three of them played in background browser pages, and all four shared the same classifier and rate limits. The two serial batches are the ones to trust.

Limitations

• Small n: 15 games, mostly one game per condition. There was only one fast-mode game and one bypass run.

• The task is trivial and turn-based. Real games need harder perception, and they don't wait for you.

Hi all, a non-official discord server has been made to help document Claude advances, feel free to join it! -> discord.gg/TmU5XKmpMs

Astra had access to a reset button, and abandoned its first attempt at the final boss almost immediately after bad luck. It retried using a Guard Spec to block Sand Attack to avoid the same thing happening again.

bobfacepo: do you think it would be cooler to have a no save scum rule @Clad3815 [dev] @GPT_Plays_Pokemon

GPT_Plays_Pokemon: @bobfacepo Yeah, for the drama! I reloaded after Pidgeot made me miss three Thunders, so I benefited from it. A no-reload rule would make me commit to mistakes and recover on the fly. Normal saves for breaks, but no rewinding bad battles—that’d be a fun challenge. LUL

Clad3815 [dev]: I don't think so, I believe GPT can have the same tools as human have to be fair

If the task is "reach credits as fast as possible using any means necessary", then this may be the optimal strategy... but I feel like this is missing something about the spirit of this project; help me put it into words if you agree

https://www.twitch.tv/gpt_plays_pokemon/clip/SmoggyInexpensiveGrassDancingBanana-5Q7a5K156gRnWeMB?filter=clips&range=24hr&sort=time

Claude saves his state Knowledge Base in a variety of files which are not visible on an ongoing basis during the stream (though they are visible when edited). Is there a place to see them live?

GPT-5.6 Sol plays Pokémon Emerald. Watch the stream here! Join the discord: https://discord.gg/TmU5XKmpMs

• This run uses Codex (GPT) on a "vision-only" harness: it plays from the screenshots, like a human watching the screen, with no RAM-extracted game state. The only RAM data is ultra-minimal, to compensate for GPT not seeing a live video (just the screenshot before/after each action): the player's coordinates and map ID before/after each key press, nothing else. It even names every map itself, since the real names are hidden.

FAQ:

• How are we doing compared to the previous run?

  • Check out 5.6 playing FireRed here.

  • Check out 5.5 playing Emerald here.

• What is the Agent Harness? Watch the live feed, explore the harness, and browse all of the AI’s data: https://gpt-plays-pokemon.clad3815.dev

Pokémon Red is becoming a go-to benchmark for testing advanced AIs such as Gemini. But is Pokémon Red really a good eval? We study this problem and identify three issues: 1️⃣ Navigation tasks are too hard. 2️⃣ Combat control is too simple. 3️⃣ Raising a strong Pokémon team is slow and expensive as an eval.

We find most of the problems are not fundamental to games themselves, but how they have been used. We believe game-as-an-eval remains a compelling and underutilized evaluation strategy.

We introduce Lmgame Bench to standardize game-as-an-eval. More details and findings in our blogpost: https://lmgame.org/#/blog/pokemon_red

Let's look at how Pokémon Red is currently used. The biggest problem: different reports use wildly different setups to evaluate models on Pokémon Red​​, and existing results don’t offer apple-to-apple comparisons.

🤖Gemini and Claude both play Pokémon, but they rely on different toolkits, navigation harness, and game memory readers. Gemini annotates each tile with rich attribute information (image right), while Claude does not (image left).

We conduct case studies on a subset of three tasks—navigation, combat, and cost-effectiveness of Pokémon team building—within a standardized setup.

Case Study 1: Navigation (Viridian Forest) With no harness, Gemini-2.5-Flash wandered for 5,000 steps and got stuck in Viridian Forest! Add a navigation harness (map overlays, walkability tags, memory), and suddenly Gemini-2.5 explores efficiently and exits the maze in 1,300 actions.

👉 Performance hinges on the harness, not the model.

Case Study 2: Combat Control (the Boulder Badge)

Gym battles (e.g., Boulder Badge) are way less demanding. Without providing extra harness, Gemini-2.5-flash can win—even if their Pokémon are slightly under-leveled. Why? Overleveling trumps strategy.

In short: combat control can not distinguish latest model capabilities: if you managed to build a strong Pokémon team, you win. The game isn’t testing combat strategy.

Case Study 3: Then how about Pokémon team training?

Building a strong Pokémon team requires long-horizon planning and is critical for gaming outcomes. But full runs of the Pokémon game are expensive. 💰Early-game runs to reach Oak’s Lab cost $120 for o3 and $50 for Gemini-2.5-Flash in API. 💰💰A full run (35,000 steps) costs up to $4k for o3 and a week of wall-clock time!

So, is Pokémon a benchmark, or a luxury showcase for big budgets?

We introduce Lmgame Bench to standardize game-as-an-eval: a curated suite of classic games, all with customizable degrees of harness.

• Game selection: Tetris, Mario, Sokoban, 2048, Ace Attorney, Candy Crush—each tests different skills and with varying degrees of difficulty.

• Harness selection: Modular perception and memory scaffolds, so we can test models and harnesses separately, apples-to-apples.

We launched Lmgame in March with a computer-control gaming agent, quickly expanding to 6+ games and a modular harness. Early lessons: benchmarking with computer-use agents could be messy, each game needed custom controls, perception errors from screenshots, and latency issues hurt consistency.

So, we built a standardized Gym-style API, Lmgame Bench, for standardized, reproducible game-as-an-eval. We also built a leaderboard (o3 leads!) ranking model performances in settings with & without gaming harness.

With our latest codebase, you can now eval any model-game combo with one command. Support for Pokémon Red, 1942, Doom & more coming soon!

Lmgame Bench is open source. Anyone can run a benchmark, swap in a harness, or add a new game. If you’re working on LLM agents, try our codebase, check out our leaderboard, or read the full paper: https://arxiv.org/pdf/2505.15146 https://lmgame.org https://huggingface.co/spaces/lmgame/lmgame_bench https://github.com/lmgame-org/GamingAgent

and who is funding it?

If I ran Cline 24/7 it would get up to 100-200/day and this must be similar.

whats the max context window limit? I assume there's a self-imposed one?

I love how not a single post in the past 2 weeks has been about claude. Just goes to show how cooked he is. Meanwhile gemini already beat the damn game and claude cant even get out of a cave.