Agent Night demo recap: Brian Douglas's self-healing Pokémon agent

The last demo of the night was about Pokémon

Agent Night ran on August 12 at the Regency Ballroom in San Francisco, and the closing lightning demo went to Brian Douglas, bdougie online, a founder of Paper Compute. His topic: six months of running Pokémon as a self-healing agent. "So I have been self-healing Pokemon as an agent for like the last six months and I've learned a lot from it," he told the room.

Pokémon is the fun piece that demos well, but the real subject was what an agent needs in order to get better from its own history. Douglas previously founded Open Sauced; his current work is agent infrastructure, including observational memory pipelines built on Kafka, Flink, and DuckDB.

The setup is a coding harness and a headless emulator

The rig is unglamorous. Douglas uses a coding harness, Claude Code, with a skill for speedrunning. The game runs on PyBoy, written in Python, which can run Pokémon headless so the emulator can be handed to a harness and played without a display. In Paper Compute's open source Pokémon agent, headless mode removes the 60fps cap and all rendering, and the emulator runs roughly 100x faster than real time. The agent reads game memory, makes strategic decisions, and plays inside a stereOS VM.

Every session and every turn that passes through the harness is captured as a trace. Each turn is a movement in the game, and each one has a screenshot. The rule Douglas set for himself: the agent has to learn Pokémon live, with a thousand turns to figure it out.

Raw traces don't mean anything yet

"The real trick from this is your traces are all pretty raw," Douglas said, and that's the actual problem. So he labeled them by type: milestones, like capturing a Pokémon, separated from raw telemetry traces for up, down, left, and right. A labeled trace is searchable. A raw one is a transcript you have to reread.

He has written up the distinction: a dream is structured interpretation over traces, not magic memory. A label is an interpretation attached to events that already happened, and the trace stays the source of truth. Underneath it sits a reflection, the per-session note that reduces one run into the observations that might matter later and points back into the trace.

His write-up prints one of these labels, from a run that kept dying in Viridian Forest:

{
 "type": "perception_anomaly",
 "location": "viridian_forest",
 "observation": "the model repeatedly treats the screen edge as a wall",
 "cause": "navigation is inferred from one screenshot per turn",
 "recovery": "cross the viewport boundary and inspect the next frame"
}


Brian captures a screenshot after each turn and feeds that image into the next one, so the agent sees a sequence of still frames. It kept treating the edge of the frame as the edge of the world, while tree stumps were real blockers.

The heal button

The part of the UI Douglas actually wanted to show was anomalies. When a run fails, he can click a heal button, pass a prompt, and spin up a healing session against that failure. He was candid that this is the unfinished part: "But it's not fun when you do it manually. It'd be great if you get to self heal and like have this work on its own".

The early failures were legible once he could see them. The first one he showed was the agent failing to get out of the room; the fix turned out to be that it had to talk to NPCs, and once it learned that, it started talking to NPCs. An earlier version of this agent played for 1,000 turns without leaving the bedroom, pressing A on an empty room because it read memory address 0xC4F2 as a text-box flag when that address is actually a position in the Game Boy background tile map. Douglas' conclusion from that run: "I didn't fix the agent by writing better prompts. I fixed it by recording everything and reading it back". The generalizable version, in his words, is that silent failures which present as progress are the hardest bugs in any system.

Dreams, borrowed and applied

Douglas credited the ideas openly. He applied observational memory to the Pokémon harness after watching Abhi give a talk on it three months earlier, and built the dreams concept around Claude Code dreams, which Anthropic has since renamed auto memory. His write-up traces the lineage further: Anthropic first shared the concept with him under the name Dreams and still uses it for a managed-agents research preview, and Mastra's observational memory runs an Observer that compresses conversation into a dense log and a Reflector that reorganizes it.

The capture layer is tapes.dev, which patches the trace data through into a database where the observations accumulate. tapes sits between an agent and its model providers as a proxy, recording every request and response. It's open source at github.com/papercomputeco/tapes, and the team is working toward Agent Trace, an open specification. Douglas runs the Pokémon agent overnight and uses the stored memories to get a high-level read on what happened while he slept.

The discipline that keeps this from turning into a model writing plausible-sounding advice: "Every observation needs a receipt: the session, turn, tool result, or outcome that caused the label to exist".

Inceptions are the part that acts

A dream finds a lesson. An inception tests it. Douglas demoed inceptions as a Rust CLI that runs Pokémon across multiple fanned-out agents, and named his source on stage: "So the idea with AlphaGo from DeepMind — there was a paper about this — spinning up 10 agents, pick the best four, and then keep iterating. That's what this is doing with inceptions. It's like acting on your dream". Without it, he goes back to previous sessions and pulls the material out by hand.

The repo version of this is the evolution harness, credited to DeepMind's AlphaEvolve: it runs 10 agent variants in parallel, scores them with a composite fitness function, and keeps the winner. Its first finding was that a door cooldown of 2 beats the default of 8, confirmed across two milestones with 10 independent runs each. In the write-up, an inception replays a session at low, medium, and high reasoning with the task held fixed, then compares outcome, cost, and time. The traces showed that some parts of the run didn't need the strongest model at all.

Four layers, in Douglas' order: capture the trace, write the reflection, run the dream, run the inception.

The artifacts are public

Because the traces are labeled, they're a dataset. Douglas has published bdougie/pokemon-red-telemetry-sft on Hugging Face, 590 rows in a single train split across five domain classes, one of which is battle-action, prompted with "You are the battle advisor for a Pokemon Red agent. Answer with only the requested JSON.".

The repo's published milestones give a sense of what winning costs: roughly 200 turns to grab a starter and beat the rival, and somewhere between 80,000 and 300,000 turns to reach the Elite Four depending on strategy tier. He published a blog post the morning of the demo covering all of it, linked from his Twitter: Dreams, Reflections, and Inceptions, dated August 12.

The signal

Pokémon is over 30 years old, and a nearly perfect harness for this argument, because the agent either gets out of the forest or it doesn't. There's no eval to argue with.

The pattern Douglas is building has a strict order of operations, and most teams skip to the last step. You cannot heal a failure you never recorded, and a memory system that can't produce the receipt for a lesson is just a model with opinions. His own summary of the layers: capture gives the agent a past, reflections make that past readable, dreams find the lesson, and inceptions replay the evidence until you know which lessons hold.

Agent Night's four demos covered sandboxes, memory and agentic systems, and runtimes. This one closed the loop between them: the interesting artifact from an agent run isn't the outcome, it's the recording, and the recording is only worth keeping if something downstream reads it back.

Watch the demo and the entire event recording here: