Sentry's Greg Pstrucha on why a better prompt won't fix your agent's code

At the AI Engineer World's Fair in San Francisco, I sat down with Greg Pstrucha, a staff software engineer at Sentry, on the day he gave his talk "Stop Prompting". The title is the whole argument. If the code your agent hands back keeps making the same elementary mistakes, the answer is rarely a better prompt. It's tooling in the coding harness that raises the floor of what the agent can produce in the first place.

Sentry is not a place where code goes unread. The team still reads a lot of code, because breaking production for customers is expensive, and they aren't doing full-on loop engineering. What Greg is optimizing is the part in between: the repetitive feedback engineers give agents over and over, which he thinks can largely be swept away by deterministic rules, things like linters, good tests, a proper type system, and skills that steer the agent. Sentry's own description of the talk frames it as tooling, type systems, and frameworks enforcing limits on the slop AI agents produce inside large codebases.

The cheapest version of that is embarrassingly cheap:

"You can spend half an hour today and you're going to significantly improve the output by just adding good linters."

The policies a linter can't hold

Deterministic rules get you the floor. Above the floor sit the judgments Greg calls the policies of your codebase, the things that resist codification:

"there are these qualitative measures that are going to be hard to codify in any reasonable way. And those are what I would call like policies of your code base, things such as you shouldn't write too many tests, but you should write stronger tests."

Stronger has a specific meaning here. A stronger test is one that doesn't try to test behavior that isn't actually important. His example comes from writing evals, where the temptation is to assert on strings:

"I would rather have good tests that use LLM as judge to say whether the output of the agent is correct as opposed to match like assert whether a string had a substring in it because that can be extremely misleading to the actual output that are more semantic."

Asked how to split deterministic tests from evals, he said he doesn't have a clean answer, and that you want both: a good set of tests as the baseline, with evals reserved for the agentic code and LLM chats where plain assertions stop working.

An eval that stops at the root cause is only half an eval

Greg works on Seer, Sentry's debugging agent, which uses issue details, tracing data, logs, and profiles to troubleshoot errors and performance problems. Seer is mostly tested through evals, an extensive harness of input and output combinations checked against the expected outcome.

The interesting part is where the expected outcome sits. Seer's Autofix scans issues as they arrive, finds root causes, and can generate a fix and open a PR, on GitHub or as a merge request on GitLab. So the eval can't stop at the analysis. What Sentry checks is whether the agent gets the root cause right, then whether it produced the correct PR artifact, in the correct source control system, at a high enough code quality bar.

Evaluating an agent that writes code takes the same apparatus you would use to review your own code. The linters and tests are the grading rubric, not a developer nicety sitting outside the agent loop.

All agents are coding agents

Seer writes code twice over. It writes code as its output, and it writes code it runs itself to pull context and telemetry out of Sentry's public APIs, a mechanism the team calls code mode. Those throwaway snippets are held to the same standard as anything else:

"That code itself, that snippet of code must be correct, must be linted, must match the API schema that Sentry presents."

Which is where his framing lands:

"This is what we say is that all agents are coding agents."

It follows a thesis a lot of companies now share: agents do better when they are writing code or working against developer interfaces and APIs. If that holds, the tooling investment doesn't split between "our coding agent" and "our product agent." It's one investment.

The metrics that didn't survive contact

I asked what he had tried that didn't work, because that's usually the more useful half of the story. He gave two, both from the same family:

"There is a little bit of a snake oil when you try to put a quantitative metric on something that's inherently qualitative."

The first was cyclomatic complexity, a metric that estimates how maintainable and testable code is from how many branches it takes and how deep the call stack goes. The second was test coverage:

"Using test coverage as a number that says how well the code is tested, which in my experience doesn't really tell you how well the code is tested, just how many tests you have."

This is Goodhart's law with a much faster optimizer attached:

"They give a numerical target for the agents that then the agent can just drive down or up depending on the metric."

A proxy metric doesn't capture the thing you wanted it to capture, and Greg is blunt about the alternative:

"I don't know how to make a good measure of quality other than use engineering minds for that."

That's an ugly answer if you were hoping to buy your way to an autonomy dashboard, and it's the one that matches how his team actually works.

What would have to be true to stop reading the code

Since Sentry still reviews code by hand, I asked what would have to change for that to stop. His answer treats review as a spectrum rather than a policy: on side projects he barely reviews, because the stakes are low; at Sentry he reviews more, especially on the critical path, where a mistake hurts. Sentry already runs automated reviewers, Cursor's Bugbot plus their own code review tool, and Greg runs a review locally before opening a PR.

The signal he's waiting on is empirical:

"if those stop giving me actionable feedback and if my code review that I internally run locally before creating a PR stops giving me feedback and I keep reading the code and I'm like that's good. Then I'm going to be more comfortable with not reading the code."

He wants the endpoint:

"I think there is a future hopefully where you just don't have to have code reviews. It just goes straight to main, straight to main. And I know some people already live that future. I just don't see it yet."

What he's watching for is a new generation of coding harnesses that treat measuring quality as a first-class part of the framework, rather than something each team bolts on afterward. He knows of a few, and he's waiting to see whether any of them can prove to him that he no longer has to read the code.

The gap is infrastructure, not intelligence

Internally, Sentry's tooling is unglamorous: vanilla Claude Code and Codex, with some engineers on OpenCode through API access. They experimented with cloud coding agents like Devin, and Greg's verdict was that it was fine but not there yet for his workflow:

"you're working on a mature code base. Multiple services. You have to like really really create your workspace to work. And doing that sort of on a whim in sandboxes just didn't really work yet for me."

He's careful about where he assigns the blame: this is a tooling and infrastructure problem rather than a model problem, and he expects it to be solvable. For Sentry, the capability showed up before the plumbing that would let a mature, multi-service codebase use it.

Which is why the half hour of linters is the right first move. Everything else here, the harness, the eval framework, the cloud sandboxes, is still being worked out in public on a timeline nobody can call:

"if you look at what we've been doing a year ago with agents, it was like it's so different. I can't even guess what we're going to be doing 1 year from [now]. All these predictions don't even matter cuz things change so much and so fast."

Half an hour, good linters, today. That part doesn't depend on which harness wins.