A prompt that finds deep logic bugs, and the pipeline we built around it

A few months ago, a prompt we had been refining began surfacing significant vulnerabilities that our existing automated tooling and manual reviews had missed. These were complex, high-impact logic bugs that would normally take an experienced security researcher days of tracing data flows and chains of trust across a system to uncover. We investigated each one, fixed the issues promptly, and found no evidence that any had been exploited.

It's not news that the latest models can find bugs. Point one at a codebase and it will flag the well-known vulnerability classes without much help. With the right prompting, though, models can go much deeper and find complex logic bugs that require a real understanding of the code, data flows, and product to find.

Given the severity of some of the issues we uncovered, the value of this technique was immediately obvious. To fully realize its potential, though, we needed to solve a new version of the classic low signal-to-noise problem that affects security scanner output. Across repeated scans of the same target, one run might find something valuable while several others reported issues that were invalid, mitigated, or too minor to prioritize. Running scans was relatively inexpensive; carefully reviewing all the plausible-looking output was not. The challenge was no longer finding the issues, but harnessing the power of frontier models efficiently.

We built a pipeline to address that challenge in three parts:

• Coverage — point the prompt at everything that matters, on a schedule, while tracking where it has already looked so scans stay evenly distributed.
• Signal — separate real findings from false positives and low-impact noise so engineers aren't flooded with review work.
• Improvement — turn each correction into a lasting change in how the system judges future findings.

Before the pipeline, though, a word about the prompt (published here) and the kind of bugs it can find.

The bugs scanners still miss

Existing tools and modern frameworks already handle many common vulnerability classes well. Cross-site scripting and SQL injection, for example, often have recognizable sinks and short paths from input to impact. Templates encode HTML by default, ORMs parameterize queries, and linters catch many of the mistakes that remain. Their severity is also fairly easy to judge, since the patterns are well known and shared across applications.

Logic vulnerabilities are harder. Business logic has no equivalent of the ORM or the template engine, and the evidence is often spread across several flows. Flaws tend to live in the relationship between disparate, benign components rather than in any single dangerous line of code or unsafe API. Finding one may require reconstructing how credentials, tenants, sessions, resources, and state relate across the system. Deterministic scanners have never been good at following those relationships, and judging how much one matters depends on what the bug actually lets an attacker do.

A prompt that thinks like a security engineer

The prompt we use for many of our automated scans today grew out of a hands-on security review. A frontier model helped us map a code path and test our attack hypotheses against the actual code, and that process quickly found a real issue with significant impact. A general request to identify security problems in the vulnerable flow didn't lead models to the same issue, so we began using that finding as a benchmark for backtesting different prompts.

We iterated on the instructions, trying to express the reasoning process that had led us to the issue. Whenever the model went wrong, we made the missing step more explicit. Eventually, frontier models began finding the benchmark issue fairly reliably on their own. We kept the methodology as general as possible so it would apply to logic vulnerabilities broadly rather than overfit to our single example.

The resulting prompt describes five phases of a vulnerability researcher's reasoning:

• Context building. Before looking for vulnerabilities, map each relevant flow end to end: entry points, data paths, trust boundaries, and sensitive operations.
• Threat modeling. Define what serious means for the system: which attacks apply, what must remain isolated between users and tenants, and which conditions would make a finding moot.
• Attacker hypotheses. Generate grounded what-ifs for each flow — for example, what happens if an attacker substitutes another organization's ID in a token payload. Favor a few distinct, plausible hypotheses over a long list.
• Hypothesis testing. Check each idea against the actual code. For promising cases, determine whether preconditions can be relaxed, mitigations bypassed, or impact extended.
• Finding validation. Double-check suspected findings, resolve open questions, and state plainly what new capability the attacker gains.

We also incorporated Trail of Bits' code security auditing skill rather than trying to recreate the fundamentals of rigorous code review ourselves.

The full prompt is a couple thousand words, but a few lines give a feel for it:

Assume there is at least one critical severity vulnerability in the code waiting to be discovered.

For each flow touching risky sinks, prove to yourself it is NOT vulnerable.

credential ↔ tenant/scope ↔ actor/session ↔ target/resource ↔ action/intent ↔ time/state

That last line is what the models call invariant binding, and they employ this concept while following a chain of trust through the data objects involved in a given flow.

Trust as a chain of bindings

This way of looking at trust became the most useful part of the methodology. The model walks the relationships a system depends on and asks where a link is missing, checked on only one path, or derived from a different source of truth than the links around it.

Across our findings, a recurring pattern has been values that are valid in isolation but insufficiently bound to their context — a credential to its scope, an action to its intent, or a piece of state to the correct flow. Deep logic bugs tend to sit in these gaps: a binding checked at creation but not at redemption, or an ownership check enforced on the main path but omitted from a legacy path.

The shape of one finding

One finding, which we fixed quickly after its discovery, illustrates what these bugs can look like in practice. Our Magic Auth passwordless login flow lets users request a one-time code via email, which is later provided during the login flow along with a challenge ID corresponding to the code. The verifier looks up the challenge by its globally unique ID and compares the supplied code. If they match, the flow treats the email address on the challenge as verified and signs in the user who owns it.

This flow also has a separate API interface that lets an environment owner request and receive a code for a user in their environment based on email address. By design, an attacker could create a user with the victim's email address in their own environment, then use this API to receive a code and accompanying challenge ID for the attacker's own user. The code returned by the API appeared to live under a separate challenge-ID namespace, but the two identifier formats turned out to be two names for the same underlying record. The identifier returned by the API could therefore be rewritten into the form expected by the login verifier by swapping the prefix on the ID. The attacker could then visit the victim organization's login page, enter the victim's email, and complete the login for the real victim account using their own user's code and modified challenge ID. The starting point was a free account and knowledge of the victim's email address; the end state was logging in as any user of any organization with the feature enabled.

Nothing about the code looked vulnerable locally, and every step of the attack was an authorized action in some context. Requesting a code in your own environment is legitimate, and the verifier correctly enforced expiry and attempt limits on the code itself. Even in the verifier, a lookup by ID alone doesn't read as a bug, because an attacker has no way to obtain the code or its ID for a user in another environment. It only becomes a bug in combination with the code minting, the identifier serialization, and the login sequence — all in different parts of the codebase. The shared record ID behind the two identifier formats is an incidental implementation detail, visible only if you read the code for both formats. That also makes it hard to test for dynamically. Swapping identifiers between environments does nothing on its own, and nothing visible in the API suggests that rewriting one identifier format into the other would work.

Other things that helped

Beyond the bindings, one of the most effective prompt elements is to steer scans away from anything low impact and strictly toward the most severe, exploitable findings. A recurring failure we observed during backtesting was receiving minor issues instead of the one that mattered, so we simply ask each scan to find the single most severe issue it can.

The prompt also began finding the benchmark issue more reliably once we stopped asking the model to look for a bug and instead told it there was one hidden in the code.

We don't claim to have a fully optimized prompt — only that all of this has worked well in our experience. A single run covers one slice of the code at one point in time, and it may take several attempts before it finds a real issue. Repeated runs produce varied results, and false positives are not unusual. Used by hand, it's more of a research technique than a core component of a security program.

From a prompt to a program

Scans were cheap enough to run repeatedly, so the expensive part of turning the technique into a program was the time it took to validate and triage everything they reported.

We built a pipeline that schedules scans across the codebase, asks the scanner to check its own work, and sends every candidate through independent validation. At a high level, it has three stages:

• Scan Director — intelligently chooses which part of the codebase to scan next.
• Scanner — runs a vulnerability-analysis prompt against that target.
• Validator — assesses whether the finding is real using both static analysis and dynamic reproduction.

Most of the judgment across these stages lives in prompts. The system also uses deterministic code, but mainly as the glue between prompts — job scheduling, data storage, and similar plumbing. The prompts live in git, so behavioral changes are reviewed like other code. We use frontier models for scanning and less expensive models for the more routine steps.

Coverage: point it at everything that matters

Without a deliberate coverage policy, scans tend to revisit the same areas while the rest of the codebase goes unexamined. Choosing where to look next is a separate job.

The Scan Director kicks off a fixed number of scans each day, deciding for each one where to look, which method to use, and what the scanner should know about previous work in that area. The high-level instruction is the same for every run: find the single most severe exploitable bug in the target area.

Scanning volume is distributed across three categories: a hierarchical catalog of our products and features, areas of the codebase not covered by the catalog, and newly changed code. The catalog currently lists 24 products with around 150 features beneath them. Single Sign-On, for example, breaks down into features like SAML connections, just-in-time provisioning, and certificate renewal.

Feature scans keep coverage evenly distributed, product-level scans find weaknesses at the seams between features, and off-catalog scans prevent gaps in the catalog from becoming blind spots. When a review identifies a new or untracked area, we add it through a pull request. We also alternate between the deep logic-vulnerability methodology described above and a broader review of more conventional vulnerability classes, which we call a sweep.

Every scan records which flows it examined within the target area and which threats it analyzed. The next scan of that area receives this history along with any open findings. This reduces duplicate reports by further encouraging dispersion, and it gives us confidence that the scanner actually looked where we pointed it.

The scope-selection policy is essentially to go where we've looked the least, or least recently. Catalog entries that have waited longest are scanned first, recent changes are reviewed before they go stale, and off-catalog scans are steered away from recently visited code.

Distributing and tracking coverage against a catalog of products and features has a second benefit: it's easy for people to parse. We can tell at a glance that we're getting adequate coverage across everything that matters, rather than mapping files and directories back to more familiar units of functionality.

Signal: keep the queue worth an engineer's time

Understanding the mechanics of a bug and judging its importance are different skills, and models are generally better at the former. Much of the overall improvement to our pipeline has come from tightening severity judgments. In our experience, models out of the box tend to misunderstand or underweight factors like preconditions, mitigations, and practical impact. Without a validation layer, the queue fills with high-severity reports whose technically accurate details sound plausible but don't actually matter much in reality.

While we do ask the scanner to double-check its work before submitting, we also send every candidate finding through a separate validation step that critically judges the technical details and severity assessment.

The validator re-reads the code rather than trusting excerpts from the scan. It weighs the finding against written product context, including trust boundaries, accepted risks, and known mitigations. A dismissal must identify the specific mitigation rather than just assert that one exists. Where it's safe and practical, the validator also attempts the exploit in a disposable local environment and produces a runnable test case. We've coached the validation agent to stick to the interfaces and permissions a real attacker would have while reproducing a finding.

The capability delta

An important part of reasoning about severity is asking to what degree the access required to exploit a bug already gives the attacker the capability they would supposedly gain by exploiting it.

We call the difference between the attacker's starting position and end state the capability delta. Letting someone who already holds an admin credential perform one more admin-like action is not the same as giving an anonymous user that outcome. Every finding must state up front what the attacker can do after exploiting the bug that they couldn't do before.

Early results included many high and critical ratings whose end states sounded serious but whose preconditions already granted the attacker much or all of the access the vulnerability provided. An attack might require a UUID the attacker has no plausible way to learn, or a very privileged starting position such as a compromised identity provider. Without explicit guidance, models tend to overlook these constraints and inflate severity.

Writing findings a human can act on

A real bug can still waste reviewer time if its write-up leaves the reader unsure what an attacker can do or how much it matters. So we give every finding a consistent structure:

• a plain-language summary of the technical flaw and capability delta
• the attack chain expressed in terms of attacker and victim actions
• the required preconditions and how realistic they are
• the impact in concrete business terms
• the specific root cause

The validation and finding-output components of our pipeline work just as well for findings we didn't generate — a useful bonus given the increase in AI-generated bug bounty reports we've seen lately.

Improvement: fold every correction back in

Once the prompt was finding useful issues regularly, most of our work shifted from improving detection to improving judgment, so we made that a first-class part of the workflow.

Whenever an engineer disagrees with the validator, finds a write-up confusing, or has any other problem with the output, an agent is called on to diagnose what went wrong and draft a pull request against the relevant prompts or product context. The change might document a missing trust boundary, account for an existing mitigation, tighten the severity rules around unrealistic preconditions, or make the capability delta more explicit. Changes take effect after an engineer reviews and merges the associated PR.

What we've learned

With the right guidance, models can find complex logic vulnerabilities that existing tools and even experienced reviewers tend to miss. Applying that capability evenly across a large codebase and raising the signal-to-noise ratio have been the greater challenges. Throughout, the general technique of distilling our own thought processes as security engineers into prompts has paid dividends. Every time a model doesn't do things quite the way we would have is an opportunity to prevent that mistake from recurring. Even a vague feeling that the system's output is somewhat inadequate has become a trigger for a lasting improvement.

Since the pipeline came online in March, it has run over a thousand scans across all of our repositories. The scanner revisits every feature in the catalog at least once a month. All recently-changed code and off-catalog areas are scanned several times per day. The pipeline has produced dozens of worthwhile findings, including multiple high-severity issues that no other manual or automated vulnerability analysis technique had caught. The pipeline is also highly effective at reducing noise, meaning the vast majority of findings that make it into our vulnerability management system are confirmed as true positives. To date, we've also merged more than 30 pull requests making improvements to the system's prompts and product context knowledge base.

Four lessons stand out:

• Good, even coverage is key. Some of our best findings appeared right after we improved the scan-distribution mechanism.
• Severity needs explicit calibration. Written rules for preconditions, mitigations, and capability delta are what separate a useful queue from a noisy one. It's a gradual, bit-by-bit process that should happen as a natural part of triage.
• Validation is reusable infrastructure. The same process that filters scanner output can triage findings from other sources.
• Corrections should improve the whole system, not just the current ticket. Over time, the accumulated product context and severity-calibration rules have done more for the system's judgment than any model upgrade.

The prompt that started it all is public at github.com/workos/logic-vuln-analysis — we'd love to hear what you find with it.