When the action succeeds and the audit write doesn't

Almost every audit logging guide tells you which events to capture and how to shape the payload. Very few tell you how to make sure the event is actually there when someone goes looking, which is the only property that makes an audit log worth having.

Here is the code that ships in most applications:

  
await deleteDocument(documentId);

await auditLogs.createEvent(organizationId, {
  action: 'document.deleted',
  actor: { type: 'user', id: user.id },
  targets: [{ type: 'document', id: documentId }],
  occurredAt: new Date(),
});
  

Two writes, two systems, no relationship between them. That is a distributed transaction with none of the machinery, and it fails in three directions.

Three ways this breaks

The silent gap. The document is deleted. The audit call times out, or hits a rate limit, or the sink is briefly down. The user's action completed and no record of it exists. Nobody finds out until an auditor asks about that document six months later, and the honest answer is that you do not know who deleted it.

This is the failure everyone has and nobody sees, because of what happens next in real code. Failing the user's delete because a logging call failed is unacceptable, so the call gets wrapped:

  
try {
  await auditLogs.createEvent(/* ... */);
} catch (err) {
  logger.warn('audit write failed', err);
}
  

That catch block is a silent-drop path you built on purpose. It is the right instinct and the wrong implementation.

The phantom event. Flip the order to log first, and you get the opposite problem. The event is written, then the delete fails or its transaction rolls back. Now your audit log asserts that a document was deleted when it was not. A gap is bad. A record of something that never happened is worse, because the whole artifact exists to be trusted as evidence.

The duplicate. Add a retry to fix the first problem and you create the third. A request that times out after the server processed it looks identical to one that never arrived. Retry, and a reviewer sees two deletions where there was one.

Why this failure is different

Every system that writes to two places has this problem. Audit logs are worse for two reasons.

The failure is invisible by construction. A dropped metric shows up as a dip on a graph. A dropped audit event produces a log that looks exactly like a correct one, because absence of evidence is what an uneventful period looks like. There is no shape to the hole.

And you cannot reconstruct it. If your analytics pipeline drops a day, you replay it from source. The audit event was the source. The state it recorded is gone, the request is over, and nothing else in your system knows that a person, rather than a process, took that action.

So the reliability of the write path is not an implementation detail underneath audit logging. It is the feature.

Decide the policy before you write any code

One question comes first, and most teams never answer it explicitly: if the audit write fails, does the user's action fail?

For the large majority of events, no. Blocking a customer's work because a logging sink is degraded is a bad trade, and the rest of this article is about not having to make it.

For a small set, yes. If you are in a regulated flow where the record is a legal requirement rather than a convenience, failing closed is correct, and you should know which events those are before an incident forces the question. Write the list down.

Either way, a failed audit write is an error, not a warning. It belongs in your alerting with enough context to replay it, not in a warn line nobody reads.

Step 1: Write it in the same transaction

If your audit events live in your own database, the fix is short. Write the event inside the same transaction as the action:

  
await db.transaction(async (tx) => {
  await tx.delete(documents).where(eq(documents.id, documentId));

  await tx.insert(auditEvents).values({
    action: 'document.deleted',
    actorId: user.id,
    targetId: documentId,
    occurredAt: new Date(),
  });
});
  

Both or neither. The silent gap and the phantom event both disappear, because the database is enforcing the thing you were hoping for.

This works right up until the audit trail needs to leave your database, which it will the first time a customer asks for their events in Splunk. The moment the sink is an external service, you are back to two systems and no shared transaction.

Step 2: The outbox

The standard answer is a transactional outbox, and it is standard because it keeps the guarantee above while allowing the destination to be somewhere else.

In the same transaction as the action, insert the event into a local table. Nothing is sent yet:

  
await db.transaction(async (tx) => {
  await tx.delete(documents).where(eq(documents.id, documentId));

  await tx.insert(auditOutbox).values({
    payload: {
      action: 'document.deleted',
      actor: { type: 'user', id: user.id },
      targets: [{ type: 'document', id: documentId }],
      occurredAt: new Date().toISOString(),
    },
    deliveredAt: null,
  });
});
  

A separate worker drains it:

  
SELECT id, payload
  FROM audit_outbox
 WHERE delivered_at IS NULL
 ORDER BY id
 LIMIT 100
   FOR UPDATE SKIP LOCKED;
  

Deliver each row, then set delivered_at. If delivery fails, the row stays and the next pass picks it up. If the process dies mid-batch, SKIP LOCKED means another worker takes over without double-processing.

What you have bought is not just retries. It is visibility. Before the outbox, a lost event and a delivered event were indistinguishable. Now an undelivered event is a row you can count, and the question "is anything missing?" has an answer.

Which means the alert to build is not queue depth. It is age of the oldest undelivered row. A backlog of ten thousand events that is draining is fine. One row stuck for four hours is a gap forming in slow motion, and depth alone will not show it to you.

Step 3: Make delivery idempotent

The outbox guarantees retries, so it guarantees duplicates unless delivery is idempotent. A timeout tells you nothing about whether the server processed the request, and the only way to retry safely is to make the second attempt collapse into the first.

Send a stable key derived from the outbox row, so every retry of row 48213 carries the same key:

  
await auditLogs.createEvent(organizationId, event, {
  idempotencyKey: `audit-outbox-${row.id}`,
});
  

WorkOS Audit Logs supports this directly through an idempotency-key header, and if you do not send one it derives a key from the event content. That default protects you from the naive case, where a network-level retry replays an identical request. Supplying your own key is stronger, because you decide what counts as the same event rather than inferring it from the payload, and two genuinely separate actions that happen to serialize identically will not be merged.

Step 4: Prove the trail is complete

With the first three steps you have removed the known failure modes. You still cannot answer "is my audit trail complete" without checking, and an audit trail you have not verified is a claim rather than a control.

Two checks are enough for most teams.

• ‍Reconcile counts on a schedule. For a bounded window, count the business actions that should have produced events and compare against events delivered. Documents deleted yesterday versus document.deleted events for yesterday. Divergence is a bug, and it is the only way you will catch an event type that a code path forgot to emit, which no amount of delivery machinery protects you from.‍
• Test the failure path deliberately. In staging, stop the audit sink, perform a handful of auditable actions, bring it back, and confirm every event lands. This takes twenty minutes and is the difference between believing the outbox works and knowing it does.

What this leaves you

Two things stay yours no matter which audit backend you use: the outbox table and worker, and the reconciliation job. Nobody can write those for you, because they live inside your transaction and your domain.

What you can hand off is the far end. An idempotent ingestion API so retries are safe, storage that stays queryable as events accumulate, retention, and delivery onward into whatever SIEM your customer runs. That is the part that turns into a project if you build it, and configuration if you do not.

The distinction worth keeping in mind is that a managed audit backend does not make your trail complete. Your write path does that. What it does is make sure that once you have got the event as far as the API, it is still there and still findable when someone finally comes asking.