Monitoring and Operations

This page covers running the sync engine in production: what to watch, what to alert on, how to deploy it, and how to recover when something goes wrong. It assumes you have completed the quick start.

A stalled sync is silent. Nothing errors, no request fails, the tables simply stop changing, and every query keeps returning plausible-looking results. It becomes unrecoverable at a deadline, because the events API retains 90 days of history and a cursor older than that cannot be replayed.

So the single most important alert is poll age, not error rate.

npx workos-sync-postgres status

{
  "production": {
    "cursor": "event_01HXXXXXXXXXXXXXXXXXXXXXX",
    "rangeStart": null,
    "started": true,
    "cursorAgeDays": 0.0004,
    "lastPolledAt": "2026-10-22T18:04:11.902Z",
    "pollAgeSeconds": 3.1,
    "lease": {
      "holder": "sync-7f9c4:1:a3f1b2",
      "expiresAt": "2026-10-22T18:05:11.902Z"
    },
    "rejected": 0,
    "streams": [
      {
        "stream": "users",
        "rows": 14203,
        "deleted": 87,
        "lastSyncedAt": "2026-10-22T18:04:09.471Z",
        "stalenessSeconds": 41.2,
        "lagSeconds": 2.1,
        "lagSampleSize": 12
      }
    ]
  }
}

Three of these numbers deserve an alert, and two of them look similar but mean different things.

• pollAgeSeconds is how long since a poll last completed without error, whether or not it returned anything. If it climbs, the process is down, stuck, or failing every request. Alert on this. It is null until the first poll.
• cursorAgeDays is the age of the newest event applied. It measures distance from the 90-day cliff, not liveness: an environment where nothing happens for a week will show seven days here while being perfectly healthy. The engine warns at 75 days and logs an error past 90. Alert well before that.
• rejected is how many events the tail has set aside because they fail deterministically. The tail kept going, so poll age looks healthy, but the objects those events describe are missing or stale until you replay them. Alert on this too. See Rejected events.

lease names the process currently tailing this environment and when its claim lapses if it stops renewing. It is null when nothing is listening, which alongside a climbing pollAgeSeconds tells you the poller is gone rather than stuck.

status opens a database connection and queries every stream, which is the wrong shape for something that runs every few seconds. listen serves the same poll age from memory when WORKOS_SYNC_HEALTH_PORT is set. The container image sets it to 8080.

curl -s localhost:8080/healthz

{
  "state": "listening",
  "fatal": null,
  "startedAt": "2026-10-22T17:58:02.114Z",
  "healthy": true,
  "environments": {
    "production": {
      "lastPolledAt": "2026-10-22T18:04:11.902Z",
      "pollAgeSeconds": 3.1,
      "lastEventAt": "2026-10-22T18:04:09.471Z",
      "cursor": "event_01HXXXXXXXXXXXXXXXXXXXXXX",
      "lastError": null,
      "counters": {
        "ticks": 71,
        "events": 42,
        "created": 3,
        "updated": 38,
        "skipped": 1,
        "deleted": 0,
        "errors": 0
      }
    }
  }
}

The status code is the verdict. It returns 200 while the process is starting and while every environment has polled successfully within WORKOS_SYNC_HEALTH_MAX_POLL_AGE_MS, which defaults to 60 seconds. It returns 503 once any environment’s poll is older than that, once the engine has stopped on a fatal error, or once listen has returned. No request to it touches the database, so it keeps answering when the database is the thing that is down.

With the health port set, listen also serves GET /metrics in the Prometheus text format, from the same in-memory state. Every per-environment series carries an env label.

- alert: WorkOSSyncStalled
  expr: workos_sync_poll_age_seconds > 300 or workos_sync_up == 0
  for: 2m
# A listener that has never polled successfully has a ticks series but no
# poll-age series, so nothing above matches it. The `for` is the startup grace.
- alert: WorkOSSyncNeverPolled
  expr: workos_sync_ticks_total unless workos_sync_poll_age_seconds
  for: 5m
- alert: WorkOSSyncCursorNearRetention
  expr: workos_sync_cursor_age_days > 60

Every table carries _updated_at (source time) and _synced_at (write time), which makes the important signals plain SQL.

Lag is how far behind the mirror is. Measure it only over rows that changed recently. Across every row, _synced_at - _updated_at reports how old the objects are, not how late the sync is: a mirror that has just backfilled year-old users would report a lag of a year while being perfectly current.

SELECT AVG(_synced_at - _updated_at) AS mean_lag, COUNT(*) AS sample
  FROM workos.users
  WHERE _updated_at > NOW() - INTERVAL '1 hour';

Liveness is whether a stream has written anything recently:

SELECT 'users' AS stream, MAX(_synced_at) AS last_write FROM workos.users
UNION ALL
SELECT 'organizations', MAX(_synced_at) FROM workos.organizations
ORDER BY last_write;

Ingest rate, from the event archive:

SELECT DATE_TRUNC('hour', created_at) AS hour, COUNT(*) AS events
  FROM workos.events
  WHERE created_at > NOW() - INTERVAL '24 hours'
  GROUP BY 1
  ORDER BY 1;

The engine emits structured JSON logs with per-batch counts. skipped is the health metric to graph: a steady trickle is normal, because it is the staleness gate rejecting out-of-order deliveries as designed, while a sudden spike means something changed about delivery ordering.

{
  "level": "info",
  "msg": "processed events",
  "env": "production",
  "events": 42,
  "pages": 1,
  "created": 3,
  "updated": 38,
  "skipped": 1,
  "deleted": 0,
  "cursor": "event_01H..."
}

Three cadences, ordered by the cheapest first:

# Report only. Exits 2 when drift is found, so it works as a scheduled check.
npx workos-sync-postgres reconcile

# Fix what it finds.
npx workos-sync-postgres reconcile --repair

Reconciliation compares ids. It reports localOnly rows (in the mirror but gone upstream, which are missed deletes) and remoteOnly rows (upstream but not in the mirror, which are missed creates). With --repair it tombstones the first set and inserts the second. It does not re-read rows that match, so it is not the tool for refreshing fields; see Refreshing partial fields.

The comparison runs in the database, not in the sync process. The upstream list is streamed into a temporary table one page at a time and the set difference is a pair of anti-joins in SQL, so the memory reconcile needs is bounded by the size of the drift, not the size of the stream. The live ids are captured before the first page is requested, so a row the tail writes while the list is in progress is never a deletion candidate.

One gap remains for streams listed through a parent (memberships, directory users, and directory groups): a parent and its first child that the tail writes during the listing can put the child in the snapshot with no parent to list it under, and --repair tombstones it. Run --repair for those streams in a quiet moment, or run reconcile without it first and read the report.

Streams that cannot be reconciled say so explicitly rather than reporting a misleading zero:

{ "stream": "events", "skippedReason": "append-only stream: nothing to reconcile" }
{ "stream": "organization_domains", "skippedReason": "no list endpoint upstream: drift cannot be detected for this stream" }

Most event payloads are the full API object, so a row written from events is as complete as one written from the REST API. Deprecated fields are the exception: they are left out of event payloads, so on a row only events have touched they are null. Two commands re-list from the REST API and replace each row, which also repairs anything a future event payload leaves out:

• refresh re-lists every top-level stream from the REST API and replaces each row. It skips streams that are listed per parent (memberships, sessions, directory users and groups, groups, and organization roles).
• backfill without --resume re-lists everything, including the per-parent streams, and replaces each row. This is the complete refresh.

npx workos-sync-postgres refresh     # top-level streams
npx workos-sync-postgres backfill    # everything

Hydration re-fetches an object by id when its event arrives and writes the REST object instead of the payload. It is off by default, since it costs a request per event and rarely changes anything. Turn it on per stream with WORKOS_SYNC_HYDRATE, for example organizations=on_create, or always to re-fetch on every event.

The engine paces itself against the documented limits, but they shape how long a backfill takes. Backfill checkpoints after every page, so an interrupted run resumes with backfill --resume.

There is one cursor per environment. Without a quarantine, one event that fails deterministically (a payload a generated column cannot cast, a timestamp the database rejects, a router bug on a shape nobody has seen) would fail every poll, the cursor would never move, and every stream in the environment would stop, sign-in capture included, while the 90-day retention clock ran down.

So the tail isolates such an event and, once it has failed on two consecutive polls, sets it aside in workos._rejected and moves the cursor past the batch. Transient failures never qualify: a hydration 5xx, any API error, a dropped connection, a deadlock, or a full disk all fail the poll and retry as before.

When it happens, you get three signals: an error log line per event with the event id, type, object id, and the database’s message; rejected in status; and a non-zero exit from rejected, so it can run as a scheduled check next to reconcile.

npx workos-sync-postgres rejected

{
  "production": [
    {
      "eventId": "event_01HXXXXXXXXXXXXXXXXXXXXXX",
      "eventType": "user.updated",
      "objectId": "user_01HXXXXXXXXXXXXXXXXXXXXXX",
      "createdAt": "2026-10-22T18:04:09.471Z",
      "rejectedAt": "2026-10-22T18:04:21.118+00:00",
      "error": "error: invalid input syntax for type boolean: \"maybe\"",
      "raw": { "id": "event_01H...", "event": "user.updated", "data": { "...": "..." } }
    }
  ]
}

Then, in order:

1. Read the error. It is the database’s own message, so it names the column or the cast. The whole event is in raw, and the same row is in workos._rejected if SQL is more convenient.
2. Fix the cause. Usually a release: a widened column type, a corrected extraColumns path, a router fix. Occasionally the payload upstream.
3. Replay. Everything, or one event by id. Each event goes back through the normal write path, oldest first. Those that succeed are forgotten; those that fail again stay, with the error updated, and the command exits 2.

npx workos-sync-postgres rejected --replay
npx workos-sync-postgres rejected --replay --id event_01HXXXXXXXXXXXXXXXXXXXXXX

The staleness gate still applies on replay, so a replayed event older than the row it targets is a no-op: later events already carried the newer state, and the row is already correct.

• Backfill was interrupted. Run backfill --resume.

• One event keeps failing. It is quarantined automatically after two polls, and status shows rejected > 0. Fix the cause, then run rejected --replay.

• The sync was down for hours or days. Start it again. The cursor is where you left it and the engine replays from there, as long as the cursor is inside the 90-day window.

• The cursor is past retention. The events are gone. Run backfill to re-establish current state, then reconcile --repair to clear out anything deleted while you were away, then listen. Backfill notices that the stored cursor is older than retention and re-captures the starting position itself.

• Start the tail over from now. Delete the tail row and run backfill, which captures a fresh position before it lists anything:

  Reset the tail cursor

  DELETE FROM workos._sync_state
    WHERE sync_id = 'default' AND env = 'production' AND stream = '__tail__';
  

• Suspected corruption in one stream. reconcile --repair fixes membership drift. To rebuild a stream from scratch, re-list only that stream:

  Terminal

  npx workos-sync-postgres backfill --streams users
  

• A bad deploy wrote wrong data. The event archive is a local, ordered log, so you can inspect exactly what arrived without an API call:

  Events for one object

  SELECT id, event, created_at, data AS payload
    FROM workos.events
    WHERE object_id = 'user_01HXXXXXXXXXXXXXXXXXXXXXX'
    ORDER BY created_at;
  

• init exits 2 after a deploy. A catalog column changed its path or type and the database still has the old projection. The rows are right in _raw; only the typed column is stale. Ingest carries on meanwhile. The log and the printed report carry the exact DDL, and init --repair rebuilds non-key columns from _raw. Key columns, columns a view of yours depends on, and columns carrying an index or constraint of your own are left for you, with the DDL to run.

Embedding the engine in a service you already run is the common case. There is no extra process, and it uses the connection pool you already have.

const engine = new SyncEngine({
  destination: new PostgresDestination({ databaseUrl }),
  environments,
});

engine.listen().catch((err) => {
  logger.error(err);
  process.exit(1);
});

To expose the same health endpoints from your own process, startHealthServer(engine, { port }) serves /healthz and /metrics, or healthReport(engine.health()) and renderMetrics(engine.health()) return the JSON body and the exposition text for a server you already have.

The published image takes the same commands, flags, and environment as the CLI, runs as a non-root user, and honors HTTPS_PROXY and NO_PROXY for calls to the WorkOS API. Run init as a one-shot container on each deploy before starting listen.

docker run -d --restart unless-stopped \
  -e DATABASE_URL -e WORKOS_API_KEY -e WORKOS_ENVIRONMENT_NAME=production \
  -p 8080:8080 \
  ghcr.io/workos/sync-postgres:0.2.0 listen

The image sets WORKOS_SYNC_HEALTH_PORT=8080, exposes that port, and declares a HEALTHCHECK that fetches /healthz every 30 seconds after a 30-second start period, so docker ps and a Docker Compose depends_on condition of service_healthy reflect whether the tail is actually polling. The check only means something for listen; one-shot commands exit before it would ever run.

Point both probes at /healthz. The engine answers 200 while it is still starting, so the liveness probe does not need a long initial delay. What it needs is a failureThreshold and periodSeconds that together exceed WORKOS_SYNC_HEALTH_MAX_POLL_AGE_MS, or a transient API outage restarts a pod that would have recovered on its own.

containers:
  - name: sync
    image: ghcr.io/workos/sync-postgres:0.2.0
    args: ['listen']
    ports:
      - name: health
        containerPort: 8080
    readinessProbe:
      httpGet: { path: /healthz, port: health }
      periodSeconds: 10
    livenessProbe:
      httpGet: { path: /healthz, port: health }
      periodSeconds: 30
      failureThreshold: 4

Add a prometheus.io/scrape: "true" annotation, or a ServiceMonitor, on the health port to collect /metrics.

Run one listener per environment, each with its own WORKOS_API_KEY and WORKOS_ENVIRONMENT_NAME. They can share a database. The library form is the exception: one embedded engine can hold several environments in its environments array.

Only one process tails a given environment and WORKOS_SYNC_ID at a time. The tail runs under a single-writer lease held in the database, so a rolling deploy is safe: the new process starts, finds the lease held, logs a warning once a minute and waits; the old one receives SIGTERM, finishes its current batch, releases the lease, and exits; the new one takes over on its next poll. Nothing is double-applied and the cursor has one writer throughout.

If the old process crashes instead, the lease expires after WORKOS_SYNC_LEASE_TTL_MS (60 seconds by default) and the replacement takes over then, which is the most lag a crash can add. Set the TTL comfortably above the longest poll you expect. A process that stalls past it loses the lease to its replacement and logs lost the events tail lease when it notices. It still applies the batch it had already fetched, which is idempotent, but the database refuses its checkpoint, so the replacement’s cursor is never rewound.

Two syncs that genuinely must tail the same environment take different WORKOS_SYNC_ID values, which gives each its own cursor and its own lease. backfill, reconcile, refresh, status, and redact do not take the lease. They are idempotent and safe to run while a listener is tailing.

If a few minutes of lag is acceptable, tick replaces the long-running process:

*/5 * * * * workos-sync-postgres tick

tick takes the lease for the duration of the poll and releases it on exit. If a listener holds it, tick exits 3 without polling and names the holder on stderr, which usually means both deployment styles are running at once.

The mirror inherits the deletion obligations of any other copy of the data, and archived event payloads embed personal data too, so both have to go:

npx workos-sync-postgres redact --user user_01HXXXXXXXXXXXXXXXXXXXXXX

This hard-deletes, rather than tombstones, the user’s row in users and every row in a table with a user_id column (organization_memberships, sessions, auth_attempts), and removes archived and quarantined events whose object is the user or whose payload carries that user_id. It leaves nothing behind in those tables.

It does not reach rows that reference the person another way. An invitation keeps inviter_user_id, accepted_user_id, and the invitee’s email, and a directory user is a separate identity with its own id. Cover those with your own DELETE statements if your obligation includes them.

The mirror is small, because it holds your identity data rather than an event firehose. The one table that grows without bound is workos.events, which keeps every event. To trim it, keep the window you actually query:

DELETE FROM workos.events WHERE created_at < NOW() - INTERVAL '180 days';

Do not prune auth_attempts on the same schedule unless you mean to. That history exists nowhere else, and once deleted it cannot be recovered from the API.