# Monitoring and Operations ## Introduction This page covers running the [sync engine](https://workos.com/docs/events/data-syncing/database) in production: what to watch, what to alert on, how to deploy it, and how to recover when something goes wrong. It assumes you have completed the [quick start](https://workos.com/docs/events/data-syncing/database/postgres-quick-start). *** ## The most important failure mode A stalled sync is silent. Nothing errors, no request fails, the tables simply stop changing, and every query keeps returning plausible-looking results. It becomes unrecoverable at a deadline, because the events API retains 90 days of history and a cursor older than that cannot be replayed. So the single most important alert is **poll age**, not error rate. ```bash title="Terminal" npx workos-sync-postgres status ``` ```json title="Status output" { "production": { "cursor": "event_01HXXXXXXXXXXXXXXXXXXXXXX", "rangeStart": null, "started": true, "cursorAgeDays": 0.0004, "lastPolledAt": "2026-10-22T18:04:11.902Z", "pollAgeSeconds": 3.1, "lease": { "holder": "sync-7f9c4:1:a3f1b2", "expiresAt": "2026-10-22T18:05:11.902Z" }, "rejected": 0, "streams": [ { "stream": "users", "rows": 14203, "deleted": 87, "lastSyncedAt": "2026-10-22T18:04:09.471Z", "stalenessSeconds": 41.2, "lagSeconds": 2.1, "lagSampleSize": 12 } ] } } ``` Three of these numbers deserve an alert, and two of them look similar but mean different things. - **`pollAgeSeconds`** is how long since a poll last completed without error, whether or not it returned anything. If it climbs, the process is down, stuck, or failing every request. **Alert on this.** It is `null` until the first poll. - **`cursorAgeDays`** is the age of the newest event applied. It measures distance from the 90-day cliff, not liveness: an environment where nothing happens for a week will show seven days here while being perfectly healthy. The engine warns at 75 days and logs an error past 90. Alert well before that. - **`rejected`** is how many events the tail has set aside because they fail deterministically. The tail kept going, so poll age looks healthy, but the objects those events describe are missing or stale until you replay them. **Alert on this too.** See [Rejected events](#rejected-events). `lease` names the process currently tailing this environment and when its claim lapses if it stops renewing. It is `null` when nothing is listening, which alongside a climbing `pollAgeSeconds` tells you the poller is gone rather than stuck. ### The health endpoint `status` opens a database connection and queries every stream, which is the wrong shape for something that runs every few seconds. `listen` serves the same poll age from memory when `WORKOS_SYNC_HEALTH_PORT` is set. The container image sets it to `8080`. ```bash title="Terminal" curl -s localhost:8080/healthz ``` ```json title="Health output" { "state": "listening", "fatal": null, "startedAt": "2026-10-22T17:58:02.114Z", "healthy": true, "environments": { "production": { "lastPolledAt": "2026-10-22T18:04:11.902Z", "pollAgeSeconds": 3.1, "lastEventAt": "2026-10-22T18:04:09.471Z", "cursor": "event_01HXXXXXXXXXXXXXXXXXXXXXX", "lastError": null, "counters": { "ticks": 71, "events": 42, "created": 3, "updated": 38, "skipped": 1, "deleted": 0, "errors": 0 } } } } ``` The status code is the verdict. It returns `200` while the process is starting and while every environment has polled successfully within `WORKOS_SYNC_HEALTH_MAX_POLL_AGE_MS`, which defaults to 60 seconds. It returns `503` once any environment's poll is older than that, once the engine has stopped on a fatal error, or once `listen` has returned. No request to it touches the database, so it keeps answering when the database is the thing that is down. ### Prometheus metrics With the health port set, `listen` also serves `GET /metrics` in the Prometheus text format, from the same in-memory state. Every per-environment series carries an `env` label. | Metric | Type | What it is | | ------------------------------ | ------- | ---------------------------------------------------------------------------------------------------------- | | `workos_sync_up` | gauge | `1` while the engine is running; `0` once it has stopped or hit a fatal error. | | `workos_sync_poll_age_seconds` | gauge | Seconds since the last successful poll. **Alert on this.** Absent until the first successful poll. | | `workos_sync_cursor_age_days` | gauge | Age of the newest event applied: distance from the 90-day cliff, not liveness. Absent until the first event. | | `workos_sync_ticks_total` | counter | Polls attempted, successful or not. | | `workos_sync_events_total` | counter | Events applied. | | `workos_sync_writes_total` | counter | Rows written, with a `kind` label of `created`, `updated`, `skipped`, or `deleted`. | | `workos_sync_errors_total` | counter | Polls that failed with a retryable error. Fatal errors stop the process instead. | ```yaml title="prometheus.rules.yml" - alert: WorkOSSyncStalled expr: workos_sync_poll_age_seconds > 300 or workos_sync_up == 0 for: 2m # A listener that has never polled successfully has a ticks series but no # poll-age series, so nothing above matches it. The `for` is the startup grace. - alert: WorkOSSyncNeverPolled expr: workos_sync_ticks_total unless workos_sync_poll_age_seconds for: 5m - alert: WorkOSSyncCursorNearRetention expr: workos_sync_cursor_age_days > 60 ``` *** ## What to monitor Every table carries `_updated_at` (source time) and `_synced_at` (write time), which makes the important signals plain SQL. **Lag** is how far behind the mirror is. Measure it only over rows that changed recently. Across every row, `_synced_at - _updated_at` reports how old the objects are, not how late the sync is: a mirror that has just backfilled year-old users would report a lag of a year while being perfectly current. ```sql title="Mean lag over the last hour" SELECT AVG(_synced_at - _updated_at) AS mean_lag, COUNT(*) AS sample FROM workos.users WHERE _updated_at > NOW() - INTERVAL '1 hour'; ``` **Liveness** is whether a stream has written anything recently: ```sql title="Last write per stream" SELECT 'users' AS stream, MAX(_synced_at) AS last_write FROM workos.users UNION ALL SELECT 'organizations', MAX(_synced_at) FROM workos.organizations ORDER BY last_write; ``` **Ingest rate**, from the event archive: ```sql title="Events per hour" SELECT DATE_TRUNC('hour', created_at) AS hour, COUNT(*) AS events FROM workos.events WHERE created_at > NOW() - INTERVAL '24 hours' GROUP BY 1 ORDER BY 1; ``` The engine emits structured JSON logs with per-batch counts. `skipped` is the health metric to graph: a steady trickle is normal, because it is the staleness gate rejecting out-of-order deliveries as designed, while a sudden spike means something changed about delivery ordering. ```json title="A processed batch" { "level": "info", "msg": "processed events", "env": "production", "events": 42, "pages": 1, "created": 3, "updated": 38, "skipped": 1, "deleted": 0, "cursor": "event_01H..." } ``` ### Alerts worth having | Alert | Condition | Why | | ---------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | Sync stalled | `pollAgeSeconds > 300`, or `null` long after deploy; `/healthz` returns 503 | The poller has stopped. | | Cursor near retention | `cursorAgeDays > 60` | Replay is still possible, but the window is closing. | | Stream not writing | `max(_synced_at) < now() - interval '1 hour'` on a busy stream | One stream failing while others continue. | | Skipped writes spiking | `skipped` in the logs | Out-of-order delivery or a backfill colliding with the tail. Usually benign, occasionally not. | | Drift detected | `reconcile` exits 2 | Missed creates or deletes. | | Column drift | `init` exits 2 after a deploy | A generated column no longer matches the catalog; the projection is stale until repaired. | | Events quarantined | `rejected > 0` in `status`, or `rejected` exits 2 | One payload the mirror cannot apply. The tail kept going; the object waits on a replay. | | Process exited | `listen` returned or crashed | A rejected credential or a backwards-paging cursor stops the engine on purpose. | | Lease lost | `lost the events tail lease` in the logs | The process stalled past the lease TTL and another one took over. Check for a hung tick. | *** ## Reconciliation schedule Three cadences, ordered by the cheapest first: | Cadence | Command | Finds | | ------------------------------ | -------------------- | -------------------------- | | Continuous | `status` | Stalls and lag | | Daily | `reconcile` | Missed creates and deletes | | Weekly, and after any incident | `reconcile --repair` | The same, and fixes them | ```bash title="Terminal" # Report only. Exits 2 when drift is found, so it works as a scheduled check. npx workos-sync-postgres reconcile # Fix what it finds. npx workos-sync-postgres reconcile --repair ``` Reconciliation compares ids. It reports `localOnly` rows (in the mirror but gone upstream, which are missed deletes) and `remoteOnly` rows (upstream but not in the mirror, which are missed creates). With `--repair` it tombstones the first set and inserts the second. It does not re-read rows that match, so it is not the tool for refreshing fields; see [Refreshing partial fields](#refreshing-partial-fields). The comparison runs in the database, not in the sync process. The upstream list is streamed into a temporary table one page at a time and the set difference is a pair of anti-joins in SQL, so the memory `reconcile` needs is bounded by the size of the drift, not the size of the stream. The live ids are captured before the first page is requested, so a row the tail writes while the list is in progress is never a deletion candidate. One gap remains for streams listed through a parent (memberships, directory users, and directory groups): a parent and its first child that the tail writes during the listing can put the child in the snapshot with no parent to list it under, and `--repair` tombstones it. Run `--repair` for those streams in a quiet moment, or run `reconcile` without it first and read the report. Streams that cannot be reconciled say so explicitly rather than reporting a misleading zero: ```json title="Streams reconcile skips" { "stream": "events", "skippedReason": "append-only stream: nothing to reconcile" } { "stream": "organization_domains", "skippedReason": "no list endpoint upstream: drift cannot be detected for this stream" } ``` *** ## Refreshing partial fields Most event payloads are the full API object, so a row written from events is as complete as one written from the REST API. Deprecated fields are the exception: they are left out of event payloads, so on a row only events have touched they are `null`. Two commands re-list from the REST API and replace each row, which also repairs anything a future event payload leaves out: - **`refresh`** re-lists every top-level stream from the REST API and replaces each row. It skips streams that are listed per parent (memberships, sessions, directory users and groups, groups, and organization roles). - **`backfill`** without `--resume` re-lists everything, including the per-parent streams, and replaces each row. This is the complete refresh. ```bash title="Terminal" npx workos-sync-postgres refresh # top-level streams npx workos-sync-postgres backfill # everything ``` Hydration re-fetches an object by id when its event arrives and writes the REST object instead of the payload. It is off by default, since it costs a request per event and rarely changes anything. Turn it on per stream with `WORKOS_SYNC_HYDRATE`, for example `organizations=on_create`, or `always` to re-fetch on every event. *** ## Rate limits The engine paces itself against [the documented limits](https://workos.com/docs/reference/rate-limits), but they shape how long a backfill takes. Backfill checkpoints after every page, so an interrupted run resumes with `backfill --resume`. *** ## Rejected events There is one cursor per environment. Without a quarantine, one event that fails deterministically (a payload a generated column cannot cast, a timestamp the database rejects, a router bug on a shape nobody has seen) would fail every poll, the cursor would never move, and every stream in the environment would stop, sign-in capture included, while the 90-day retention clock ran down. So the tail isolates such an event and, once it has failed on **two consecutive polls**, sets it aside in `workos._rejected` and moves the cursor past the batch. Transient failures never qualify: a hydration 5xx, any API error, a dropped connection, a deadlock, or a full disk all fail the poll and retry as before. When it happens, you get three signals: an error log line per event with the event id, type, object id, and the database's message; `rejected` in `status`; and a non-zero exit from `rejected`, so it can run as a scheduled check next to `reconcile`. ```bash title="Terminal" npx workos-sync-postgres rejected ``` ```json title="Rejected output" { "production": [ { "eventId": "event_01HXXXXXXXXXXXXXXXXXXXXXX", "eventType": "user.updated", "objectId": "user_01HXXXXXXXXXXXXXXXXXXXXXX", "createdAt": "2026-10-22T18:04:09.471Z", "rejectedAt": "2026-10-22T18:04:21.118+00:00", "error": "error: invalid input syntax for type boolean: \"maybe\"", "raw": { "id": "event_01H...", "event": "user.updated", "data": { "...": "..." } } } ] } ``` Then, in order: 1. **Read the error.** It is the database's own message, so it names the column or the cast. The whole event is in `raw`, and the same row is in `workos._rejected` if SQL is more convenient. 2. **Fix the cause.** Usually a release: a widened column type, a corrected `extraColumns` path, a router fix. Occasionally the payload upstream. 3. **Replay.** Everything, or one event by id. Each event goes back through the normal write path, oldest first. Those that succeed are forgotten; those that fail again stay, with the error updated, and the command exits `2`. ```bash title="Terminal" npx workos-sync-postgres rejected --replay npx workos-sync-postgres rejected --replay --id event_01HXXXXXXXXXXXXXXXXXXXXXX ``` The staleness gate still applies on replay, so a replayed event older than the row it targets is a no-op: later events already carried the newer state, and the row is already correct. *** ## Recovery - **Backfill was interrupted.** Run `backfill --resume`. - **One event keeps failing.** It is quarantined automatically after two polls, and `status` shows `rejected > 0`. Fix the cause, then run `rejected --replay`. - **The sync was down for hours or days.** Start it again. The cursor is where you left it and the engine replays from there, as long as the cursor is inside the 90-day window. - **The cursor is past retention.** The events are gone. Run `backfill` to re-establish current state, then `reconcile --repair` to clear out anything deleted while you were away, then `listen`. Backfill notices that the stored cursor is older than retention and re-captures the starting position itself. - **Start the tail over from now.** Delete the tail row and run `backfill`, which captures a fresh position before it lists anything: ```sql title="Reset the tail cursor" DELETE FROM workos._sync_state WHERE sync_id = 'default' AND env = 'production' AND stream = '__tail__'; ``` - **Suspected corruption in one stream.** `reconcile --repair` fixes membership drift. To rebuild a stream from scratch, re-list only that stream: ```bash title="Terminal" npx workos-sync-postgres backfill --streams users ``` - **A bad deploy wrote wrong data.** The event archive is a local, ordered log, so you can inspect exactly what arrived without an API call: ```sql title="Events for one object" SELECT id, event, created_at, data AS payload FROM workos.events WHERE object_id = 'user_01HXXXXXXXXXXXXXXXXXXXXXX' ORDER BY created_at; ``` - **`init` exits 2 after a deploy.** A catalog column changed its path or type and the database still has the old projection. The rows are right in `_raw`; only the typed column is stale. Ingest carries on meanwhile. The log and the printed report carry the exact DDL, and `init --repair` rebuilds non-key columns from `_raw`. Key columns, columns a view of yours depends on, and columns carrying an index or constraint of your own are left for you, with the DDL to run. *** ## Deployment ### As a library Embedding the engine in a service you already run is the common case. There is no extra process, and it uses the connection pool you already have. ```ts title="Run the tail inside an existing service" const engine = new SyncEngine({ destination: new PostgresDestination({ databaseUrl }), environments, }); engine.listen().catch((err) => { logger.error(err); process.exit(1); }); ``` To expose the same health endpoints from your own process, `startHealthServer(engine, { port })` serves `/healthz` and `/metrics`, or `healthReport(engine.health())` and `renderMetrics(engine.health())` return the JSON body and the exposition text for a server you already have. ### As a container The published image takes the same commands, flags, and environment as the CLI, runs as a non-root user, and honors `HTTPS_PROXY` and `NO_PROXY` for calls to the WorkOS API. Run `init` as a one-shot container on each deploy before starting `listen`. ```bash title="Terminal" docker run -d --restart unless-stopped \ -e DATABASE_URL -e WORKOS_API_KEY -e WORKOS_ENVIRONMENT_NAME=production \ -p 8080:8080 \ ghcr.io/workos/sync-postgres:0.2.0 listen ``` The image sets `WORKOS_SYNC_HEALTH_PORT=8080`, exposes that port, and declares a `HEALTHCHECK` that fetches `/healthz` every 30 seconds after a 30-second start period, so `docker ps` and a Docker Compose `depends_on` condition of `service_healthy` reflect whether the tail is actually polling. The check only means something for `listen`; one-shot commands exit before it would ever run. ### On Kubernetes Point both probes at `/healthz`. The engine answers `200` while it is still starting, so the liveness probe does not need a long initial delay. What it needs is a `failureThreshold` and `periodSeconds` that together exceed `WORKOS_SYNC_HEALTH_MAX_POLL_AGE_MS`, or a transient API outage restarts a pod that would have recovered on its own. ```yaml title="Pod spec excerpt" containers: - name: sync image: ghcr.io/workos/sync-postgres:0.2.0 args: ['listen'] ports: - name: health containerPort: 8080 readinessProbe: httpGet: { path: /healthz, port: health } periodSeconds: 10 livenessProbe: httpGet: { path: /healthz, port: health } periodSeconds: 30 failureThreshold: 4 ``` Add a `prometheus.io/scrape: "true"` annotation, or a `ServiceMonitor`, on the `health` port to collect `/metrics`. ### One listener per environment Run one listener per environment, each with its own `WORKOS_API_KEY` and `WORKOS_ENVIRONMENT_NAME`. They can share a database. The library form is the exception: one embedded engine can hold several environments in its `environments` array. Only one process tails a given environment and `WORKOS_SYNC_ID` at a time. The tail runs under a single-writer lease held in the database, so a rolling deploy is safe: the new process starts, finds the lease held, logs a warning once a minute and waits; the old one receives `SIGTERM`, finishes its current batch, releases the lease, and exits; the new one takes over on its next poll. Nothing is double-applied and the cursor has one writer throughout. If the old process crashes instead, the lease expires after `WORKOS_SYNC_LEASE_TTL_MS` (60 seconds by default) and the replacement takes over then, which is the most lag a crash can add. Set the TTL comfortably above the longest poll you expect. A process that stalls past it loses the lease to its replacement and logs `lost the events tail lease` when it notices. It still applies the batch it had already fetched, which is idempotent, but the database refuses its checkpoint, so the replacement's cursor is never rewound. Two syncs that genuinely must tail the same environment take different `WORKOS_SYNC_ID` values, which gives each its own cursor and its own lease. `backfill`, `reconcile`, `refresh`, `status`, and `redact` do not take the lease. They are idempotent and safe to run while a listener is tailing. ### As a scheduled job If a few minutes of lag is acceptable, `tick` replaces the long-running process: ```bash title="crontab" */5 * * * * workos-sync-postgres tick ``` `tick` takes the lease for the duration of the poll and releases it on exit. If a listener holds it, `tick` exits `3` without polling and names the holder on stderr, which usually means both deployment styles are running at once. *** ## Deletion requests The mirror inherits the deletion obligations of any other copy of the data, and archived event payloads embed personal data too, so both have to go: ```bash title="Terminal" npx workos-sync-postgres redact --user user_01HXXXXXXXXXXXXXXXXXXXXXX ``` This hard-deletes, rather than tombstones, the user's row in `users` and every row in a table with a `user_id` column (`organization_memberships`, `sessions`, `auth_attempts`), and removes archived and quarantined events whose object is the user or whose payload carries that `user_id`. It leaves nothing behind in those tables. It does not reach rows that reference the person another way. An invitation keeps `inviter_user_id`, `accepted_user_id`, and the invitee's email, and a directory user is a separate identity with its own id. Cover those with your own `DELETE` statements if your obligation includes them. *** ## Database sizing The mirror is small, because it holds your identity data rather than an event firehose. The one table that grows without bound is `workos.events`, which keeps every event. To trim it, keep the window you actually query: ```sql title="Trim the event archive" DELETE FROM workos.events WHERE created_at < NOW() - INTERVAL '180 days'; ``` Do not prune `auth_attempts` on the same schedule unless you mean to. That history exists nowhere else, and once deleted it cannot be recovered from the API.