Service disruption on July 16, 2026
This post outlines the timeline and impact to our services and shares how we will improve resiliency for WorkOS products.
On July 16, 2026, from approximately 08:00 to 10:10 UTC, WorkOS experienced elevated latency, timeouts, and HTTP 5xx responses across the User Management API, AuthKit, and SSO sign-in flows.
At approximately 07:01 UTC, an anomalous increase in concurrent authentication session creation traffic began. The workload continued in concentrated waves, and at approximately 08:00 UTC a larger synchronized wave caused enough requests to enter a defective code path at the same time to exhaust a core database connection pool. Each such request held one database connection open while waiting for a second connection for a lower-priority AuthKit Analytics write. Once the pool was saturated, queued requests, ongoing normal traffic, and automatic client retries kept it saturated even after the triggering traffic subsided, degrading other application traffic that shared the pool.
At 10:07 UTC, we disabled AuthKit Analytics, which contained the defective code path. Customer-facing service recovered by approximately 10:10 UTC, and database connection metrics returned to their normal range by approximately 10:15 UTC.
We're sorry for the disruption we caused for you and your users. Authentication and user-management are critical services, and during this incident we did not meet the reliability standard you expect from WorkOS. We take responsibility for the impact and the causal factors. We've already addressed the proximate root cause, and have begun work on additional factors and deeper architectural work to prevent similar incidents in the future.
Customer impact
During the incident, customers experienced elevated response times and intermittent failures when authenticating users or calling affected User Management endpoints. AuthKit and SSO requests also timed out because they depended on the same service path.
Impact varied by endpoint and over time, but many requests experienced high latency and timeouts. Aggregate error rates measured at our edge over the impact period:
This was an availability issue — our investigation found no evidence of unauthorized access, data exposure, or corruption of customer data.
AuthKit analytics data for 2026-07-16 will contain gaps until a backfill is completed. The backfill will include all sessions created during this period, but sessions refreshed during this window will not be backfilled, resulting in fewer active users reported than expected for some customers.
Directory Sync and Audit Logs were unaffected.
What happened
A normal session-creation request performs two relevant database writes:
- It records the product event associated with the new session.
- It records an additional event used for AuthKit Analytics.
These writes were intended to share one database connection. The analytics write was wrapped in what the application treated as a nested transaction with the intent to reuse the existing connection.
Due to a bug in our database ORM library, the nested transaction requested a second database connection while the first transaction remained open.
A sudden increase in concurrent session creation pushed the system past a critical threshold:
- Each request acquired one database connection and completed its primary event write.
- Before releasing that connection, the request waited to acquire a second connection for the analytics write.
- At high concurrency, the available connections became occupied by requests that were all waiting for another connection from the same pool.
- Useful throughput fell sharply, causing requests to queue and eventually time out.
- The affected pool served shared application traffic, so the degradation extended beyond the workload that initially exposed the defect.
The traffic increase was the trigger, not the root cause. The system should have absorbed or safely limited that traffic. The root cause was a latent bug in our transaction library, combined with insufficient workload isolation, which allowed lower priority analytics work to retain a database connection while waiting for another, starving the connection pool for higher priority work.
Why the incident continued after the initial traffic increase
The service entered a self-sustaining connection-starvation state.
Once most connections were held by requests waiting for a second connection, very little work could complete. Requests already accepted by the service remained queued, normal customer traffic continued, and some failed or timed-out requests were retried, leading to a request storm. Whenever a connection became available, queued work acquired it and could enter the same two-connection pattern.
As a result, the individual requests holding connections changed continuously, but the pool remained saturated. The service could not drain at its normal rate even after the initial burst subsided.
In addition, our connection-borrow timeout was too long, which prolonged connection exhaustion.
Resolution
At 10:07 UTC, we disabled the AuthKit Analytics write through a runtime control. New session-creation requests then required only one connection for the affected database path.
This broke the hold-and-wait cycle:
- Requests completed their primary event write.
- Transactions committed and released their connections.
- The queued backlog began to drain.
- Error rates and connection-acquisition latency returned to normal within minutes.
The last correlated transaction error was observed at approximately 10:10 UTC, and database connection metrics had returned to their normal range by approximately 10:15 UTC.
Timeline (UTC)
July 16, 07:01 An anomalous pattern of concurrent session-creation traffic begins and continues in concentrated waves over the next hour. Customer-facing service remains healthy.
08:00 A larger synchronized wave pushes the affected database connection path into starvation. Customer-visible latency, timeouts, and errors begin.
08:06–08:26 Automated alerts fire and an incident is declared with on-call responders engaged.
08:26–09:48 Responders rule out common incident drivers: deployments, feature flags, volumetric attacks, etc. Responders identify connection starvation as the proximate cause with the database itself healthy. Service capacity is increased without improvement, consistent with the failure mode. Isolation of the traffic that started the incident was difficult due to the number of signals the team was processing and the narrow nature of the use case that combined with a latent bug.
09:50 Root cause isolated: the analytics write is acquiring a second database connection while the first is held, contrary to the intended design.
10:07 WorkOS disables the AuthKit Analytics write, removing the second connection acquisition from new session requests.
10:10 Correlated transaction errors cease and customer-facing service levels recover.
10:15 Database connection and acquisition-latency metrics return to their normal ranges.
Corrective actions
We are addressing the defect itself, the conditions that allowed it to affect shared traffic, and our ability to contain similar failures sooner.
Completed
- Immediate containment — Disabled the AuthKit Analytics write that acquired the second database connection.
- Permanent transaction fix — Changed the analytics write to use the connection already held by the primary event transaction. Error isolation now uses a real savepoint on that same connection.
- Regression testing — Added tests verifying both writes use the same connection and that an analytics failure doesn't invalidate the primary event transaction.
- Analytics restoration — Re-enabled AuthKit Analytics after the permanent fix deployed.
- Operational controls — Added independent runtime controls for session-creation and session-refresh analytics, so optional writes can be disabled without affecting authentication.
- Observability — Improved observability and telemetry for relevant database metrics.
In progress
- Connection-borrow timeout (Jul 17) — Shorten the timeout so blocked work fails quickly and releases capacity.
- Architectural isolation (Jul 17) — Move non-critical analytics work out of the synchronous auth path so analytics latency or failure can't affect an auth response.
- Request cleanup (Jul 24) — Strengthen cancellation handling so work from a timed-out request can't keep holding or using transaction resources.
- Broader transaction audit (Jul 24) — Review other critical paths for nested-transaction or connection-acquisition patterns that could hold one connection while waiting for another.
Planned
- Overload protection (Jul 31) — Add bounded concurrency and workload-aware safeguards so anomalous traffic is limited before it consumes shared capacity.
- Load and failure testing (Jul 31) — Expand testing above normal connection-pool concurrency: slow analytics writes, database errors, request cancellation, and automated rollback thresholds.
How these changes reduce future risk
We have already removed the specific code defect that caused this incident. The additional work itemized above is intended to improve defense in depth:
- Degrade safely: Keep AuthKit Analytics outside the critical authentication path.
- Limit blast radius: Bound concurrency before one workload can consume shared resources.
- Prevent recurrence: Load test above pool capacity to discover possible similar bugs. Conduct a broader audit for similar database connection use patterns.
- Improve response time: Improve observability and interpretation of relevant database metrics to identify similar situations in the future.
The initiating traffic should have resulted in bounded queueing or targeted rate limiting, not broad service degradation. We are treating the transaction fix as necessary but not sufficient. The combination of corrected connection handling, stronger overload protection, and architectural separation is designed to reduce both the likelihood and potential impact of a similar event.
Closing
An increase in traffic on a narrow slice of our API exposed a specific failure mode in a shared critical path. We have fixed the defect and are following through with additional safeguards so that unusual traffic is contained, low-priority work cannot impair critical authentication flows, and responders receive clearer signals earlier in an incident.