Intermittent Login Failures | System Status

Summary

On April 9, 2026, some Atmos Pro users were temporarily unable to log in. The issue lasted approximately 2 hours and was caused by two separate but related problems in the login flow. No customer data was lost or compromised, and users who were already logged in were largely unaffected.

Impact

Some users experienced failed login attempts or were shown a generic error page
Users signing in with GitHub were most affected by the second issue
The issue was intermittent — not all login attempts were affected
No data loss, no unauthorized access, and no impact to running workflows

Timeline

We began receiving reports of intermittent login failures

Our team began investigating the issue

Root cause identified: an internal analytics call in the login flow was timing out and blocking authentication

First fix deployed — hardened auth callbacks so analytics and database failures cannot block login

Second issue identified affecting GitHub sign-ins specifically: a tightened validation check in our authentication library began rejecting legitimate GitHub responses because our provider configuration relied on a default value

GitHub OAuth issuer fix deployed

All logins confirmed functioning normally

What Happened

Two separate issues surfaced during the same incident window, both exposed by a recent dependency upgrade to internal libraries used in our authentication pipeline:

Brittle auth callback handling. An internal monitoring call in the login flow was timing out. Rather than handling the delay gracefully, the authentication layer treated it as a hard failure and rejected the login attempt. Related session-handling code threw on transient errors, which logged users out instead of letting them retry. Error details were also not being logged correctly, which slowed diagnosis. All of these were pre-existing code paths that became observable after the dependency upgrade changed how parsing and validation errors propagate.
Over-strict GitHub callback validation. While investigating the first issue, we discovered that a defense-in-depth validation check in our authentication library had become stricter. The check verifies that the authentication provider responding to a sign-in is the one the user started with — a safeguard against cross-provider confusion. Our GitHub provider configuration was relying on the library's default value for that check, which no longer matched the value GitHub now returns, so the library conservatively rejected every GitHub callback. No insecure path was opened; valid sign-ins were simply being over-rejected. The improved error reporting from our first fix helped us identify and resolve this quickly.

What We Did

Made the analytics call non-blocking so that an analytics timeout can never block the login flow again
Fixed the login flow to handle temporary service delays gracefully instead of failing outright
Improved session handling so that users who are already logged in are not disrupted by transient issues
Improved error reporting so our team can diagnose similar issues faster — this directly helped us catch the second issue
Aligned our GitHub authentication provider configuration with the current expectations of our authentication library, so legitimate GitHub sign-ins are no longer over-rejected by the defense-in-depth validation check
Added monitoring and alerting for the specific conditions that triggered this failure, so our team is notified before customers are affected

Timeline

We began receiving reports of intermittent login failures

Our team began investigating the issue

Root cause identified: an internal analytics call in the login flow was timing out and blocking authentication

First fix deployed — hardened auth callbacks so analytics and database failures cannot block login

GitHub OAuth issuer fix deployed

All logins confirmed functioning normally

What Happened

Two separate issues surfaced during the same incident window, both exposed by a recent dependency upgrade to internal libraries used in our authentication pipeline:

Brittle auth callback handling. An internal monitoring call in the login flow was timing out. Rather than handling the delay gracefully, the authentication layer treated it as a hard failure and rejected the login attempt. Related session-handling code threw on transient errors, which logged users out instead of letting them retry. Error details were also not being logged correctly, which slowed diagnosis. All of these were pre-existing code paths that became observable after the dependency upgrade changed how parsing and validation errors propagate.

Over-strict GitHub callback validation. While investigating the first issue, we discovered that a defense-in-depth validation check in our authentication library had become stricter. The check verifies that the authentication provider responding to a sign-in is the one the user started with — a safeguard against cross-provider confusion. Our GitHub provider configuration was relying on the library's default value for that check, which no longer matched the value GitHub now returns, so the library conservatively rejected every GitHub callback. No insecure path was opened; valid sign-ins were simply being over-rejected. The improved error reporting from our first fix helped us identify and resolve this quickly.

What We Did

Made the analytics call non-blocking so that an analytics timeout can never block the login flow again

Fixed the login flow to handle temporary service delays gracefully instead of failing outright

Improved session handling so that users who are already logged in are not disrupted by transient issues

Improved error reporting so our team can diagnose similar issues faster — this directly helped us catch the second issue

Aligned our GitHub authentication provider configuration with the current expectations of our authentication library, so legitimate GitHub sign-ins are no longer over-rejected by the defense-in-depth validation check

Added monitoring and alerting for the specific conditions that triggered this failure, so our team is notified before customers are affected