(Resolved) 2017-10-04 21:56 UTC: Service restored.

Customers may have experienced timeouts while authenticating to the application (app.thousandeyes.com) and the API (api.thousandeyes.com). Customers should not experience loss of data.

The root cause of this outage involves a long-standing known issue with MongoDB, in which failure in a secondary member of a database cluster can induce failure in the primary, and failover to the failing secondary. Unfortunately, the alerts generated for the affected database component were unintentionally filtered, preventing our Operations team from proactively addressing the problem, and also lengthening the time to diagnose the issue. This alerting issue has been addressed and we're working on additional ways to improve cluster reliability. We're sorry for any inconvenience that this issue has caused our customers or partners. We know that problems with 3rd party technologies we use in our product are our problem, and we are working hard and learning from bad things that happen to prevent them from recurring in the future.

2017-10-04 21:36 UTC: Some users are having difficulties with authentication on app.thousandeyes.com and our API. Our Operations team is investigating this issue.