Today, we experienced problems with alert notification delivery between midnight and 10am pacific time (4am to 2pm UTC).  During this time, alert notifications were generated and held in a queue prior to being dispatched.

The issue affected alert notifications using email, Webhook and PagerDuty, for all alerts generated during the affected window.

Affected systems

Alert notifications, emailed reports

Affected period

9/9/2015 04:00 UTC through 9/9/2015 14:00 UTC

Resolution

Upon detection and resolution at 10:01am pacific time, these services were restored, and queued notifications were delivered.

Our team is still looking into the root cause and will post an update once one has been determined.