(Resolved) Agent Collector outage: 2014-09-10

Last updated: Mon Feb 04 23:45:51 GMT 2019

Our agent collector infrastructure is undergoing emergency maintenance, due to an issue with the storage subsystem.  At this time, users will be unable to run instant tests.  Test results are continuing to be collected, and are being held by the respective agents.  These results will be synchronized with our system as soon as the agent collector system is back online.

Status Updates

TimeUpdate
2014-09-10 18:19All agents have at this time checked in with their assigned collector, and results
have been uploaded.  At this time we are once again 100% operational.  Thank you for your patience.
2014-09-10 17:54 PTThe Agent Collector system is once again operational.  

As the global agent infrastructure checks in and commits data, we may see sporadic
offline notifications due to exceptionally high traffic.  

This will self-resolve once we are back in sync (est 30-90 mins)
2014-09-10 17:45 PTEmergency maintenance is complete; services are starting
2014-09-10 17:15 PTTwitter notification posted, this KB article created
2014-09-10 16:50 PTOur operations team has confirmed that our agent collector infrastructure is presently down for emergency maintenance
2014-09-10 16:45 PT Reports of agent unavailability, lagging responses and inability to run instant tests have been reported

 Root Cause 

This problem was due to the shared filesystem used by our Agent Collector infrastructure becoming temporarily unstable.  Upon identification of the issue, our operations team engaged and resolved.  We do not expect further occurrences of this issue at this time.