On November 30th, 2017 from 19:30 to 20:21 UTC, we experienced a brief firewall outage, which resulted in the following:

• BGP Peering went momentarily down
• 4 rounds of DNS+ data were lost

We worked around the issue by switching to the secondary firewall in the primary-secondary configuration.

On December 1, 2017 at 14:02 UTC, and 17:30 UTC we again forced a firewall failover to avoid service interruption.

On December 6, 2017 at 16:15 UTC we again forced a firewall failover to avoid service interruption. As of 22:00 UTC, we identified the root cause and entered an emergency maintenance window to make changes to prevent further interruptions to BGP Peering and DNS+ data.

On December 7, 2017 at 01:00 UTC we ended the emergency maintenance window. The firewall configuration changes should prevent subsequent data loss; however up to a dozen BGP and DNS+ rounds of data was lost.