Seattle Network Routing Anomalies [05/24/2009]


Recommended Posts

Date: Sunday, May 24th, 2009 (05/24/2009)

Start time: 07:45 PDT

End time: 08:15 PDT

Location: SEA01

Services Affected: Seattle Public Network

Datacenter Engineers were made aware of loss of public connectivity in the Seattle datacenter at approximately 7:45 AM PDT. While the root cause of this issue is still being determined, engineers were able to restore service to the public network at approximately 8:15 AM PDT.

A RFO will be forthcoming once the cause of the outage has been determined.

Link to comment
Share on other sites

Date: 05-24-2009

Start time (CDT): 9:41am

End time (CDT): 10:15am

Services Affected: Public network connectivity to portions of SEA01.SR01 and SR02

Device: FCR01.SEA01

Location: SEA01

Duration: 30 minutes


At approximately 9:41am CDT, a datacenter technician in the sea01 facility failed to properly follow procedures when dealing with a standard administrative/maintenance issue related to the fcr01.sea01 network device. Immediately upon realizing that the mistake had caused a shutdown of the device, network teams reacted to the incident and worked to restore service.

Future Mitigation:

Employment of the technician in question has been terminated. In addition, procedures will be reviewed and all related staff will go through additional training on a regular basis. Additional safeguards are being reviewed for future implementation.

Link to comment
Share on other sites

This topic is now closed to further replies.