Tony Posted May 24, 2009 Report Posted May 24, 2009 Date: Sunday, May 24th, 2009 (05/24/2009) Start time: 07:45 PDT End time: 08:15 PDT Location: SEA01 Services Affected: Seattle Public Network Datacenter Engineers were made aware of loss of public connectivity in the Seattle datacenter at approximately 7:45 AM PDT. While the root cause of this issue is still being determined, engineers were able to restore service to the public network at approximately 8:15 AM PDT. A RFO will be forthcoming once the cause of the outage has been determined.
Tony Posted May 24, 2009 Author Report Posted May 24, 2009 Date: 05-24-2009 Start time (CDT): 9:41am End time (CDT): 10:15am Services Affected: Public network connectivity to portions of SEA01.SR01 and SR02 Device: FCR01.SEA01 Location: SEA01 Duration: 30 minutes Description: At approximately 9:41am CDT, a datacenter technician in the sea01 facility failed to properly follow procedures when dealing with a standard administrative/maintenance issue related to the fcr01.sea01 network device. Immediately upon realizing that the mistake had caused a shutdown of the device, network teams reacted to the incident and worked to restore service. Future Mitigation: Employment of the technician in question has been terminated. In addition, procedures will be reviewed and all related staff will go through additional training on a regular basis. Additional safeguards are being reviewed for future implementation.
Recommended Posts