Cody R. Posted April 19, 2011 Report Posted April 19, 2011 We currently lost all connectivity at our Seattle data center. Network engineers have acknowledged there is an ongoing issue on their end and they're actively investigating. We'll post more information as we receive it. Engineers are aware of the sporadic packetloss and/or connectivity to FCR01.SEA01. Currently engineers are working on resolving this issue; however, there is a change a reboot to the router will be required. In the event this needs to happen, a notice will be posted here along with any other information gathered.
Cody R. Posted April 19, 2011 Author Report Posted April 19, 2011 Everything is currently online. We're waiting for more information from the network engineers on what the issue was.
Tony Posted April 19, 2011 Report Posted April 19, 2011 Update for when the service was restored: Service has been restored to customers behind FCR01.SEA01. During the process of working on the router, the issue manifested itself into 100% CPU resulting in upstream and downstream links going down along with routing protocols. Engineers were able to stabilize the router without a reload, and are currently monitoring it to determine if the fix is permanent.
Tony Posted April 19, 2011 Report Posted April 19, 2011 Latest update: Engineers have determined this is not a permanent fix for the FCR01.SEA01 issue. During the course of troubleshooting this issue with Cisco, it has been determined that the best course of action is to upgrade the router to the latest IOS version. This will be happening at approximately 01:30 CDT.
Tony Posted April 19, 2011 Report Posted April 19, 2011 Everything in Seattle is once again down to reload the router to hopefully have a permanent fix to the issue.
Tony Posted April 19, 2011 Report Posted April 19, 2011 Everything is back up now just waiting for networking to give their report on the issue.
Tony Posted April 19, 2011 Report Posted April 19, 2011 Another update: Engineers are continuing to work with Cisco TAC on this issue. At this point, the router has been restored to service at approximately 03:20 CDT. Some customers may continue to experience intermittent packetloss behind FCR01.SEA01.
Recommended Posts