Tony Posted December 30, 2010 Report Share Posted December 30, 2010 Date: 12-30-2010 Start time (CDT): 02:00 Services Affected: Public Network (Globally) Device: Public Routers Location: Global Duration: Emerging Issue Description: SoftLayer Engineers are aware of a routing issue impacting a subset of customers globally on the SoftLayer backbone. Engineers have traced this issue back to a hardware forwarding limitation on a number of Cisco routers int he backbone. Engineers are currently in contact with Cisco to determine the best course of action to resolve this issue. We apologize for the inconvenience and appreciate your patience during this issue. Link to comment Share on other sites More sharing options...
Tony Posted December 30, 2010 Author Report Share Posted December 30, 2010 ================================================== SoftLayer Engineers will be performing emergency maintenance on BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01 in an effort to correct routing issues customers are currently experiencing to various destinations including AT&T residential services. In conjunction with Cisco, Engineers have identified a bug that has resulted in the router entering a degraded performance state. To correct this degraded state, the router must be reloaded. Customers can expect to see latency / packetloss as routes reconverge across redundant routers on both the public and private networks. While the maintenance window is set for 2 hours, we expect the maintenance to take no longer than 20 minutes during each router reload. Customers in the Seattle datacenter will lose backend connectivity to all other datacenters (local Seattle private network traffic will not be impacted). ================================================== Link to comment Share on other sites More sharing options...
Tony Posted December 31, 2010 Author Report Share Posted December 31, 2010 Engineers have completed the reload of BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01. This has resolved the immediate issues with customer connectivity. During work on these routers, it was determined the same issue is occurring at a much smaller scale at a number of network POPs around the SoftLayer backbone. Engineers will be reloading each of those routers later tonight. Prior to each reload, traffic will be shifted off to redundant paths. Engineers expect no interruption to customer services during these reboots; however, some customers may experience higher latency as their routes shift to other paths. Link to comment Share on other sites More sharing options...
Recommended Posts