Routing Issues [12/30/2010]


Tony
 Share

Recommended Posts

Date: 12-30-2010

Start time (CDT): 02:00

Services Affected: Public Network (Globally)

Device: Public Routers

Location: Global

Duration: Emerging Issue

Description: SoftLayer Engineers are aware of a routing issue impacting a subset of customers globally on the SoftLayer backbone. Engineers have traced this issue back to a hardware forwarding limitation on a number of Cisco routers int he backbone. Engineers are currently in contact with Cisco to determine the best course of action to resolve this issue.

We apologize for the inconvenience and appreciate your patience during this issue.

Link to comment
Share on other sites

==================================================

SoftLayer Engineers will be performing emergency maintenance on BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01 in an effort to correct routing issues customers are currently experiencing to various destinations including AT&T residential services. In conjunction with Cisco, Engineers have identified a bug that has resulted in the router entering a degraded performance state. To correct this degraded state, the router must be reloaded.

Customers can expect to see latency / packetloss as routes reconverge across redundant routers on both the public and private networks. While the maintenance window is set for 2 hours, we expect the maintenance to take no longer than 20 minutes during each router reload.

Customers in the Seattle datacenter will lose backend connectivity to all other datacenters (local Seattle private network traffic will not be impacted).

==================================================

Link to comment
Share on other sites

Engineers have completed the reload of BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01. This has resolved the immediate issues with customer connectivity.

During work on these routers, it was determined the same issue is occurring at a much smaller scale at a number of network POPs around the SoftLayer backbone. Engineers will be reloading each of those routers later tonight. Prior to each reload, traffic will be shifted off to redundant paths. Engineers expect no interruption to customer services during these reboots; however, some customers may experience higher latency as their routes shift to other paths.

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
 Share