Tony

The rebuild has finished and everything is optimal once again.

The drive has been replaced and now is rebuilding. 3% complete thus far

The Tornado server just had a drive fail around 7pm PST January 9th 2011. We're going to swap the drive out as soon as possible and then run the rebuild. Since this machine is hot swap capable there should be no downtime during the swap.

The rebuild has been completed.

The drive has been replaced and is 40% complete rebuilding thus far.

We've just been notified one of the drives on the Yoda server has failed on January 9th 2011. We're going to swap out the failed drive as soon as possible and start a rebuild of the new drive. We'll update this post as soon as we have more information as far as when we're doing this.

The machine is now optimal with the problem that caused the issues addressed.

The machine is now online and services starting up.

The machine is rebooting now should be back online in about 10 minutes.

90% complete and we've found the cause which we'll take care of once the server is online. It won't require any further maintenance.

70% complete

56% complete.

20% complete file system check

The saturn server continues to set partitions read only so we're going to be taking it down immediately (10:25am CST) to run a check and see if we can find the culprit for why this has started happening. So we'll investigate possible hardware problems as well. The file system check portion will take about an hour any other maintenance is not determined as of yet. This is necessary to do now as certain partitions you're unable to write data to now.

Engineers have completed the reload of BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01. This has resolved the immediate issues with customer connectivity. During work on these routers, it was determined the same issue is occurring at a much smaller scale at a number of network POPs around the SoftLayer backbone. Engineers will be reloading each of those routers later tonight. Prior to each reload, traffic will be shifted off to redundant paths. Engineers expect no interruption to customer services during these reboots; however, some customers may experience higher latency as their routes shift to other paths.

Engineers have completed the reload of BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01. This has resolved the immediate issues with customer connectivity. During work on these routers, it was determined the same issue is occurring at a much smaller scale at a number of network POPs around the SoftLayer backbone. Engineers will be reloading each of those routers later tonight. Prior to each reload, traffic will be shifted off to redundant paths. Engineers expect no interruption to customer services during these reboots; however, some customers may experience higher latency as their routes shift to other paths.

Engineers have completed the reload of BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01. This has resolved the immediate issues with customer connectivity. During work on these routers, it was determined the same issue is occurring at a much smaller scale at a number of network POPs around the SoftLayer backbone. Engineers will be reloading each of those routers later tonight. Prior to each reload, traffic will be shifted off to redundant paths. Engineers expect no interruption to customer services during these reboots; however, some customers may experience higher latency as their routes shift to other paths.

Refer to:

================================================== SoftLayer Engineers will be performing emergency maintenance on BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01 in an effort to correct routing issues customers are currently experiencing to various destinations including AT&T residential services. In conjunction with Cisco, Engineers have identified a bug that has resulted in the router entering a degraded performance state. To correct this degraded state, the router must be reloaded. Customers can expect to see latency / packetloss as routes reconverge across redundant routers on both the public and private networks. While the maintenance window is set for 2 hours, we expect the maintenance to take no longer than 20 minutes during each router reload. Customers in the Seattle datacenter will lose backend connectivity to all other datacenters (local Seattle private network traffic will not be impacted). ==================================================

================================================== SoftLayer Engineers will be performing emergency maintenance on BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01 in an effort to correct routing issues customers are currently experiencing to various destinations including AT&T residential services. In conjunction with Cisco, Engineers have identified a bug that has resulted in the router entering a degraded performance state. To correct this degraded state, the router must be reloaded. Customers can expect to see latency / packetloss as routes reconverge across redundant routers on both the public and private networks. While the maintenance window is set for 2 hours, we expect the maintenance to take no longer than 20 minutes during each router reload. Customers in the Seattle datacenter will lose backend connectivity to all other datacenters (local Seattle private network traffic will not be impacted). ==================================================

================================================== SoftLayer Engineers will be performing emergency maintenance on BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01 in an effort to correct routing issues customers are currently experiencing to various destinations including AT&T residential services. In conjunction with Cisco, Engineers have identified a bug that has resulted in the router entering a degraded performance state. To correct this degraded state, the router must be reloaded. Customers can expect to see latency / packetloss as routes reconverge across redundant routers on both the public and private networks. While the maintenance window is set for 2 hours, we expect the maintenance to take no longer than 20 minutes during each router reload. Customers in the Seattle datacenter will lose backend connectivity to all other datacenters (local Seattle private network traffic will not be impacted). ==================================================

Date: 12-30-2010 Start time (CDT): 02:00 Services Affected: Public Network (Globally) Device: Public Routers Location: Global Duration: Emerging Issue Description: SoftLayer Engineers are aware of a routing issue impacting a subset of customers globally on the SoftLayer backbone. Engineers have traced this issue back to a hardware forwarding limitation on a number of Cisco routers int he backbone. Engineers are currently in contact with Cisco to determine the best course of action to resolve this issue. We apologize for the inconvenience and appreciate your patience during this issue.

Date: 12-30-2010 Start time (CDT): 02:00 Services Affected: Public Network (Globally) Device: Public Routers Location: Global Duration: Emerging Issue Description: SoftLayer Engineers are aware of a routing issue impacting a subset of customers globally on the SoftLayer backbone. Engineers have traced this issue back to a hardware forwarding limitation on a number of Cisco routers int he backbone. Engineers are currently in contact with Cisco to determine the best course of action to resolve this issue. We apologize for the inconvenience and appreciate your patience during this issue.

Date: 12-30-2010 Start time (CDT): 02:00 Services Affected: Public Network (Globally) Device: Public Routers Location: Global Duration: Emerging Issue Description: SoftLayer Engineers are aware of a routing issue impacting a subset of customers globally on the SoftLayer backbone. Engineers have traced this issue back to a hardware forwarding limitation on a number of Cisco routers int he backbone. Engineers are currently in contact with Cisco to determine the best course of action to resolve this issue. We apologize for the inconvenience and appreciate your patience during this issue.

Networking issue for users who's ISP's using att. Networking is investigating what's causing it.