Tony Posted March 22, 2009 Report Posted March 22, 2009 One of the drives on the raid-10 array of Yoda has failed. This was causing large amounts of i/o wait on the node as the drive had not yet been dropped. Once the drive was dropped from the raid array due to inconsistencies the i/o wait has come back down. We are already preparing to replace the bad drive as our machines support hot-swap which means we can replace the drive on the fly. We do not have a time as of yet as we're in the process of coordinating with datacenter staff. We will update this thread once we have an exact time on when the bad drive is being replaced. Date: 03/21/2008 Start time (EST): 10:00pm EST End time (EST): Ongoing Estimated Down Time: None Duration: Unknown ruptuphorse, vodsaccourorp, Joxofssofsdip and 2 others 5
Tony Posted March 22, 2009 Author Report Posted March 22, 2009 We will be replacing the drive within the next 30 minutes. After that it'll be a matter of the rebuild which may cause slightly higher i/o wait but nothing major to slow things to a crawl.
Tony Posted March 22, 2009 Author Report Posted March 22, 2009 The drive has been replaced. It's just rebuilding now.
Tony Posted March 22, 2009 Author Report Posted March 22, 2009 It just finished rebuilding things should be getting back to normal load wise now. There were a few spikes near the end of the rebuild. There was not much we could do about it as keeping data safe out ways a few minutes of less then stellar performance.
Recommended Posts