Tony Posted February 7, 2010 Report Posted February 7, 2010 As of this moment 12pm PST we're going to be rebooting this server as there are some issues with the file system we cannot correct while it's live. Once back up we'll be replacing a bad drive that the raid card just reported as bad.
Tony Posted February 7, 2010 Author Report Posted February 7, 2010 The machine is back online serving requests we'll be swapping the bad drive and rebuilding it in the background.
Tony Posted February 7, 2010 Author Report Posted February 7, 2010 The bad drive has been replaced with a new one and it will rebuild in the background for the next few hours.
Tony Posted February 7, 2010 Author Report Posted February 7, 2010 It appears as though there is some corruption with the /usr partition. We're going to have no choice but to take the machine down to run a fsck on it. Otherwise we're going to continue to have outages and other related problems. We'll be taking it down in the next 5 minutes.
Tony Posted February 7, 2010 Author Report Posted February 7, 2010 The fsck is currently running on /usr. It should hopefully finish soon as it is a small partition.
Tony Posted February 7, 2010 Author Report Posted February 7, 2010 Forgot to hit enter last time but the fsck has completed and we're out of the rescue system and booting the server back up now.
Tony Posted February 7, 2010 Author Report Posted February 7, 2010 It is now back up ans as far as we can tell this should resolve the issues. We'll keep this open until we know for sure though.
Tony Posted February 7, 2010 Author Report Posted February 7, 2010 The rebuild has completed and we're no longer seeing issues with the file system either so the fsck fixed that.
Recommended Posts