Brian Posted November 9, 2010 Report Posted November 9, 2010 Skyline crashed and we were forced to reboot it as a of 11:40PM CST today. More information forthcoming as we have it.
Brian Posted November 9, 2010 Author Report Posted November 9, 2010 System is currently in rescue mode and we are running fsck on the effected partitions.
Brian Posted November 9, 2010 Author Report Posted November 9, 2010 fsck is at 25% on the /home partition. We anticipate having to run fsck on multiple partitions once /home is completed.
Brian Posted November 9, 2010 Author Report Posted November 9, 2010 We are now investigating a possible failed drive as the array is showing degraded.
Brian Posted November 9, 2010 Author Report Posted November 9, 2010 Currently at 50% on the fsck of the /home partition.
Brian Posted November 9, 2010 Author Report Posted November 9, 2010 Confirmed to be failing disk. The raid did not respond correctly and caused file system issues. We'll be swapping the drive as soon as we can.
Brian Posted November 9, 2010 Author Report Posted November 9, 2010 fsck is now at 70% on the /home partition
Brian Posted November 9, 2010 Author Report Posted November 9, 2010 fsck is at 90% on the /home partition right now.
Brian Posted November 9, 2010 Author Report Posted November 9, 2010 fsck on /home partition completed successfully, we're now running fsck on the /var partition which is at 50%. We will be doing /usr after /var completes.
Cody R. Posted November 9, 2010 Report Posted November 9, 2010 The /var partition is finished and we're now starting on /usr. It's currently at 50% completed.
Cody R. Posted November 9, 2010 Report Posted November 9, 2010 The file system checks are now completed. The machine is rebooting now.
Brian Posted November 9, 2010 Author Report Posted November 9, 2010 Machine is back online and all sites will begin loading again here shortly.
Cody R. Posted November 9, 2010 Report Posted November 9, 2010 The machine is currently online and booting services. Please allow a few minutes for it to stabilize / warm caches. We will also be replacing a hard drive in this machine shortly however we anticipate no downtime doing this as all of our machines have hot swappable bays.
Cody R. Posted November 9, 2010 Report Posted November 9, 2010 Everything is online and the RAID is rebuilding. Once its completed and the RAID in is an optimal state we'll be swapping out the faulty drive. There may be some slowness in regards to IO during the rebuild.
Cody R. Posted November 9, 2010 Report Posted November 9, 2010 The RAID is still rebuilding and IO has settled down / everything has been online and responsive for awhile. Once the RAID is done we'll still be swapping out the bad drive.
Tony Posted November 9, 2010 Report Posted November 9, 2010 The drive has been swapped and is currently rebuilding.
Brian Posted November 10, 2010 Author Report Posted November 10, 2010 Array rebuild has been completed and the system is back at 100% health.
Recommended Posts