Brian Posted November 9, 2010 Report Share Posted November 9, 2010 Skyline crashed and we were forced to reboot it as a of 11:40PM CST today. More information forthcoming as we have it. Link to comment Share on other sites More sharing options...
Brian Posted November 9, 2010 Author Report Share Posted November 9, 2010 System is currently in rescue mode and we are running fsck on the effected partitions. Link to comment Share on other sites More sharing options...
Brian Posted November 9, 2010 Author Report Share Posted November 9, 2010 fsck is at 25% on the /home partition. We anticipate having to run fsck on multiple partitions once /home is completed. Link to comment Share on other sites More sharing options...
Brian Posted November 9, 2010 Author Report Share Posted November 9, 2010 We are now investigating a possible failed drive as the array is showing degraded. Link to comment Share on other sites More sharing options...
Brian Posted November 9, 2010 Author Report Share Posted November 9, 2010 Currently at 50% on the fsck of the /home partition. Link to comment Share on other sites More sharing options...
Brian Posted November 9, 2010 Author Report Share Posted November 9, 2010 Confirmed to be failing disk. The raid did not respond correctly and caused file system issues. We'll be swapping the drive as soon as we can. Link to comment Share on other sites More sharing options...
Brian Posted November 9, 2010 Author Report Share Posted November 9, 2010 fsck is now at 70% on the /home partition Link to comment Share on other sites More sharing options...
Brian Posted November 9, 2010 Author Report Share Posted November 9, 2010 fsck is at 90% on the /home partition right now. Link to comment Share on other sites More sharing options...
Brian Posted November 9, 2010 Author Report Share Posted November 9, 2010 fsck on /home partition completed successfully, we're now running fsck on the /var partition which is at 50%. We will be doing /usr after /var completes. Link to comment Share on other sites More sharing options...
Cody R. Posted November 9, 2010 Report Share Posted November 9, 2010 The /var partition is finished and we're now starting on /usr. It's currently at 50% completed. Link to comment Share on other sites More sharing options...
Cody R. Posted November 9, 2010 Report Share Posted November 9, 2010 The file system checks are now completed. The machine is rebooting now. Link to comment Share on other sites More sharing options...
Brian Posted November 9, 2010 Author Report Share Posted November 9, 2010 Machine is back online and all sites will begin loading again here shortly. Link to comment Share on other sites More sharing options...
Cody R. Posted November 9, 2010 Report Share Posted November 9, 2010 The machine is currently online and booting services. Please allow a few minutes for it to stabilize / warm caches. We will also be replacing a hard drive in this machine shortly however we anticipate no downtime doing this as all of our machines have hot swappable bays. Link to comment Share on other sites More sharing options...
Cody R. Posted November 9, 2010 Report Share Posted November 9, 2010 Everything is online and the RAID is rebuilding. Once its completed and the RAID in is an optimal state we'll be swapping out the faulty drive. There may be some slowness in regards to IO during the rebuild. Link to comment Share on other sites More sharing options...
Cody R. Posted November 9, 2010 Report Share Posted November 9, 2010 The RAID is still rebuilding and IO has settled down / everything has been online and responsive for awhile. Once the RAID is done we'll still be swapping out the bad drive. Link to comment Share on other sites More sharing options...
Tony Posted November 9, 2010 Report Share Posted November 9, 2010 The drive has been swapped and is currently rebuilding. Link to comment Share on other sites More sharing options...
Brian Posted November 10, 2010 Author Report Share Posted November 10, 2010 Array rebuild has been completed and the system is back at 100% health. Link to comment Share on other sites More sharing options...
Recommended Posts