Brian Posted December 9, 2009 Report Posted December 9, 2009 Titan just fell offline, at the outset appears to be filesystem related. We're investigating this now.
Brian Posted December 9, 2009 Author Report Posted December 9, 2009 The server was rebooted and we now have to run fsck on the /usr partition
Tony Posted December 9, 2009 Report Posted December 9, 2009 There are problems with the superblock on the /usr partition. We were attempting to find an alternative via console which is not exactly easy. We've switched over to the recovery operating system which is much more responsive when we're not directly at the machine. So the machine is actually responding to pings but we're still working on finding a suitable superblock to recover the file system.
Tony Posted December 9, 2009 Report Posted December 9, 2009 We have been unable to recover this partition. We're going to use R1Soft to restore it from a previous restore point. We're going to do this as fast as we can but we estimate it may take several hours for us to complete all of this then hope the machine comes back online.
Brian Posted December 9, 2009 Author Report Posted December 9, 2009 It appears we've resolved the issues without having to resort to R1Soft, the server is rebooting one last time now.
Brian Posted December 9, 2009 Author Report Posted December 9, 2009 The server is online, we're currently resolving issues fsck was unable to. We expect to have all services back online shortly.
Brian Posted December 9, 2009 Author Report Posted December 9, 2009 All services and sites are now back online. We're continuing to investigate the root cause of this issue, and will closely monitor the server over the next 24 hours to ensure everything is operating in optimal health.
Tony Posted December 9, 2009 Report Posted December 9, 2009 There appears to be some lingering issues. We're still working on solving them.
Tony Posted December 9, 2009 Report Posted December 9, 2009 The one issue we know of that still exists is interment mysql connection issues. We're working hard on figuring out why but we have no ETA at this time.
Tony Posted December 9, 2009 Report Posted December 9, 2009 mySQL is up besides Innodb databases we're working on getting backups for those.
Tony Posted December 9, 2009 Report Posted December 9, 2009 Still working on the innodb databases but we ended up replacing one of the drives in the raid which was reporting medium errors which typically is a sign it's having issues.
Tony Posted December 9, 2009 Report Posted December 9, 2009 Innodb databases are now fixed as well. Now we're just waiting on the raid to rebuild then hopefully should be no more issues.
Tony Posted December 9, 2009 Report Posted December 9, 2009 The raid array is once again optimal. As far as other issues we have fixed a few things but other than that everything should be back to normal so we're considering this resolved at this point. If you run into any issues though do not hesitate to make a support ticket.
Recommended Posts