Tony

The rebuild is complete. Everything should be optimal once again.

The bad drive has been replaced and is now rebuilding.

We have been notified by the raid card on the Jedi server today (August 7th) that one of the drives on the raid array has become bad. We are going to be replacing this drive tonight between 5:00pm and 8:00pm EDT. This was the earliest window that we could do in order to replace the drive. We estimate no down time during the replacement as the server supports replacing drives while the machine is running. Once the drive is replaced it will then require a rebuild which will take several hours to complete. Date: 08/07/2009 Start time (EDT): 5:00pm End time (EDT): 8:00pm Duration: 3 Hours Estimated Down Time: None

It is now accessible again as the system has no known vulnerabilities and is in progress of being open source. Lots of other providers have started using it again and we feel no reason to hold our VPS's back any further. We are still working on our own system but we want something complete before it goes live opposed to something that lacks required features.

That has been completed. That should be the end of this now hopefully.

Verify on the array is now being ran to check for any bad data or drives.

Raid card has been replaced and machine is back online.

The server should be going off line in the next 10 minutes to replace the card. Once replaced we'll re-check the array and hopefully find nothing wrong and it was just the raid card messing with us.

We're going to replace the raid card it's acting very strange as if it's going to fail reporting all sorts of things that make no sense. We're going to do this ASAP and it'll require the machine be off line for hopefully no more than 20 minutes.

At about 1am PDT on August 4th we were alerted to bad blocks being reported on the Titan servers raid array. The server currently shows an optimal state on the array but bad blocks are something that should be investigated. We're going to run a verify on the raid array right now at 12pm PDT to verify the integrity of the data. We're hoping this reveals a drive to be bad which we will then replace. Date: 08/04/2009 Start time (PDT): 12:00pm End time (PDT): Unknown Duration: Unknown Estimated Down Time: None Currently

The rebuild is finished. Everything should be back to normal.

All VZ's are running again and have been for at least 5 minutes if not longer. So just the drive rebuild now.

The drives have been replaced it is now rebuilding the array which unfortunately will mean i/o wait will be much higher than normal due to the fact the server only has two functioning drives currently.

Server is back up all VM's are starting now.

We will be replacing the Backup battery unit in the next 30 minutes. It will require most likely 15 minutes to complete which unfortunately the machine will have to be down to do. Once that is done we will then replace the two bad drives while the server is online (hot swap).

This morning at 9:42 AM CDT we were alerted to a degraded array on the Marlin server. Upon inspection we discovered two failed drives as well as a failed backup battery unit. The battery unit acts as protection against power failure so if data has not yet been written to disk it will when power is restored. Without the battery the write caching featured is then disabled which decreases performance. We are currently still investigating our options in this case as this machine has had a backup battery fail before. We may choose to replace the raid card in this instance. We will be doing this maintenance as soon as possible which will unfortunately require down time. Once an assessment has been made by a datacenter technician we'll update this thread with the estimated down time and when this maintenance will be done.

You can delete the files that's an option or you can disable them through cPanel. You can find the frontpage option under the advanced category within cPanel.

It uses OR. It all comes down to if we do not use these listings our mail servers will be under 10 times as much load if not more. For the one guy who once in a while gets listed it's not worth not having them at all. Any time we've had a machine listed we had no issues resolving the problem in under 24 hours. It's not like we're walking about weeks or months to resolve it.

The use of spamcop and spamhaus are used at the server level they will not be disabled. If someone is sending mail to you and they're on those lists they should resolve the spam complaints and get removed. Until they're not on those lists their mail will be sent back to them saying they're listed on spamhaus or spamcop. The spam assassin portion is account level and is in cPanel.

Just making use of RBL listings like spamcop and spamhaus along with the use of spam assassin.

You cannot do a full restore but if you make a ticket we will be able to restore your backup for you.

The url used would be https://serverhostname:2096/ . We use anonymous server names and the SSL versions work without error as they each have their own SSL certificate.

All machines now have their initial seeds with some having two days worth of backups already. With this completed all backups should now be finished in record time and no longer should there be extended wait times for restores due to backups running.

Just one machine left.

Over 50% completed.