Pseullyneense

April was unfortunately a tough month at Devoted Host as our primary datacenter SoftLayer was running on generators due to the inclement weather in the area. When switching back to utility power a 2500amp breaker failed placing all our servers on UPS backup power. Unfortunately after 30 minutes machines started to go offline. After about 3 hours of work from SoftLayer electricians onsite power was restored to servers.

I'd like to make things clear here things happen from time to time that are unexpected. Hardware does fail the best you can do is restore service as fast as possible. The 3 hour timeframe this was done in was significantly better than other datacenters in the past have replaced breakers of this size.

So here is our uptime reports for all our servers it includes the no longer active Apollo as well as the now active Jupiter.

On Saturday March 31st between 1pm EST and 5pm EST we

On Saturday between 10pm EST and 12am EST we will be migrating Mars, Mercury and Venus to new primary IP addresses. The reason for this is to expand out our web server VLAN in order to accommodate more machines. Once this is done it will now be possible to easily move secondary IP's from one server to another. We do not expect any down time during this time however anyone referencing the servers by their main IP can no longer do this.

Once this is completed over the next few months we have plans of upgrading Mars, Mercury and Venus to much higher specification machines gradually. Thanks to the maintenance on Saturday we will also be able to transfer accounts to the upgraded servers without the need for dns changes or any down time or issues what so ever.

As always if you have any questions about the maintenance please contact our support department.


Date: 7/14/2007
Start time (EDT): 10:00pm
End time (EDT): 12:00am
Duration: 5-10 minutes

We will be updating the kernel on the Yoda VPS node to the latest version between 12:00am and 1:00am EST.

The maintenance window is for 4 hours however the estimated downtime is 5 to 10 minutes for the actual downtime.

Date: 05/30/2011 - 5/30/2011
Start time (EST): 05/30/2011 12:00am
End time (EST): 05/30/2011 1:00am
Duration: 1 hour
Estimated Down Time: 0-10 minutes

March was a shortened month as we only started doing the new monitoring on the 14th. So keep in mind this dataset is inaccurate and does not reflect the overall uptime that month actually had. I'm going to try to explain any downtime reflected on the reports as well as we generally know why something went down. This report also includes a few images of the uptime. Feel free to use them as you wish.

This a topic dedicated to potential issues you could experience due to the migration over to LiteSpeed.

Scheduled Backbone Maintenance [09/18/2011]
Date: Sunday, September 18, 2011 [09/18/2011]
Start Time: 12:00 AM CDT
End Time: 08:00 AM CDT
Services affected: Global Backbone
Location: GLOBAL
Duration: 8 hours

==================================================

Network Engineers will be performing global maintenance on their backbone. During this change, Engineers will be implementing a number of policy changes. This work is required to support continued growth within the network and future global expansion. Due to the scope of this change, the work has been scheduled for the following day:

Sunday September 18th 00:01 - 08:00 CDT

While Engineers expect no outage during this maintenance, customers may experience higher latency and/or packet loss while routes reconverge across the network.

==================================================

On Wednesday July 11th starting at 2PM CDT we will be taking the Mars server down to run a file system check due to some reported errors after reviewing system logs. We estimate the actual file system check will take less than an hour, however we've left the window open for 3 hours in the event we run into unexpected issues.

Date: 07/11/2012
Start time (CDT): 2:00PM
End time (CDT): 5:00PM
Duration: 3 hours
Estimated Down Time: 60 minutes or less

Throughout this window our support team will be available for any questions. They can be contacted via ticket at https://support.hawkhost.com.

Mars came under a large distributed denial of service attack around 10:00AM EST - we mitigated it within a few minutes. There may have been a brief period of slowness or service outage as a result.

We're currently monitoring the service and working with data center technicians to fully mitigate the attack.

We'll leave this ticket open for 24 hours until we can confirm service is full restored and back to normal.

We've received RAID alerts that the BBU failed on this machine (non-service impacting). However to ensure the integrity of the RAID in case of a failure we're scheduling an emergency maintenance to get this replaced on Wednesday October 31st.

If you have any questions or concerns please contact us via our support desk.

Date: 10/31/2012
Start time (CDT): 9:00am
End time (CDT): 12:00pm
Duration: 3 hours
Estimated Down Time: 30 minutes

Due to filesystem inconsistencies we've been forced to bring Mustang down and run a FSCK. We'll be posting updates as this progresses in this thread.

The estimated downtime is about 60 minutes.

On Thursday July 29th between 9:00am and 12:00pm CDT we will be upgrading the Mars server's PHP to the latest version. We do not expect any down time during the maintenance window. We will be upgrading the system PHP to the latest version first then the web servers so for a short time there may be different versions handling command line cron jobs. The reason for this upgrade is to get the latest version of PHP with all the latest fixes. Along with PHP we will also be upgrading the LSAPI version which will bring some performance benefits on the web server side.

If you have any questions about this maintenance window do not hesitate to contact support.

Date: 07/29/2010
Start time (CDT): 9:00am
End time (CDT): 12:00pm
Estimated Down Time: None
Duration: 3 hour