Titan Latest Topics

Kernel Upgrade [04/17/2012]

Tue, 17 Apr 2012 17:18:44 +0000

We will be updating the kernel on the Titan server to the latest version between 11:00PM and 1:00AM PDT.

The maintenance window is for 2 hours however the estimated downtime is 5 to 10 minutes.

Date: 04/04/2012

Start time (PDT): 04/19/2012 11:00PM

End time (PDT): 04/20/2012 1:00AM

Duration: 2 hours

Estimated Down Time: 5-10 minutes

Shared Hosting Accounts Migration [01/04/2012] - [01/06/2012]

Tue, 27 Dec 2011 20:43:45 +0000

Starting on Wednesday January 4th we will be migrating Titan shared hosting accounts over to a new shared hosting specific server sea006 with the hostname sea006.hawkhost.com. The reason for this is currently Titan is one of our mixed servers and we have been migrating mixed servers to strictly shared or reseller. The one major advantage is once migrated over shared hosting users will have access to the shared hosting specific features that are currently not available on our mixed servers. It also means no longer will reseller users be able to affect services with their user accounts. In this case it will also be a hardware upgrade moving from a Dual Xeon 5520 with 24gb ram and 4x600GB Raid-10 to a Dual Xeon X5670 with 24gb of memory and 4x600GB Raid-10.

Since all our IP addresses are routed to our VLAN's rather than a specific server we can make this migration very seamless. We will migrate data then re-route the IP address over to the new server. This dramatically reduces the chances of data sync issues which are usually associated with server moves due to IP addresses changing. That is not to say it cannot happen but we are available to assist with any issues users may run into if for whatever reason their data does go out of sync. As for any downtime associated with the migration we estimate it'll be about 1 minute at most as we need to remove the IP from the old server and route it to the new one which can take us about a minute.

How the migration will work is we will start migrating users on dedicated IP's first starting on Monday. After that is completed we will move onto users with shared IP addresses. We hope that this entire migration will only last two days we hope but we've created a window of three days.

The only thing to keep in mind is once migrated it is not advised to be connecting to the titan hostname if you currently are. While the data will stay there for now eventually all shared hosting user data will be removed from the Titan server as it will be fully converted to a reseller hosting only server. So make sure to connect to the sea006.hawkhost.com server once the window has been completed.

If you have any questions regarding the maintenance do not hesitate to contact us. Our system administration department is available to answer any questions regarding this.

Start Date: 01/04/2011

End Date: 01/06/2011

Estimated Down Time: less than 5 minutes

Duration: 3 days

Emergency Maintenance [12/7/2012]

Thu, 08 Dec 2011 03:51:55 +0000

Due to an unexpected power failure in our datacenter facility, we will be required to perform an emergency fsck on this server. We will update this thread as we progress. Please direct any questions to our helpdesk at https://support.hawkhost.com

RAID BBU Replacement [10/10/2011]

Sun, 09 Oct 2011 06:28:31 +0000

We've received RAID alerts that the BBU failed on this machine (non-service impacting). However to ensure the integrity of the RAID in case of a failure we're scheduling an emergency maintenance to get this replaced on Monday October 10th.

If you have any questions or concerns please contact us via our support desk.

Date: 09/16/2011

Start time (PDT): 1:00am

End time (PDT): 4:00am

Duration: 3 hours

Estimated Down Time: 30 minutes

Memory Upgrade [05/18/2011]

Fri, 13 May 2011 20:00:47 +0000

On Wednesday May 18th between 9:00am and 12:00pm PDT we will be adding additional memory to the Titan server to improve performance and general system stability. Adding the additional memory will require the machine to be turned off so there will be between 30 and 45 minutes of down time.

If you have any questions regarding this maintenance notice please do not hesitate to contact our systems administration department.

Date: 05/18/2011

Start time (PDT): 9:00am

End time (PDT): 12:00pm

Duration: 3 hours

Estimated Down Time: 30 minutes

PHP 5.2 Upgrade [04/18/2011]

Sun, 17 Apr 2011 07:38:37 +0000

On Monday April 18th between 11:00pm and 1:00pm PDT we will be upgrading the Titan server's PHP 5.2.x to the latest 5.2.x version. We do not expect any down time during the maintenance window. We will be upgrading the system PHP to the latest version first then the web servers so for a short time there may be different versions handling command line cron jobs. The reason for this is to get various fixes for LSAPI as well as general PHP fixes as well.

If you have any questions about this maintenance window do not hesitate to contact support.

Date: 04/07/2011

Start time (PDT): 11:00am

End time (PDT): 1:00pm

Estimated Down Time: None

Duration: 2 hours

Kernel Upgrade [04/12/2011]

Sun, 10 Apr 2011 18:26:47 +0000

We'll be performing a kernel upgrade on on Wednesday April 12th between 12PM-3PM PDT. While we use KSplice to keep our kernels up-to-date for critical bugs and security fixes it doesn't pull in new features from the upstream kernel. As a result we perform semi-annual kernel updates to keep everything up-to-date and in-line with upstream.

Date: 04/12/2011

Start time (PDT): 12:00pm

End time (PDT): 3:00pm

Duration: 1 hour

Estimated Down Time: 10 minutes

Kernel Upgrade [04/06/2011]

Mon, 04 Apr 2011 17:42:23 +0000

We'll be performing a kernel upgrade on on Wednesday April 6th between 12PM-3PM PDT. While we use KSplice to keep our kernels up-to-date for critical bugs and security fixes it doesn't pull in new features from the upstream kernel. As a result we perform semi-annual kernel updates to keep everything up-to-date and in-line with upstream.

Date: 04/06/2011

Start time (PDT): 12:00pm

End time (PDT): 3:00pm

Duration: 1 hour

Estimated Down Time: 10 minutes

Kernel Upgrade [01/06/2011]

Tue, 04 Jan 2011 04:47:49 +0000

We'll be performing a kernel upgrade on Thursday the 6th between 8:00am and 9:00am PST. This upgrade will allow us enhanced resource tracking to help address resource abuse issues. We estimate the server will be down for about 10 minutes while the machine is rebooting to the new kernel.

Date: 01/06/2011

Start time (PST): 8:00am

End time (PST): 9:00am

Duration: 1 hour

Estimated Down Time: 10 minutes

FSCK [09/17/2010]

Sat, 18 Sep 2010 04:44:19 +0000

We are taking the Titan server offline to fsck the /home partition it is reporting as read only and we have reason to believe there is some file system corruption. The machine is offline as of right now at about 9:42 PM PDT and we hope it will be down no more than an hour. We will update this ticket when we have more information.

DDOS [8/30/2010]

Mon, 30 Aug 2010 22:57:45 +0000

The Titan server is currently (8/30/2010 ~ 3:57 pm PDT) experiencing intermittent public network outages due to what appears to be a very large distributed denial of service attack. We are already working with networking on filtering this attack and we will post more information as we receive it from them.

PHP Upgrade [08/05/2010]

Sat, 31 Jul 2010 01:24:18 +0000

On Thursday August 5th between 9:00am and 12:00pm PDT we will be upgrading the Titan server's PHP to the latest version. We do not expect any down time during the maintenance window. We will be upgrading the system PHP to the latest version first then the web servers so for a short time there may be different versions handling command line cron jobs. The reason for this upgrade is to get the latest version of PHP with all the latest fixes. Along with PHP we will also be upgrading the LSAPI version which will bring some performance benefits on the web server side.

If you have any questions about this maintenance window do not hesitate to contact support.

Date: 08/05/2010

Start time (PDT): 9:00am

End time (PDT): 12:00pm

Estimated Down Time: None

Duration: 3 hour

Kernel Update [06/18/2010]

Thu, 10 Jun 2010 20:21:04 +0000

We will be updating the Titan server kernel to the latest version on Friday June 18th between 8:00am PDT and 11:00am PDT. We estimate this will result in about 10 minutes of down time while the server reboots. Even though we're running Ksplice to minimize reboots we like to do an update every time there is a major operating system update. In this case CentOS 5.5 was released so we'd like to run the latest 5.5 kernel to get any non security related improvements which Ksplice does not do as it's strictly for security updates.

Date: 06/18/2010

Start time (PDT): 8:00am

End time (PDT): 11:00am

Duration: 3 hours

Estimated Down Time: 10 minutes

Failed Drive Replacement [04/26/2010]

Mon, 26 Apr 2010 17:41:48 +0000

We have been notified by the raid card on the Ttitan server that one of the drives has failed. As a result we will be replacing it today (April 26th) as soon as possible so we can start the rebuilding process. This will be updated once we start the replacement process. This will be a non service impacting maintenance as the server has hot swap capability.

Once the drive has been replaced the server will have slightly higher i/o wait while the new drive are rebuilding.

Date: 04/26/2010

Start time (PDT): 10:30am

End time (PDT): ?

Estimated Down Time: None

Hardware Migration [03/05/2010] - [03/08/2010]

Mon, 01 Mar 2010 23:38:22 +0000

We will be migrating the accounts on the Titan server starting on Thursday March 5th and this will last until Monday March 8th. The reason for this upgrade is our continued commitment to having exceptional quality of service. We've seen an increasing number of drive failures for our servers running Raptor drives. Titan is the last web server we have running them so we're going to migrate it to new hardware that uses SAS drives.

For the migration process we will be moving accounts in sections based on IP address. We anticipate no more than 5 minutes of down time per account in order to make the switch. We migrate the account data then re-route the IP address to the new server. Thus there is no down time associated with the upgrade.

The time frame given we estimate will not be nearly this long. In the past we have migrated to other hardware without any down time in less than three days. So our 4 days just gives us plenty of time to do it right and make sure there is no interruption in service.

The specifications are as follows:

Old Titan:

Dual Xeon 5430

12GB RAM

4x300GB SATA Raptor Drives 10K RPM Raid-10

New Titan:

Dual Xeon 5520

12GB RAM

4x300GB SAS 15K RPM Raid-10

So we'll be using the next generation Xeon CPU's as well as making use of 15K RPM SAS drives over the 10K RPM SATA Raptor drives.

If you have any concerns about this migration please do not hesitate to contact us.

Start Date: 03/05/2010

End Date: 03/08/2010

Estimated Down Time: less than 5 minutes

Duration: 4 days

cPanel 11.25 Upgrade [02/20/2010]

Fri, 19 Feb 2010 03:19:16 +0000

On Saturday February 20th we will be upgrading the Titan server to the latest version of cPanel 11.25. We anticipate this upgrade taking no longer than an hour however with us doing many machines we cannot give a concrete time when each server will be done.

We do not anticipate any service interruptions during this time. cPanel may be slower than usual when the upgrading is being performed though. cPanel 11.25 brings many great new features and has reached stable and has been tested by us in a production environment on a limited number of servers.

If you have any questions about this maintenance window do not hesitate to contact us.

Date: 02/20/2010

Duration: 24 hours

Estimated Down Time: None

Reboot + Drive Swap [02/07/2010]

Sun, 07 Feb 2010 19:59:47 +0000

As of this moment 12pm PST we're going to be rebooting this server as there are some issues with the file system we cannot correct while it's live. Once back up we'll be replacing a bad drive that the raid card just reported as bad.

File System Issues [02/07/2010]

Sun, 07 Feb 2010 12:19:54 +0000

Titan is having file system issues this morning, more information forthcoming.

Kernel Update [01/15/2010]

Tue, 12 Jan 2010 02:53:56 +0000

We will be updating the Titan server kernel to the latest version on Friday January 15th between 6:00am PST and 9:00am PST. We estimate this will result in about 10 minutes of down time while the server reboots.

Keeping kernels up to data is a necessary step in order to stay on top of performance improvements as well as have the latest security fixes.

Date: 01/15/2010

Start time (PST): 6:00am

End time (PST): 9:00am

Duration: 3 hours

Estimated Down Time: 10 minutes

Aggregate Switch Replacement [12/29/2009]

Mon, 21 Dec 2009 18:57:25 +0000

Datacenter Engineers have identified a hardware issue an aggregate switch which provides connectivity to our Titan. To prevent an unplanned outage resulting in significant downtime, Engineers will be replacing the upstream front end aggregate switch during a scheduled maintenance window. During this maintenance, accounts on the Titan server will not be reachable. While the maintenance window is set for 1 hour, we expect no longer than 15-20 minutes of downtime.

Data Center Maintenance - FAS05.SR02.SEA01 [12/29/2009]

Date: Tuesday, December 29, 2009 (12/29/2009)

Start time (PST): 01:00

End time (PST): 02:00

Services affected: Public Network

Location: SEA01

Duration: 1 hour

Memory Upgrade [12/19/2009]

Tue, 15 Dec 2009 18:39:57 +0000

December 19th between 9:00am PST and 12:00PM PST we will be upgrading the Titan servers memory. We will be adding an additional 6GB of memory to the server to bring it to a total of 12GB. The window is for three hours but we estimate that the machine will be down no longer than 30 minutes.

The reason for this upgrade it to bring the memory specification to match that of all our other servers. This will add additional stability and performance to the server as it will have more ram to use for applications and for caching purposes.

If you have any questions or concerns about this maintenance window do not hesitate to contact support.

Date: 12/19/2009

Start time (CST): 9:00am

End time (CST): 12:00pm

Duration: 3 hours

Estimated Down Time: 30 Minutes

Filesystem Issues [12/09/09]

Wed, 09 Dec 2009 07:27:19 +0000

Titan just fell offline, at the outset appears to be filesystem related. We're investigating this now.

Emergency Network Maintenance [11/30/2009]

Wed, 25 Nov 2009 01:01:23 +0000

There will be emergency network maintenance on FCR01 which our Titan server is on. This window is on Monday November 30th starting at 12:30am PST and lasting until 1:30am PST. Although the window is for 1 hour we estimate that there will only be a down time of 15-20 minutes. This maintenance is necessary to correct an issue that could cause network interruptions that would not be scheduled.

Further information about this can be found at: Emergency Network Maintenance [11/30/2009]

Emergency Data Center Maintenance - FCR01.SEA01 [11/30/2009]

Date: Monday, November 30, 2009 (11/30/2009)

Start time (PST): 00:30

End time (PST): 01:30

Services affected: Public Network

Location: SEA01

Duration: 1 hour

Ddos [10/17/2009]

Sat, 17 Oct 2009 04:50:19 +0000

Titan has come under DDOS. We are currently working on the issue further updates will follow.

Raid Card Swap [10/10/2009]

Sat, 10 Oct 2009 16:34:43 +0000

We have received notification of another failed drive on the Titan server this time being on the same port as a drive that was replaced just recently. This is pretty rare thing to happen so we believe it be best to replace the raid card. It has also been throwing various errors and emails us events suggesting it's just not right. This could be due to the power issue a while ago or this card could have been faulty for some time and it just took us seeing two drives on the same port fail within a short while.

We will be doing this as soon as possible the current estimate is we'll be replacing the card at about 9:45AM PDT and the machine should be down no more than 20 minutes. We're very sorry about the short notice regarding this but when the raid card is acting strange we rather replace it as soon as possible then avoid possible data loss.