Venus Ongoing Issue [11-26-2008]

[Cody R.]

For some reason the /home partition on the Venus server keeps getting set to read-only causing issues for anything requiring to make / update a new file.

We've rebooted the machine so it should be fine now.

We'll update this thread once we get more information.

[Tony]

We have suspicion it's just a fluke /home has been the one with the issue. We believe it's a problem with the raid array which we are investigating now. We will most likely need to swap out a bad drive in the array which is causing the operating system to set the file system read only when it sees errors.

[Cody R.]

After having the drives / RAID inspected it appears to be fine - if it continues to be an issue we'll be forced to run FSCK on the drive.

[Cody R.]

The /home filesystem set itself to read-only again so we're going to run FSCK to see if it picks up on anything.

We'll update the ticket upon completion.

[Cody R.]

As a result of FSCK the machine will have some downtime - the amount of time can range depending on numerous variables.

We'll try to get an accurate ETA here shortly for you.

[Cody R.]

We're starting this now.

Start Time: 3:50PM EST

[Cody R.]

The FSCK has been completed and reported no errors - we'll be closely monitoring the suspect partition to verify everything is in working order.

[Tony]

It hasn't happened in a while but we're verifying the raid array which takes a while but can expose problems the raid controller would not show by just checking it's status.

If it does happen after the verify we have reason to believe the ram may have gone bad. But we'll move towards that if is still an issue.

[Tony]

Things have been fine for almost 24 hours. It seems the problem has been resolved.