Server Outage (01/04/11)

[Chalex4]

Unfortunately there was an issue with the server we are hosted on today, so the site was unaccessible for quite a while!

UKWSD were aware of the problem this morning and worked to fix it as quickly as possible. Obviously we are now back online, as you are now reading this post!

I can't post updates if the site's down, so check out facebook if you want the lastest next time. Here's all the info from today:

UPDATE @ 09:00: We're currently running an FSCK on this server which is taking a while to complete due to the amount of data and size of the drives.

UPDATE @ 09:55: The inital FSCK has completed highlighting an issue with the data within the /var partition, a further manual FSCK is required and being ran at present.

UPDATE @ 10:10: The server has had to have several runs of FSCK to check the file system, it's doing the last pass now and the ETA is under 30 minutes.

UPDATE @ 11:15: We're still working on this server to restore full service as quickly as possible.

UPDATE @ 13:00: Having completed the FSCK earlier to verify the file system we have had to do a partial restore from our CDP backup to restore a number of corrupt system files, service has been restored however will be slightly degraded whilst the RAID array is rebuilt.

UPDATE @ 13:49: We're going to be taking the server back offline whilst we run a restore of the /var partition from the CDP backup, this should resolve any remaining issues.