Topic: Unscheduled outage May 29th 2015
I've received several reports of problems related to the chumby service that began sometime on the evening on May 29th, 2015.
Apparently what happened was some sort of catastrophic power outage at our hosting service that lasted for several hours. Here is the log from Linode about the event (in reverse chronological order):
Monitoring - We are still actively working with a couple servers and will open support tickets with affected customers. Utility power has been restored and the faulty generator is now fully functional. We will continue to monitor for issues and await an official RFO from the datacenter which we will post early next week.
May 30, 13:48 UTC
Update - We have about 13 damaged servers left and are working on transplanting their drives to hot-spare servers.
May 30, 09:08 UTC
Update - Most Linodes in Fremont should be booted at this time. We continue to work on the servers that were damaged due to the power outage.
May 30, 08:04 UTC
Update - Power was restored at approximately 11:10PM PDT and Linodes are booting. There may be several servers that need special attention due to the power failure and we are investigating those at this time.
May 30, 06:50 UTC
Update - Our upstream provider has provided a new estimated time to restoration of 11:30PM PDT.
May 30, 05:08 UTC
Update - Our upstream provider has provided a new estimated time to restoration of 10:00PM PDT.
May 30, 04:37 UTC
Update - At approximately 6:30PM PDT, the Fremont datacenter experienced a power utility outage. One out of eight generators also experienced an electromechanical failure. The estimated time to restoration is currently 9:30PM PDT. Linode has all hands on deck to get Linodes online as soon as power is restored.
May 30, 03:21 UTC
Update - We have received word from our colocation provider that there has been a power event in a section of the Fremont datacenter. The affected space is where a critical part of the datacenter network is located. The datacenter's electric provider is working with staff to restore power as soon as possible. Please watch this status page for further updates.
May 30, 02:25 UTC
Identified - We are aware of an issue within our Fremont datacenter and are investigating at this time. We will provide additional information as it becomes available.
May 30, 01:47 UTC