Topic: Unscheduled outage May 29th 2015

I've received several reports of problems related to the chumby service that began sometime on the evening on May 29th, 2015.

Apparently what happened was some sort of catastrophic power outage at our hosting service that lasted for several hours.  Here is the log from Linode about the event (in reverse chronological order):

Monitoring - We are still actively working with a couple servers and will open support tickets with affected customers. Utility power has been restored and the faulty generator is now fully functional. We will continue to monitor for issues and await an official RFO from the datacenter which we will post early next week.
May 30, 13:48 UTC

Update - We have about 13 damaged servers left and are working on transplanting their drives to hot-spare servers.
May 30, 09:08 UTC

Update - Most Linodes in Fremont should be booted at this time. We continue to work on the servers that were damaged due to the power outage.
May 30, 08:04 UTC

Update - Power was restored at approximately 11:10PM PDT and Linodes are booting. There may be several servers that need special attention due to the power failure and we are investigating those at this time.
May 30, 06:50 UTC

Update - Our upstream provider has provided a new estimated time to restoration of 11:30PM PDT.
May 30, 05:08 UTC

Update - Our upstream provider has provided a new estimated time to restoration of 10:00PM PDT.
May 30, 04:37 UTC

Update - At approximately 6:30PM PDT, the Fremont datacenter experienced a power utility outage. One out of eight generators also experienced an electromechanical failure. The estimated time to restoration is currently 9:30PM PDT. Linode has all hands on deck to get Linodes online as soon as power is restored.
May 30, 03:21 UTC

Update - We have received word from our colocation provider that there has been a power event in a section of the Fremont datacenter. The affected space is where a critical part of the datacenter network is located. The datacenter's electric provider is working with staff to restore power as soon as possible. Please watch this status page for further updates.
May 30, 02:25 UTC

Identified - We are aware of an issue within our Fremont datacenter and are investigating at this time. We will provide additional information as it becomes available.
May 30, 01:47 UTC

Re: Unscheduled outage May 29th 2015

we have been notified that there is an issue with the system that supports our content server that will require a shutdown and migration.

these migrations are typically brief in nature, but will result in a brief reduction in functionality.

Cleaning up any loose bits and bytes.

Re: Unscheduled outage May 29th 2015

Maintenance has been completed and all services are online at this time.

Cleaning up any loose bits and bytes.