Topic: On Chumby Outages...

It would be nice if the chumby could try connecting to an alternate site/server... that could store system information/service notices... when the main servers go down.... Seeing a black screen with blue text this morning was a little disconcerting... (actually having this alternate server keep system notices would be much more pleasant then just being unable to connect)...

Re: On Chumby Outages...

I stupidly spent all morning troubleshooting my network, and resetting the chumby to defaults ect. Before I realized the site was down.
Granted I only thought that because I accidently unplugged the chumby while on and I though I broke it, but still.

It would be wonderfull to even get a "Down for maintenance, ETA 8:30.". Or you know, something like that smile Maybe even a "YOUR CHUMBY ISN'T BROKEN!" lol.

Re: On Chumby Outages...

I'm a new chumby user, and this has seriously diminished my confidence in the reliability of the product. I had planned to ditch my clock radio and use chumby as my alarm clock. As of today, chumby has failed me two times out of four. Is chumby ready to play with the big boys?

Re: On Chumby Outages...

This is only the second time in my couple month ownership, that this has happened.
Plus, as long as your chumby is on if/when it happens, your alarm will still go off. Mine did. Then I unplugged it and tried starting it again ect.

Re: On Chumby Outages...

We're still doing the root cause analysis of what happened here, and we'll let you know when we find out.  The problem occurred at the company where we host our systems, not at Chumby itself.

As to what chumbys should do under these circumstances - chumbys that ware active during this outage should still have had their alarms go off.  If one had restarted the chumby, that would have resulted in alarm failures.

We have a plan for better off-line operation, but it was out in the future a bit, once overall functionality stabilized - the biggest issue is that modifying offline mode requires firmware updates, and thus more difficult and failure-prone to deploy.

Today will no doubt be consumed with internal meetings to post-mortem this event and plans for insulating chumbys from this sort of problem in the future.  We're actively committed to reduce customer impact from these types of events.

The timing, of course, couldn't be worse - we're *officially* launching the chumby today, and I wake up to a forum full of posts about an outage, and a string of emails from IT discussing the issue.  They were up all night working with the hosting company to fix this.

But then, what else could we expect? Murphy always strikes at exactly these times.

Re: On Chumby Outages...

My alarm went off, Which confused, and later impressed me because even with the servers down I still got internet radio.  I don't know why I assumed it was being routed through you guys, but I just kind of did.

Re: On Chumby Outages...

asher wrote:

My alarm went off, Which confused, and later impressed me because even with the servers down I still got internet radio.  I don't know why I assumed it was being routed through you guys, but I just kind of did.

Well, some of it is, however, it's from a different system than the one that had the outage.  We don't host the streams themselves, but for some of these we either host or proxy the directories.  In general, we've tried to minimize these sorts of issues by spreading the work across different systems, but there are still some lingering failure modes, as we've all just seen.

Re: On Chumby Outages...

Fast reply, as always Duane. Thanks!

Re: On Chumby Outages...

Today was my first real use of Chumby as an alarm clock and it failed.

I set two alarms for this morning and none of them worked.  I think the reason why the alarm failed was because they both had an action of changing to a different channel and since the network was down it couldn't change the channel and the alarm didn't go off.  Luckily I set my old alarm clock as a backup.

Here's the scenario I had:
1)  Alarm set at night to put the the chumby to display the bedtime channel (this worked)
2)  Alarm set later at night to put the chumby in night mode (this worked)
3)  Alarm set for the morning at time A to steam music and change to the wake up channel (did not work)
4) Alarm set for morning at time B to "beep" and change channel to the wake up channel (did not work)

When I woke up this morning, it was my old alarm clocking buzzing and the Chumby was still in night mode.  I thought this was odd and saw that it said the alarm will go off at time B.  I waited for time B to come around and when it did, the alarm did not go off.  It stayed in night mode and presented the next time the alarm would activate.

I really think that since I had the actions set up to change the chumby channel on wakeup and the network was down, this caused my alarm not to go off.  Can you please look into this and resolve this issue?  If a network not found condition is found, it should be caught and a regular or "default" alarm should go off at the scheduled times. 

It's a good thing I had my old alarm clock as a backup, but I would like to get rid of this old clock as it is taking up valuable table space.

Re: On Chumby Outages...

Yup - these sorts of alarms are definitely problematic during this particular type of outage.  I'll discuss with Product Management about prioritizing better handling of this scenario.

Sorry about the problem - we obviously weren't at our best.  Hopefully we can regain your trust.

Re: On Chumby Outages...

I have my alarm set to one of the internet radio stations.  The screen to "turn off the alarm" or "snooze" came up, but there was no music playing.  Luckily, I woke up before the alarm went off.   Do the internet stations go down, when chumby goes down?

Re: On Chumby Outages...

They're not supposed to - what music source did you use (ie SHOUTcast, RFC, Mediafly, etc), and which selection/station?

13 (edited by kgeissler 2008-02-25 08:50:18)

Re: On Chumby Outages...

It was set to HardRadio.  Not sure if it is on Shoutcast or what.  Also, when the screen came up to shut the alarm off.  I turned it off, and the digital clock screen came up, but the time had stopped.   I didnt think anything of it.  When I went back to my bedroom, the alarm screen was back on the screen, but no music.   Again, I hit the option to turn it off.  The clock came back, but still stopped. 

I then rebooted my chumby, and it hung at "Checking Network Connection"....I then had to go to work, and just shut the whole chumby off.

Re: On Chumby Outages...

OK, thanks - we'll see if we can reproduce that behavior here.

Re: On Chumby Outages...

Thanks for the response Duane.  I'm sure your company will resolve these issues.

When I set the alarms for this morning, I'm not sure why I had both actions set to change to the "wake up" channel.  I guess I was thinking that it shouldn't matter.

I'm thinking the way around this is to have one alarm change the channel and the other alarm not have any action.  This way the second alarm should still go off when the network is down.

Re: On Chumby Outages...

Duane,
If my internet connection goes down, when the alarm is suppose to go off to streaming music, is there any type of backup, like a chime or something?  Or will the alarm simply not go off?

Re: On Chumby Outages...

Well, again, we have a plan for these failure modes that just hasn't been deployed yet.

Chumbys regularly check that they can contact chumby.com - however, since this failure mode was in the database backend, this contact would have succeeded - so the chumbys incorrectly believed the system was fully operational.

I'm not absolutely sure what the behavior is of every combination of alarm settings with a down local network. We have been compiling a matrix of all of the possibilities in order to cover everything for a future release - obviously, failover to an internal "chime" alarm is the best solution overall - assuming we can accurately know the status of the network.

Re: On Chumby Outages...

I had two major problems with today's outage:

1. I made several changes to my channel lineup around 1:30am (renamed a few, deleted a few, etc.)  When the service finally came back up, all of my changes were gone.  Did you guys have to revert to a backup?

2. My alarms did not go off (thank god I set a backup non-chumby alarm).  I'm also guessing that it was related to the network outage because one alarm was set to come out of night mode and the other was set to switch to a channel - a channel which now appears to have been erased.

Very disheartening.  Now I can't really use it as a trusted alarm source, which is a bummer.

Re: On Chumby Outages...

Yes, we had to revert to a backup from Sunday at 2 pm PST.  This was unavoidable, because the failure that occurred at the hosting service completely corrupted the database.

Again, we don't have the root cause analysis yet - once we do, we'll be able to reproduce it as part of QA's process to make sure chumbys can survive it in the future if it ever happens again, which hopefully won't be the case.

Re: On Chumby Outages...

creep wrote:

Today was my first real use of Chumby as an alarm clock and it failed.

I set two alarms for this morning and none of them worked.  I think the reason why the alarm failed was because they both had an action of changing to a different channel and since the network was down it couldn't change the channel and the alarm didn't go off.  Luckily I set my old alarm clock as a backup.

Here's the scenario I had:
1)  Alarm set at night to put the the chumby to display the bedtime channel (this worked)
2)  Alarm set later at night to put the chumby in night mode (this worked)
3)  Alarm set for the morning at time A to steam music and change to the wake up channel (did not work)
4) Alarm set for morning at time B to "beep" and change channel to the wake up channel (did not work)

When I woke up this morning, it was my old alarm clocking buzzing and the Chumby was still in night mode.  I thought this was odd and saw that it said the alarm will go off at time B.  I waited for time B to come around and when it did, the alarm did not go off.  It stayed in night mode and presented the next time the alarm would activate.

I really think that since I had the actions set up to change the chumby channel on wakeup and the network was down, this caused my alarm not to go off.  Can you please look into this and resolve this issue?  If a network not found condition is found, it should be caught and a regular or "default" alarm should go off at the scheduled times. 

It's a good thing I had my old alarm clock as a backup, but I would like to get rid of this old clock as it is taking up valuable table space.

Same exact issue, when i woke up the chumby was still in night mode even though a few alarms had already passed. nothing happened. I really feel like you should add a backup feature so that if the channel can not change or whatever happends the the chumby will buzz.

Re: On Chumby Outages...

I woke up to no streaming alarm clock radio and thought I overslept.  neutral  I was not happy to realize that it stopped working because Chumby.com went down.  sad  I hate the fact that so much functionality dependent on Chumby.com.  For something as simple as an alarm clock, which is the sole reason I bought it, it is the most unreliable alarm clock I've ever used.  If power blinks off, it doesn't come back online .  If Chumby.com's website goes down it stops working.  When Chumby Inc goes out of business, my Chumby is basically an expensive brick (at least then it won't be an ad covered brick).

The worst part is that all these problems were basically engineered into the product through conscience software decisions.  I would tag it "defectivebydesign" if I could.  I applaud you for your efforts in your slow improvement of your software, but I know I will be recommending against anybody buying a Chumby till these reliability issues are resolved.  I don't want to hear about adding additional servers, I want to hear that the product is no longer going to be reliant on the website or even the company being around to function.  mad

22 (edited by davidharpe 2008-02-25 11:45:02)

Re: On Chumby Outages...

"if it ever happens again, which hopefully won't be the case."

Of course it's gonna happen again.  It's the Internet.  If a failure doesn't happen at colo, it's gonna happen somewhere between.  The internet is not a 100% reliable infrastructure - wasn't designed as such, definitely doesn't act that way.

jhaluska makes a very good point.  As cool as all of the other stuff is, the product really needs to be bullet-proof as an alarm clock.  Having it crash and burn when it can't call home is like having an airplane crash when the radio goes out.

23 (edited by yaesumofo 2008-02-25 12:58:08)

Re: On Chumby Outages...

Deleted due to incomplete display of the post. It makes no sence if the entire post is not shown.
PLease fix this.
I take the time and use energy to post and to have the "system" delete 1/2 of the post makes it a waste of my time.
Yaesumofo

A Chumby in hand is worth 20 in the Wherehouse.

Re: On Chumby Outages...

We agree - as I said, reliability of the alarm clock independent of the status of the network is something we're actively working on.

As to abandoning the whole concept of the Chumby Network and the network delivery of widgets, I think that's a bit premature.

25 (edited by yaesumofo 2008-02-25 12:55:12)

Re: On Chumby Outages...

Again only a portion of my post is making it. I am deleting the entire post since it makes no sense without all of making it to the server.
Yaesumofo

A Chumby in hand is worth 20 in the Wherehouse.