Topic: End of Chumby as we know it...

As most of you know, Chumby Industries went out of business just about a year ago.  The current service has been maintained by a handful of volunteers, including myself, and the service itself has been paid for by various entities - the Chumby estate, then the assignee for the assets, and finally a company that does not wish to be identified.

Unfortunately, that will be coming to a halt soon.  As of the end of this coming February, the current service will no longer be paid for.

So - what does this mean?  Well, if nothing happens, all of the devices go dark, and all of the various services, including this forum the wiki, the widgets, the Control Panel, source code, development tools, etc., all disappears.

I really don't want that to happen.

I am currently in the process of creating a new company to keep at least some form of the service running, so that we're not all left with bricks.  Since I would be funding this out of my own pocket, that means some sacrifice, at least initially, of some of the features currently enjoyed by Chumby users.

What's happening at the moment is that I'm working to clear and acquire title to the assets necessary to run the service - that means at least some of the Chumby trademarks, the domains, the copyrights to all of the servers, control panels, widgets, and other miscellaneous stuff.  We should know shortly if that will be successful.

I am also working with some ex-Chumbians to back up everything for possible restoration in the future.

From a service standpoint, my priorities are as follows, in order:

1) Provide enough service for the devices to boot, download a Control Panel, and play a single fixed widget (probably a clock).  This will allow alarms to work.  I will probably have to rev at least some of the Control Panels for this to work properly.

2) Post the source code required to be posted under the terms of the GPL and other licenses for the software incorporated in the devices - essentially the same as the current http://files.chumby.com/source - the toolchains, and other development tools.

3) Find someone in the community to host this forum and the wiki.  If you can do this, please contact me.

4) Support as many of the current music sources as possible.

5) Clean up and post the source code to some of the previously non-public software - mainly Control Panels.  Note that this will require some scrubbing to remove code covered by Chumby's content contracts, as well as third-party trademarks.

I will be dropping support for certain clients - "virtual chumby", "chumby" and "chumby Lite" Android clients, NeTV (to the extent that they're still using Chumby services) and probably all of the TV sets, at least for now.

You'll notice that I'm not currently thinking about supporting widgets - that's the largest ongoing cost to running the systems.  If there's a way to deliver widgets in some form without incurring large bandwidth bills, I will do so, but it's not my highest priority.

I will be putting up some method of donating to keep the ecosystem running - I would certainly appreciate any donations.

I wil post more about the current structure of Chumby's systems and economics.

Re: End of Chumby as we know it...

Of course, if anyone wants to pay the monthly bills for the current system, that works too.

Keeping the current system running costs between $4300 and $5500/month depending upon load.

Here's the breakdown for December 2012:

Amazon CloudFront $18.02
   US $17.15
   Europe $0.76
   Asia (Tokyo) $0.05
   Asia (Singapore) $0.02
   South America $0.02
   Australia $0.02
Amazon Elastic Computer Cloud $2260.10
  US East (Northern VA) $2100.31
    EC2 Linux
      Standard Small 744 hours $48.36
      Standard Large 2232 hours $580.32
      Standard Extra Large 2232 hours $1160.64
      Micro 2976 hours, 59.52
    EC2 EBS
      969 GB-Mo $96.90
      25,554,572 I/Os $2.56
      snapshost 94.857 GB-Mo $9.01
    Elastic IP
      1485 hrs $7.43
    Elastic Load Balancing
      685.667 GB $5.49
      5203 hours $130.08
  US West (Northern CA) $68.21
    EC2 EBS
      245 GB-Mo $26.95
      snapshot 17.290 GB-Mo $1.82
    Elastic IP
      3719 hours $18.60
    Elastic Load Balancing
      744 hours $20.83
  Asia Pacific (Singapore) $91.53
    EC2 EBS
      523 GB-Mo $57.53
      snapshot 319.222 GB-Mo $30.33
    Elastic IP
     743 hours $3.72
Simple Storage Service $535.93
   US Standard Region $524.89
     first 1024 GB-Mo $97.28
     next 4012.338 GB-Mo $320.99
     2473612 PUT COPY POST LIST requests $24.74
     81887553 GET request $81.88
   Asia Pacific (Singapore) Region $11.04
     116.105 GB-Mo requests $11.03
     3 GET requests $0.01
Amazon RDS $991.66
   US East (Northern VA)
     RDS storage
       provisioned storage multi-AZ 55.000 GB-Mo $11.00
       270577727 I/Os $27.06
       provisioned storage 50.000GB-Mo $5.00
    RDS for MySQL
      RDS large 744 hours $271.56
      RDS large multi-AZ 744 hours $543.12
      RDS small 744 hours $133.92
AWS Data Transfer
    US East (Northern VA) $637.57
      data transfer out 5105.833 GB $612.71
      regional data transfer 2286.435 GB $22.86

Total $4441.28

3 (edited by kgeissler 2013-01-10 19:03:32)

Re: End of Chumby as we know it...

Thank you Duane for trying to save our Chumby's.  I would be willing to donate, just tell me where.  I never knew it cost so much to keep our Chumby's running.

Why does it cost so much to keep the widgets going?  If the offline firmware that Zurk posted in the forum can do widgets, is it being done illegally or something?  Because that doesn't cost anything to run the widgets with that firmware.  Maybe I am missing something.

Re: End of Chumby as we know it...

Well, Zurk's firmware relies on you getting the widgets from *somewhere* initially and storing it on a dongle, and it costs something to host and serve all of those widgets.  They're actually stored in three places - when you upload a widget, it's stored as a blob in the main database (one of the large RDS instances), then duplicated into the database slave (the other large RDS instance).  There's a script that runs periodically that then copies the widget to S3, where it's cheaper and faster to serve them.

Back when Chumby was operating, it actually cost somewhere north of $25K/mo to run the service - we were required to run it on actual hardware in an actual datacenter to comply with rather strict data security requirements of one of our partners.  Once we severed that relationship, we moved to Amazon and I modified the Control Panels to be much more aggressive about caching, which took us down below $6K/mo.  Amazon has been steadily reducing prices so we're now where we are.

There's a lot of data stored - I think even if we suspend all of the EC2 and RDS instances, the storage of data alone runs about $900/month.

Some of that data include the entire subversion history of the company - many things could be eliminated or taken offline.

There are currently just around 100,000 devices registered in the system since the Sony dashes were moved out.  Of those, about 40,000 are in active use.

Re: End of Chumby as we know it...

So does that mean when Chumby goes dark at the end of next month, Zurk's firmware and widgets will stop working also?

Re: End of Chumby as we know it...

No, Zurk's firmware comes with a selection of widgets and serves them to the control panel off the Flash drive on which you install his distribution. You can modify the channels he provides and add new widgets to them (even add new channels if you like) as long as you have a copy of all the files needed to run each widget. The problem is that such editing is not particularly easy; it involves modifying an XML file, and it's easy to make a mistake in doing that kind of thing. I don't think it would be too hard to write some kind of Perl script to do most of the editing offline, but from what Duane says it's likely to be pretty costly to keep the existing web environment.

Duane, what do you think of the idea of trying to at least keep the current widget files available for manual download, to use with something like Zurk's firmware? Feasible? I would think only someone who owns the Chumby rights could do that legally though.

I love my Chumby alarm clock, but almost $5,000 a month is some serious cash to find, I wish you luck!

Re: End of Chumby as we know it...

Can't thank you enough Duane for all your sacrifice over the year(s).!!!!
This is sad news, but a ususaly, thanks for your candor and info.

I had no idea it cost that much to host per month, that is a real eye opener and puts things into perspective for me about Chumby and other companies too. I would like to help where I can, let me digest this info over the weekend and ping you. I was going to suggest we all pitch in per month and help pay, but at $10 a person (I would guess less from most) that would take a lot of people to hit the needed $5k per month. Does this also mean that BestBuy TV's that use Chumby and other stuff will die too?
It's a shame when the Chumby and the Chumby idea is so awesome, even as just a Facebook conduit and picture viewer. Then there is all the other stuff it can do, Pandora and alarm clock. I don't get it, it should have taken off IMHO.
Thanks again for all you have done, Matt Burkhard

Re: End of Chumby as we know it...

I too am willing to donate. I am not rich but I would like to help. $5,000 a month is a tough nut to crack. Even if we get enough people to donate this month, will it keep up month after month?

I wish I could be of more help (coding / financial). Duane has done a magnificent job and we are all in his debt. If this forum goes dark is there someplace else we can send/receive messages concerning all things Chumby?

Tar, feathers, congress. Some assembly required.

Re: End of Chumby as we know it...

Can we create a Google Group?

https://groups.google.com/forum/?fromgroups#!overview

Re: End of Chumby as we know it...

If I build a replacement system, it will not cost $5K/month.  My guess it will be in the low hundreds - my goal is under $200/month.

What you're seeing with the current system is a "professional" system, with full analytics, the complete build system, monitoring, backups, the whole source control history back to day one, system logs, internal files and documents, etc even though in some cases the servers that server this stuff have been shut down.

At $4400/month, with 40K active units in the field, it's actually only $0.11 per unit per month, which is pretty low.

Some of the costs are independent of the number of units - the servers have an hourly cost, and the storage is used whether or not anyone actually fetches the data.  The variable costs are the amount of data incoming and outgoing on the system, which *is* dependent upon the number of units.

If you look at the costs posted above, the variable costs are toward the bottom (the data transfer), the I/O to'from the RDS databases, and the GET requests from S3.  The rest are relatively fixed costs - just the cost of having a reliable system with a lot of data in it.

As far as donations are concerned, at the moment, my guess is that the vast majority of users have absolutely no idea that Chumby is even gone.  I get several customer support emails a day, and all but a handful clearly think that there's still a company here.  Most users have chumbys that have been cranking along completely normally.

I think it's a tribute to the guys who wrote the servers and the ops guys that set them up.  The vast majority of outages that have happened over the last year were not the servers themselves, but problems at Amazon.  I think only one outage was actually our fault: the main analytics system was shut down at one point, but the main server was still collecting the data - without the analytics system to pull that data out and delete it, it was accumulating and eventually exceeded the allocated storage of the database.  The system was changed to throw the data away and everything went back to normal.

I have no idea how customers will react when the change happens. It may be that some significant fraction of the 40K active devices are shut down and we never see them again, which would be a shame.

Note that I am not ruling out supporting widgets ever again. It's just not the highest priority.  I have something like six weeks to get the new systems in place to keep the ecosystem from dying completely.  After that we can discuss ways to restore the widgets.

We had a design at one point where the widgets were shared peer-to-peer.  A widget publisher would put the widgets on a USB dongle in their device, with a little manifest file, and using a Napster-like system, the widgets would be made available and passed around between the devices, and stored locally.  We had figured out a way to allow users to "depublish" a widget, as well as make updates with some level of confidence that all of the peers would get the updates.  Maybe that can be resurrected it would require "Classic" owners to always have a dongle in the back - there's insufficient storage in a Classic to do this.

Re: End of Chumby as we know it...

As far as Best Buy/Insignia are concerned, they know what's going on and I'm talking with them about what to do about Infocast and Smart TV devices.  Technically, if you have one of those devices, you are *their* customer, not Chumby's.  My current intent is to support the Infocast devices to the extent it's practical - but Chumby devices come first.  My guess is that the I8 may suffer a loss of Photobucket functionality.

Re: End of Chumby as we know it...

songmaster wrote:

No, Zurk's firmware comes with a selection of widgets and serves them to the control panel off the Flash drive on which you install his distribution. You can modify the channels he provides and add new widgets to them (even add new channels if you like) as long as you have a copy of all the files needed to run each widget. The problem is that such editing is not particularly easy; it involves modifying an XML file, and it's easy to make a mistake in doing that kind of thing. I don't think it would be too hard to write some kind of Perl script to do most of the editing offline, but from what Duane says it's likely to be pretty costly to keep the existing web environment.

Duane, what do you think of the idea of trying to at least keep the current widget files available for manual download, to use with something like Zurk's firmware? Feasible? I would think only someone who owns the Chumby rights could do that legally though.

I love my Chumby alarm clock, but almost $5,000 a month is some serious cash to find, I wish you luck!

Maybe the solution is as has been said.   Build an easier config system for Zurk's code and a temporary 'migration' server to let users download the widgets

So you'd boot the zurk stuff.  A 'control panel' would let you ask for specific widgets, it would download them, and configure them locally.  One time, no frills setup.  You want to add/change widgets, run the migrator again from scratch.

What do you think?

Re: End of Chumby as we know it...

$200 per month for a new server system? I am confident you can get that and more from faithful Chum owners. Maybe use any overage to put towards upgrades down the line. Music and clocks and maybe a YouStreams or My Streams would be primo. I use my chumbys mostly as an alarm clock and music source. The rest is fluff to me. My kids love to play the games but they can use there tablets. Picture frame type use would be great to come into play. GO for it Duane. You are the man.

Owner of 3 Sony Dash, 2 Info 8.

Re: End of Chumby as we know it...

philhu wrote:

Maybe the solution is as has been said.   Build an easier config system for Zurk's code and a temporary 'migration' server to let users download the widgets

So you'd boot the zurk stuff.  A 'control panel' would let you ask for specific widgets, it would download them, and configure them locally.  One time, no frills setup.  You want to add/change widgets, run the migrator again from scratch.

What do you think?

Well, it's not quite that simple - many of the widgets have several different versions depending upon what device they're for - in some cases as many as four or five (3.5", 8", Android, TV, etc).  The current system takes care of managing all of the various versions.

The system currently holds about 11,000 widgets, most of which are not public.  The private widgets are most typically development versions, but some people have widgets playing on their devices that they made entirely for themselves.

We also must respect the rights of the widget authors - they must have the right to update or remove their widgets when they no longer wish to support them.  Many of the widgets hit servers - without the ability to remove or update, the author may be stuck without the ability to change or terminate their service.

There's also the issue of configuration - the custom configurations for the widgets (things like the zipcode for the weather widgets) is stored on the server - we'd need to still have a way to configure and serve this customization data.

To support all this, I would need to restore a significant fraction of the current database - which brings us back to where we started.

Re: End of Chumby as we know it...

I'd be happy with the chumby turning into an alarm clock with weather.  I currently only run lcars clock, weather and stocks, clock for 15 min, weather and stocks for 1.

That has always worked for me

Saying that, if we could get it down to 200-300 a month, divide by the active users (the ones who know chumby died, not the ones just using the service), a 'sub' would be $40/year or less, correct?

Re: End of Chumby as we know it...

First of all, let me also say "Thanks" for Duane keeping things alive for the last year.

I have one suggestion for providing access for widgets without running up a large bandwidth bill, but this suggestion is more suitable for making Zurk's offline stuff work well, rather than "stock" chumbys.

The suggestion is to package the current set of public widgets with bittorrent; that way widgets can be made available without having to have a central repository.

Alternately: they could be pushed to github (or a similar service) which are happy to host open source files.  That way you can provide access to them without having to host them yourself.

Re: End of Chumby as we know it...

I'm interesting in donating to keep Chumby alive, but I also recognize that its a tough thing to keep going in its current setup.  I'd love to see the rest of the Chumby-built widgets open sources along with the control panel.  However, I know components like FlashLite can't really be distributed separate from the licensed platform, so the life of the current widget pool will continue to get shorter.

It would be really awesome to see the ability to use some other platform for syncing of my widgets/user settings.  A version of the offline firmware that worked with a Dropbox folder to load widgets and store settings, for example.  Alas, most services like DropBox have gotten harder to support in embedded systems with the rise of OAuth, as they require a working browser in order to authenticate users.  I guess you could use a website to do that with some pairing code.

Re: End of Chumby as we know it...

I am keen to see Chumby keep going and I would like to thank Duane for his efforts. I'd contribute or subscribe to a Chumby service, if that's what was needed to keep my Chumbies on.

Re: End of Chumby as we know it...

Duane wrote:

As far as donations are concerned, at the moment, my guess is that the vast majority of users have absolutely no idea that Chumby is even gone.  I get several customer support emails a day, and all but a handful clearly think that there's still a company here.  Most users have chumbys that have been cranking along completely normally.

I am sure that this is true. I wasn't at the forum for over a year and totally missed the company problems till 2 weeks ago. I am glad that zurk took over my offline hack and developed it further. Thanks zurk for that!

For the future, it would be great if the controlpanel would look for a folder /mnt/usb/widgets or something and add all swf files in there as a channel. This would make it easier to add downloaded widgets. For widgets that need parameters we could add a simple ini-file with the same name as the swf file. This would allow even untrained people to add widgets.

And perhaps it would be a good idea to add an info widget to all current channels to inform all users about the problems so that everyone can prepare.

Thanks Duane for your efforts!

Re: End of Chumby as we know it...

There's already an existing hack for the 3.5 devices to add local widgets to a channel.  It would be fairly easy to add that to the Control Panels for the 8" devices.

Re: End of Chumby as we know it...

I showed up rather late to the party *glances sheepishly at his reg date* and around these parts I've generally just lurked, but I've fallen quite in love with my Chumbies (Chumbys? Chumbae? Chumbii?) despite knowing the state of the company when I finally picked up my first one. Aside from the obvious difficulties in widget management, the offline firmware works admirably well (thank you zurk & Christian!), but I'd hate to see the loss of *.chumby.com, the servers, and everything else that goes with it. While I don't have a lot of discretionary income, I'd certainly be willing to donate what I can - or pay for a subscription - if it helps to keep things online.

I've also got a rather underutilized hosting account (that I should probably remember to renew), with the usual support for docu/mediawiki, phpbb, and suchlike. It's only shared hosting & I imagine someone here has a better arrangement available, but I'm not doing much more right now than hanging a domain with a few mail aliases & running a forgotten CGI or two, so I figured I'd throw the offer out. For what it's worth, I know my way around a linux box, and (once upon a time) I used to be a fairly competent scripter & coder. if I can just find my brain under all these cobwebs and dust, I'd be happy to lend a hand with any random grunt work/testing/troubleshooting on the software side of things.

And I think I've said it before, but thank you again, Duane. The way (and the fact) that you've kept everything running is fantastic.


-Bats

Re: End of Chumby as we know it...

demarks51 wrote:

$200 per month for a new server system? I am confident you can get that and more from faithful Chum owners. Maybe use any overage to put towards upgrades down the line. Music and clocks and maybe a YouStreams or My Streams would be primo. I use my chumbys mostly as an alarm clock and music source. The rest is fluff to me. My kids love to play the games but they can use there tablets. Picture frame type use would be great to come into play. GO for it Duane. You are the man.

I agree completely! I have a bunch of widgets on two Chumbys, but I can survive with just a handful -- alarm clock, music (and maybe weather and RSS if feasible). If it all worked for a couple of hundred a month, I'd gladly sign up for a monthly or annual "subscription"! As you say, there are surely enough of the faithful among us to cover the cost.

23 (edited by Christian 2013-01-13 13:36:00)

Re: End of Chumby as we know it...

Duane wrote:

There's already an existing hack for the 3.5 devices to add local widgets to a channel.

Yes, I know that hack, and I have no problem editing an xml file, but I think there are many chumby users out there who do have a problem with it. It's just a suggestion to make things easier for those people.

Re: End of Chumby as we know it...

I do think having such a thing rolled into the "final" release of the control panel would be a good idea...

Is there any way to "broadcast" an announcement to all connected Chumbies?  This might be a good idea; if not, perhaps build something into the next control panel, so you can send a message to all Chumby users @ startup to ask them visit an announcement page (which you'd presumably set up ahead of time), letting them know that the way their Chumby works will be changing in the near future (and that Android/etc will lose support), and what they can do to continue to get the most out of it.

Personally, I'd be okay with a subscription model for the New Chumby, if it can be kept cheap smile

Re: End of Chumby as we know it...

I also want to join in the chorus thanking Duane for his tireless efforts to keep my Chumbys going both during the time when Chumby Industries was in business, and especially over the last year. Duane, your work on behalf of this community is REALLY appreciated.

Of course, if something can be worked out to offer continued support for this, I would be delighted to be a part of it. In the meantime, I will watch the forums a little more diligently to see what the future will bring.