Re: End of Chumby as we know it...

Thanks Duane for all the work! You are a hero!

1) Please keep us in the loop via Facebook!! I don't come to the forum and really would have no idea what's going on until I saw the posting on FB!

2) Please don't be shy to ask for financial donations -- if I read it correctly, you are under some time pressure to get things done. If money can buy time, do let us now!

Re: End of Chumby as we know it...

Frankly, I'd rather not raise money to keep the current system running - after all, 5K for one month of the current system could probably fund the new one for two years.  It seems better to put that type of money toward restoring services on the new system.

Re: End of Chumby as we know it...

Duane wrote:

You can see the current server structure from the second post:

4 Amazon EC2 "micro"
1 Amazon EC2 "small"
3 Amazon EC2 "large"
3 Amazon EC2 "x-large"

2 Amazon RDS "large"
1 Amazon RDS "small"

Storage: 4TB
Outgoing bandwidth - 5.1TB/month

Thanks for the info, actually the number of instance wasn't clear to me from the post on the first page.

So, majority of the expense is on the CPU cycles (about $2000). My understanding is that one EC2 XL instance is more-or-less similar to a quad-core server w/16GB RAM.

Can any of these EC2 instances be shut down or reduced? Say, if somebody commits to support only original Chumby devices?

Re: End of Chumby as we know it...

The two RDS larges are the main database and a replication.  Those two systems are run by the two x-large EC2 systems.

Two of the "large" EC2 systems are the user-facing web services - the web site and data services to the devices.  The other "large" EC2 is the content system supporting data services for various widgets and music sources.

The "small" EC2 and the "small" RDS is the forum and wiki.

The "micros" are the source control system, and some other supporting services for widgets and music.

There are actually a dozen or so other EC2 instances that have been hibernated that were in use when the company was in business, mostly user-facing web servers and analytics.  We shut them down as we refactored the caches in the clients, at the expense of server-side user changes being propagated quickly to the devices.

I suspect, but don't know for sure, the the vast bulk of the devices are Chumby-branded, followed by Infocast 3.5, Chumby 8, then Infocast 8, with a spattering of Android and TVs.  The Android devices and TVs really aren't material since they only run the chumby software a small fraction of the time, while the other devices are 24/7.

Re: End of Chumby as we know it...

Duane wrote:

The two RDS larges are the main database and a replication.  Those two systems are run by the two x-large EC2 systems.

Two of the "large" EC2 systems are the user-facing web services - the web site and data services to the devices.  The other "large" EC2 is the content system supporting data services for various widgets and music sources.

The "small" EC2 and the "small" RDS is the forum and wiki.

The "micros" are the source control system, and some other supporting services for widgets and music.

There are actually a dozen or so other EC2 instances that have been hibernated that were in use when the company was in business, mostly user-facing web servers and analytics.  We shut them down as we refactored the caches in the clients, at the expense of server-side user changes being propagated quickly to the devices.

I see. Sorry to bombard you with questions, just trying to understand what would be the minimal setup if those were moved to physical servers ...

What is the database behind? MySQL? What is the size of DB itself ?

Re: End of Chumby as we know it...

The RDS instances are MySQL, about 50GB each.

We're going to be removing cruft tomorrow (unused accounts, etc) so we'll see how big they end up when we're done.

Re: End of Chumby as we know it...

roryhawke wrote:

Appreciate the continuing efforts Duane!  I originally came to Chumby as a lower cost alternative to Internet/WiFi radios and have never regretted my choice. I'm glad to hear that there may be continuing shoutcast support.

Sing it loud! It's STILL one of the best values for internet radio around. That, combined with the flexibility of alarms makes it a top value even if there were no other widgets at all.

Re: End of Chumby as we know it...

Is it too late to start a kickstarter project or two?

Re: End of Chumby as we know it...

Yes, I hadn't thought of it, but a Kickstarter project might be just what you'd need to get the funding to get everything off the ground.  Only trouble might be figuring out higher-level perks (obviously, low-level perks would include things like Chumby stickers/shirts/etc)... but it'd be a good way to raise a lot of money fast, and rekindle interest in the Chumby platform.

Re: End of Chumby as we know it...

To be honest I would prefer a solution which would require as little ongoing dependency on Duane (and funds and servers) as possible... ie an offline (USB stick?) solution.

He has done such a great job to this point, and I think it would be unfair for him to keep devoting time and effort to Chumby, when he could be moving onto bigger and better things.

Re: End of Chumby as we know it...

Well, I *do* have a real job - this is evening and weekend stuff.

I have absolutely no issue with people coming up with solutions that do not require these servers or depend upon me.  I'm assuming that might take some time to put together, and I just think we need to bridge that.  Zurk made a nice start, but it's got some way to go to be ready for the average user.

Also, the important assets need to be liberated from the shell of Chumby by *somebody*, to make them accessible to the community.

I do have a bit of concern that I put together something "good enough" and folks get complacent.

Re: End of Chumby as we know it...

Duane wrote:
aris wrote:

I would have thought that 40,000 devices could be handled by a rented host in a data centre.  As you say, a couple of hundred dollars a month.

The current system has about 4TB of stored data and serves about 5TB of data per month to the current active 40K devices.  You can see the finances above.

As I said, I'm just trying to hold off the darkness - we'll revisit the rest once we've survived the Chumpocalyse.

Can you break down where/how the bandwidth is used to a device?   Does everything that goes to a chumby get proxied through chumby servers?

Re: End of Chumby as we know it...

I suspect that the majority of Chumby users are unaware of what is going on.  I wonder if it is possible for you to push a status message to active Chumby screens across the world to update them on the current status.

Re: End of Chumby as we know it...

Hi guys

What about the wiki and the forum? There are still people out there hacking their Chumbys, and for those these resources are even more important than a working widget system :-) Still I'd really love to see a "minimal" system which avoids turning thousands of chumbys into bricks, the device is still insanely cool! I guess clock, radio, picture frame, rss feeds would do it.

Re: End of Chumby as we know it...

If the scaled down, basic software is implemented, please try to include an "image URL" widget.  It is very powerful for viewing webcams, weather maps, etc.

Re: End of Chumby as we know it...

The idea of a P2P based solutions sounds like the ideal, although that sounds like a major architectural change that I can't imagine happening in the near future.

How about a half-way house though? What about asking the community to host Chumby mirrors?

Duane (or some other volunteer) continues to host the central database which holds the user accounts and widget configurations - hopefully the bandwidth required for this would be fairly minimal, so maybe this could be run on a physical machine located in some well connected office (or similar) rather than clocking up co-location costs. Another box (or maybe Amazon S3) holds the master widget repository. The volunteer mirrors periodically rsync to this box, but no chumby's ever contact this box directly.

Volunteers then run mirrors of the repository (much as Debian mirrors are hosted), which is what the Chumby's load their widgets from - a simple round robin DNS setup distributes the load and allows Duane to add / remove mirrors as needed.

I personally would be happy to donate 100-200GB of bandwidth per month on one of my servers. If 50 other people were willing to donate a similar amount of bandwidth you have your 5TB.

Widget authors still maintain the same level as control as before - if their widget is removed from the central DB then the Chumbys won't attempt to fetch it from the mirrors, whilst the bulk of the bandwidth costs are removed.

Re: End of Chumby as we know it...

Duane, thanks so much for everything. I think your plans are good. You just need to set up a Kickstarter, and you'd have funding in no time!

Re: End of Chumby as we know it...

aris wrote:

Can you break down where/how the bandwidth is used to a device?   Does everything that goes to a chumby get proxied through chumby servers?

No - most of the data loaded into a widget (like photos, video, most data) comes directly from the third party service.

Chumby *does* provide a couple of services that are used to implement some widgets.  For instance, we do proxy the access to data *about* SHOUTcast streams, since you require an developer-assigned API key to get that data.  We take the request from the device, add the key, and forward the request to SHOUTcast , then pass the response back to the device.  the audio stream itself comes directly from the stream provider.  In this, and a few other services, we're required to keep the API key secret.  When the device makes a request to us, we verify that the device is legitimate to prevent others from leeching on the service - again, this is generally required by the contract with the third party.

There are similar systems in place for iheartradio, Pandora, Photobucket, eBay, Flickr, Instagram, and a few others.  Again, this is mainly for authorization and metadata - the big data comes directly form the provider.  In a few of them, we also enforce regional restrictions, per the contract.

Re: End of Chumby as we know it...

pythag wrote:

The idea of a P2P based solutions sounds like the ideal, although that sounds like a major architectural change that I can't imagine happening in the near future.

How about a half-way house though? What about asking the community to host Chumby mirrors?

[... detailed description ...]

Widget authors still maintain the same level as control as before - if their widget is removed from the central DB then the Chumbys won't attempt to fetch it from the mirrors, whilst the bulk of the bandwidth costs are removed.

I think this sounds like a fantastic idea, and I'd be happy to join the pool if we can get a few others to join, to ensure that my systems aren't going to be handling like 1/4 the workload.  With the round-robin DNS, that would eliminate any need for client-side changes, and the "master server" (which would be what is currently the "only" server(s)) wouldn't even need any major changes, other than allowing the rsync.

Re: End of Chumby as we know it...

I'm fine with mirrors.

There is one other little rub with hosting widgets - because of Flash's crossdomain security, widgets need to be loaded from a "chumby.com" subdomain.  That way, third party services that have only provided crossdomain files opening up the chumby.com domain will still work.

There *might* be a Control Panel hack to get around this (I need to check), but what it might mean is that we'd have to assign a chumby.com subdomain to each of the mirrors in the pool, and each pool would have to handle that (particularly in the case of virtual hosts).

Re: End of Chumby as we know it...

File this under "brainstorming":

I hacked my Chumby to point at a Squid proxy server on my network. Not only did bandwidth usage go down dramatically, but my widgets loaded much more quickly. If I "owned" the project and had all the technical resources needed to pull it off, I'd consider changing the codebase to support two separate data models:

1) Normal Chumbies, as they sit, get the "minimal" control panel with a single alarm clock widget.
2) Chumbies with a USB flash drive plugged into them get the "full" control panel with all the widgets that they can keep 100% cached locally.

Then people who wanted to keep the "full" experience could plug in a flash drive, automatically download the widgets they're subscribed to, then operate in "offline" mode as far as the central servers are concerned. After the initial download, they'd only impose a minimal amount of traffic and load on the AWS nodes.

Re: End of Chumby as we know it...

Duane wrote:

I *believe* that if you ssh into a working chumby, and copy /tmp/controlpanel.swf to a USB dongle, it should use that one instead of downloading a new one upon reboot.  However, that also means that if I change the Control Panel on the server you won't get a new one - so I wouldn't do that just yet, since I may have to put something out for this new system to work properly.

You believe correctly. That's the "system" I've been using to make my chumby standalone. I reboot in "normal" mode every so often to see if there is a CP update, but I'm happy running my local widgets & CP the way they are now.

(Thanks for all your efforts! Much appreciated from here.)

Re: End of Chumby as we know it...

Can the cross-domain hosting problem be handled by having the widget host put up a crossdomain.xml file allowing the access?   I know that was needed to allow data access, but I don't remember what policy was used for "including" another SWF in the active one.

Re: End of Chumby as we know it...

Need to get out of Amazon and into a colo. 

Amazon should host Chumby for free.

Why have (VPC's?) in other regions?  waste of money.

Are you accepting any hardware donations?

Re: End of Chumby as we know it...

Duane wrote:

I'm fine with mirrors.

There is one other little rub with hosting widgets - because of Flash's crossdomain security, widgets need to be loaded from a "chumby.com" subdomain.  That way, third party services that have only provided crossdomain files opening up the chumby.com domain will still work.

There *might* be a Control Panel hack to get around this (I need to check), but what it might mean is that we'd have to assign a chumby.com subdomain to each of the mirrors in the pool, and each pool would have to handle that (particularly in the case of virtual hosts).

TBH  I  don't see much of a  problem with assigning subdomains to the mirrors (e.g. mir001.chumby.com);  anyone with the capacity to host a  mirror should be running a VPS at the very least so the virtual host setup needed shouldn't be an issue.