Topic: What I've been doing lately.

Just wanted to give a little status update, since it would appear to anyone monitoring the forum that nothing seems to be going on.

For the past couple of weeks, since the switchover to the stub server, I've been mainly working on the main database - the one that keeps pretty much everything about users, channels, widgets, etc.

This database is quite large, and quite complicated - it took somewhat over a day just to import the dump file from the production server into a database on my local machine.

I then built a very simplified version of the production Rails application with all of the models, but none of the view and controller logic.  I then simplified the dependency relationships and eliminated many of the Rails filters to stop them from generating "change" notifications that would normally be issued to devices,as well as system audit records, ratings recalculations, etc.

Since then, I've been running various scripts to trim the database - remove the remnants of dash user accounts, dash-specific channels and widgets, unverified accounts (typically created by spammers), throw away various stale analytics, and various other things.

Some of these scripts take literally *days* to run.  Eliminating a user or widget record is an *extremely* expensive operation.  My computer's fan has really been getting a workout.

The idea is to reduce the database to actual accounts for actual users with actual widgets running on actual registered devices, which should cut it down by a significant fraction.

There is *at least* another week to go on that.  After that, there will be some schema changes to optimize certain tables and simplify the types of database requests needed to respond to devices.  Some these tables, even after the trimming, still have millions of rows, so, again, that will take days to process.

One question one could logically ask is - why wasn't this done when Chumby was around?  The answer is simple - we could not take the systems down for the days to weeks necessary for this type of cleanup.

Once this is done, I can start building the system back up and we can see where we are.

One caveat - there's a chance that some of these scripts might have been aggressive with regard to hunting down and destroying Sony accounts - if your first device was a dash, and you subsequently purchased a *real* chumby and registered it to the same account, there's a possibility that the account might have been destroyed.  If so, you would have to create a new account and re-register your device.  Hopefully, that will not affect too many people.

Re: What I've been doing lately.

I know my combee@techwood.org account was Chumby first, then added a Dash, then added more Chumby devices.  Should be a useful test point, although I'm pretty sure I saw the Dash disappear from my device list after the split happened.

Re: What I've been doing lately.

Hi Duane and thanks for the info.
Good to know whats going on in the background.
It would be nice to having an update from time to time.
Reorganising the database is definitely a major thread to having the system back in a better condition. Keep going!

Re: What I've been doing lately.

Duane,
Thanks for your effort
Me and my chumby are following up your work from Israel...
Dov

5 (edited by kathleen 2013-03-18 13:01:35)

Re: What I've been doing lately.

Thanks for the update -- can't wait until we're back up!

Well, I WILL wait <smile>

Re: What I've been doing lately.

unwiredben wrote:

I know my combee@techwood.org account was Chumby first, then added a Dash, then added more Chumby devices.  Should be a useful test point, although I'm pretty sure I saw the Dash disappear from my device list after the split happened.

I checked and your account survived.  I think the only ones that would have been affected were ones where the account was initially created for a dash, not where the dash was added to an existing account.

In retrospect, I probably should have checked for this situation better, but since the script ran for 77 hours as it was, I wasn't too keen on re-importing the database and re-running the job.  Hopefully the affected users will be understanding.

Re: What I've been doing lately.

Because many users (like me) are migrating to Chumby after the original server died, how will accounts work? If the only way to do this is through the forum, I can post my serial number now.

Re: What I've been doing lately.

nathanm wrote:

Because many users (like me) are migrating to Chumby after the original server died, how will accounts work? If the only way to do this is through the forum, I can post my serial number now.

What do you mean 'migrating to Chumby after the original server died'? The original server was Chumby.
Your old account info is on the database that Duane is sorting out now and it will eventually re-emerge on the new system (or at least that's how I understand the situation - I don't know a great deal about managing large databases, but I have faith that Duane has come up with a better system than asking 40 000 users to tell him their serial numbers so he can enter them in manually).

Re: What I've been doing lately.

What I think he means is that he picked up his device post-Chumby, and thus doesn't have an account, and never registered his device.

nathanm, to answer your original question, I imagine Duane will be coming up with a much more elegant device/account registration system once he has something more "complete" (above and beyond the stub server) in place.

Re: What I've been doing lately.

thanks for the update! Its much appreciated.

Re: What I've been doing lately.

What to do with Chumby device now

Re: What I've been doing lately.

Just let them run the way they are until you are informed that something has changed. It is useful as a great alarm clock and the very good music source. You can also add pictures and movies to the larger chumbys and have a digital picture frame with clock overlay.

Re: What I've been doing lately.

Hi Duane,

It seems you still believe in the users... ;-)

Looking forward to whatever the future may bring - great job as always!

14

Re: What I've been doing lately.

Thanks!!!

Re: What I've been doing lately.

Have you thought of starting a BOINC project for this? http://boinc.berkeley.edu/trac/wiki/ServerIntro
It seems to be relatively easy (Considering the great efforts you have gone through so far) to create a project and I would gladly keep my computer running overnight if it would contribute to the cause. I'm sure there are some people here that agree with me. If the computer you're running these scripts on is powerful enough, can you please do this? If you do feel like this, BOINC has a VM you can upload the file to. Just a stupid idea I had.

Re: What I've been doing lately.

Unfortunately, the types of things I'm doing are not conducive to distributed processing.

It's really classic MySQL stuff - delete records that meets these conditions, but first delete these other records that depend upon them, but first delete *these* records.  These are done as transactions so that an error can result in a rollback.  These things are naturally sequential.

The issue is that the graph of interdependent records can be quite deep, especially for user accounts (Did they rate or comment on widgets? Did they send widgets to anyone?  Were the sent widgets accepted?  Did they make any widgets?  Did anyone rate them? Were their widgets the only ones left in the channels of other users? etc etc), and some of the conditions are complex - but most important, there are *so many* of them.

Re: What I've been doing lately.

This is absolutely fascinating to hear some of the inner workings of the Chumby ecosystem.  I feel like a dog staring in to a Lava Lamp.  I don't understand what I am looking at but it's got my total attention.  smile

Thanks for the update, Duane.

Re: What I've been doing lately.

I just hope Duane gets his replacement service on before the community becomes a ghost town. A good example of a discontinued device's community going away is Planet Cybiko. A Cybiko is a small handheld created around the turn of the century. It pretty much was like the Chumby: a device with good ambitions but misunderstood by the general public. It was essentially a PDA for kids. It had no touch screen, but did have wireless connectivity with other Cybikos. It was also hackable, similar to the Chumby. For example, you could use the (WAP) internet wirelessly (By that I don't mean WiFi, but by some form of infrared or radio signal that worked for up to 500 feet) by connecting a spare Cybiko to the serial(!) port on the back of your computer and running a special program. The company folded shortly after its release and because of this, and the extreme discounts afterwards, the Cybiko became extremely popular. However, this bubble burst after several years. You can still find new Cybikos on eBay for about $20, as a result of this. What happened after that was that almost every website about the Cybiko is down with one exception: planetcybiko.net. However, the last post on the forum is called "Is Anyone Still Here?" and was last updated before 'Stubbs', my I3.5, was procured from eBay. Before that, the last time anyone posted was over a year ago. The thing with the Cybikos is, all of the apps are still perfectly usable without modification. This is not the case with Chumby (Although what there is is better than everything else on the market). However, I'm glad there's still hope for Stubbs and his fellow Chumbys to come back, unlike the Dash. There, everything seems to be breaking down. It seems like Sony doesn't care, and could shut down its servers at any moment. For example, http://community.sony.com/t5/Dash/Lots- … 51#U101351,http://community.sony.com/t5/Dash/Buh-B … 91#U102391, and http://community.sony.com/t5/Dash/SHOUT … 0387#M4414.

TL;DR:I hope a replacement service comes out before the community goes and leaves, but I'd much rather be in the situation I am in right now than the situation that Sony Dash owners are in.

19 (edited by bobsz 2013-03-25 14:37:55)

Re: What I've been doing lately.

@nathanm- things may seem bleak in these 3 weeks since you bought your first Chumby. Honestly, things have seemed a lot worst in the past. Before the stub server, it looked like there would be no server at all. Read some posts from about a year ago to see how dismal some folks felt then. Believe in the users.

Re: What I've been doing lately.

As I've mentioned, there's a lot of stuff going on behind the scenes - the database cleanup is nearly complete.  My machine was doing that pretty much exclusively running these scripts all weekend.  There's still one more left to go that needs to churn through a bit less than four million records (the number of widgets in all channels for all users).  My guess is that that script will take a few days to run.

Re: What I've been doing lately.

Bobsz- I understand what you're saying. I was just saying that I hoped that Chumby takes off, and that I didn't want to see it die, like so many other technologies I loved. (WebOS, the PowerPC architecture, and BeOS, for example) Believe me, the I3.5 is one of the best purchases I have ever made. I definitely am not complaining about it! For only $30, Stubbs outpreforms some $300 competitors (Here's looking at you, every stereo with an iPhone dock connector on it, as if that makes it an 'internet radio'). Plus, I love the fact that for some reason I don't get Pandora ads even with a free account, and there doesn't seem to be a 40 hour/month limit. I definitely knew how Chumby was a year ago (I used to be a lurker on these forums, and that message on the front page about how Chumby was focusing on software and cutting hardware development was as disheartening to me as it was to you (although I didn't have an account at the time) What I wanted to say was that I'm not complaining, I'm just glad we have what we do.

Duane-Thanks for the updates! However, I have a few more quick questions:
1. When the service launches, will just the Chumby widgets be up, or will all widgets be online?
2. How will you treat nonpaying customers? What I'd do is just not let them use the service, as with Pandora, a much bigger company with a larger user base, only 11% of users actually pay. In fact, they lost a million dollars last year. If you went on a free with donations basis, then you would not make enough money.
3. Did your RSS reader run off of Google Reader? If so, then Feedly, an alternative RSS service is cloning the API, but you may want to switch it over.
4. Do you still have possession of the Chumby YouTube account? If so, what are you planning to do with it?

As always, my humble gratitude for you doing this at all, as well as being extremely active and transparent with the community.

Re: What I've been doing lately.

1) My plan is to bring all of them back.  I'm sure it will be rough going for a little as I bring back up some of the content services that many of the widgets rely upon.  We'll also have to go through the third party widgets to see which are still working.

2) I have not determined how the system will be paid for long term.

3) No, the RSS reader went direct to each feed, so it should not be affected except for those folks that configured it to use Google Reader

4) I haven't looked into the status of that account.

Re: What I've been doing lately.

Any progress?

Re: What I've been doing lately.

sky123 wrote:

Any progress?

Between my paying job getting very busy, and a bad cold passed around my family a couple of times, and Spring Break for my son, things have been going somewhat slower lately.  I hope to pick up the pace again shortly.

Re: What I've been doing lately.

Duane wrote:

I hope to pick up the pace again shortly.

Is there anything we can do to help? I realise that the source code needs to be cleansed first, but there might be small tasks that we can help with. For example:

  • Write a script to perform a task

  • Implement an API (it should be possible to release the API)

  • Perform some research to answer questions

I realise it takes time even to ask for help, but there must be a few of us who could help a little just because we want to see our chumby working properly again.