26 (edited by chumbyfly 2012-04-08 07:13:32)

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Zurk,

     Thanks again for helping me locating the profile and show files for chumby8.
Can you please provide the path and name of the script I need to run after I modify both the profile and show files? I cannot find any script in that same directory but I can only find a shell script named startup.sh two directories above it, but the contents does not look like it has anything to do with the profile. Thanks in advance.

CF

zurk wrote:

yup. it will show ZZ as the location regardless. no way to change that without help from the original author which is unlikely. It will however match the forecast and temp to your local area based on what you have placed in zwapi.sh.
For the widgets you are probably editing the chumby one profile xml. Both the chumby1 and chumby8 firmwares are built from the same common source.
The one you need to edit for the chumby8 is located in /mnt/usb/psp/lighty/html/xapi/dev/show. There is also a shell script you need to run in that same directory after you edit and save the profile. just edit the profile, save, run the script and reboot the chumby.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Zurk:

Great work!

I have my own swf(never upload to chumby b4), how do I add it to the offline chanel?

TIA!

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

cf : wrong directory sorry....i just checked. look in  /mnt/usb/lighty/html/xapis/profile/, modify show and run link.sh...that will do it.
lgxb : look at  /mnt/usb/lighty/html/xapis/profile/show with a text editor where you can see the Weather Underground RapidFire widget which is not available on chumby.com. Replace the name with your own widget, copy your widget over to the same directory where the weather underground widget is located and  it should work. you need to run link.sh as noted above. You can also add another widget if you require by copying and pasting the chunk of XML which controls the weather underground widget.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Dear Zurk,

I have a Chumby Classic, a Dash, and a Chumby 8.  Virtually all of the technical aspects of what's coming with the Chumby servers are far beyond my comprehension.  That said, you've made it so simple, I feel confident even I could download your software and install it on the Chumby 8.  But then I have no idea what comes next.  How do I install the roughly 30 widgets, or is that even possible?

Your volunteer work on behalf of the Cnumby community is amazing.  Thank  you!

I don't want to impose by asking experts to waste their time explaining to  me what's coming and how to prepare for it, but I'm hoping someone can point me to a post or a url that explains it for a neophyte.

Many thanks!

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

its fairly easy to get 30 or even 100 widgets in your own version of the offline firmware. you need 2 files. the first is your profile file which is an xml file. then you need to download all files referenced by your profile from chumby servers using a tool called wget. the second is the offline firmware profile located in your usb disk once you have downloaded it. both profiles need to match so you have to modify the second to match the first. once you have done that its over and the chumby will play your widgets.
learning about xml and linux is your best starting point. see here: http://www.ibm.com/developerworks/linux … -v3-103-1/ and http://www.xmlfiles.com/xml/
i suggest you first download the c8 offline fw and get used to it though. it has a few very basic widgets you can start playing with. unzip to usb, boot up your c8 and ssh into it and modify weather widget locations for starting.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Thanks a lot! That works, you DA MAN!

zurk wrote:

lgxb : look at  /mnt/usb/lighty/html/xapis/profile/show with a text editor where you can see the Weather Underground RapidFire widget which is not available on chumby.com. Replace the name with your own widget, copy your widget over to the same directory where the weather underground widget is located and  it should work. you need to run link.sh as noted above. You can also add another widget if you require by copying and pasting the chunk of XML which controls the weather underground widget.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Dear Zurk,

Maybe I'm hopeless.  I don't even know what "ssh into it" means.

Sorry.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Zurk,

   Thanks for the path, I found the show file, edited it on the usb stick from my pc, put it back in the chumby8, rebooted, and it picked up all the changes without running the link.sh script.  I was curious so I sshed into the chumby8, find the show file, edited it using vi to change the weather station to a different one, edited a few widget names to show the real name rather than pic or picx on the chumby, etc. Then I run link.sh and reboot chumby, thinking that it should pick up all the new changes I just made out of curiousity. However, the chumby apps did not change after the reboot.  I checked the show file to make sure the changes are there, but they are gone as if I never saved them after I edited the file.  I then take out the USB drive and plug it back to my PC and made those changes on the show file.  Plug it back to the chumby8 and reboot.  This time, the changes worked, again without the need to run link.sh.  I am puzzled but I am glad that it worked and I got the weather code personalized successfully thanks to your advices.   I appreciated your help and effort to make the offline chumby work so well.

CF

zurk wrote:

cf : wrong directory sorry....i just checked. look in  /mnt/usb/lighty/html/xapis/profile/, modify show and run link.sh...that will do it.
lgxb : look at  /mnt/usb/lighty/html/xapis/profile/show with a text editor where you can see the Weather Underground RapidFire widget which is not available on chumby.com. Replace the name with your own widget, copy your widget over to the same directory where the weather underground widget is located and  it should work. you need to run link.sh as noted above. You can also add another widget if you require by copying and pasting the chunk of XML which controls the weather underground widget.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Hi Zurk,

   From your explanation, it seemed to imply that we can only have one channel in the offline chumby with all the widgets in it.  Is it possible to have more than one channels with different selection of widgets like the online chumby?  If so, how do we set up the xml profiles for each channel?  Thanks in advance.

CF

zurk wrote:

its fairly easy to get 30 or even 100 widgets in your own version of the offline firmware. you need 2 files. the first is your profile file which is an xml file. then you need to download all files referenced by your profile from chumby servers using a tool called wget. the second is the offline firmware profile located in your usb disk once you have downloaded it. both profiles need to match so you have to modify the second to match the first. once you have done that its over and the chumby will play your widgets.
learning about xml and linux is your best starting point. see here: http://www.ibm.com/developerworks/linux … -v3-103-1/ and http://www.xmlfiles.com/xml/
i suggest you first download the c8 offline fw and get used to it though. it has a few very basic widgets you can start playing with. unzip to usb, boot up your c8 and ssh into it and modify weather widget locations for starting.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

you can only have one channel with the offline firmware at the moment. Having multiple channels is certainly possible but you would need to write code for it and put an executable shell script into the place of show which would then read the input and output the correct profiles as required. its non trivial but doable if you really want to do it. i am happy with one channel so i wont be developing it however until i have a need for it.  for an example of how you can go about doing it look at the shell script thats available in the speech directory. if you do it you can release your own offline firmware and post it up in the forum somewhere.

36 (edited by bobsz 2012-04-08 21:55:45)

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

@wuberry- I posted something just like you a month ago, totally clueless. I can actually do the basics now. It's not as mysterious as it sounds. SSHing means using a program like PuTTY to let your computer get inside your Chumby. Look up my recent posts and you'll see the forum folks gave me a lot of info, starting with my post called "Need help with basics of debugchumby, SSH, etc please?" at-
http://forum.chumby.com/viewtopic.php?id=7856

More like that followed, like- http://forum.chumby.com/viewtopic.php?id=7900
If you want more "basic" advice, maybe you should start a new topic, and some of us could help you there. This thread might be a little too specific for where you're at. And I still can't do things like the commands they're talking about above, but I'm getting there.

@chumbyfly- I also thought about multiple channels but from my totally non-technical point of view. I could never do anything like you're attempting, so I figured I could just put different channels onto different USB sticks. They're so cheap now, and you don't need anything big or fast to run all of Zurk's widgets, and others I've tried. Zurk's firmware seems to boot up really fast now after changing USB sticks. I've even found the tiny toy-like sticks like the PNY Micro Sleek Attache work fine.

@zurk- my apologies for taking this thread so far off track. I have some more pertinent questions I'll be posting later, but thanks again for your work and for sticking around the forums.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

bobsz,

   I like your idea of using a different USB stick for different channel.  Now if I can just get those widgets from different channels to show up in my widgetcache so I can download them.  I SSH'ed into my online chumby8 and cd to the widgetcache directory after clicking the play button on a channel and wait until it starts playing the widgets of the channel... All I saw in the widgetcache is just 4 widgets and an xml file which only contains info for one of the widgets.  My channel has over 70 widgets, where did the rest of them go?  Does that mean Chumby8 does not always cached all the widgets in a channel?  Or do I need to wait for the channel to play through one cycle (i.e. each widget appears once) before the widgetcache will be filled? 

Regards,

CF

bobsz wrote:

@wuberry- I posted something just like you a month ago, totally clueless. I can actually do the basics now. It's not as mysterious as it sounds. SSHing means using a program like PuTTY to let your computer get inside your Chumby. Look up my recent posts and you'll see the forum folks gave me a lot of info, starting with my post called "Need help with basics of debugchumby, SSH, etc please?" at-
http://forum.chumby.com/viewtopic.php?id=7856

More like that followed, like- http://forum.chumby.com/viewtopic.php?id=7900
If you want more "basic" advice, maybe you should start a new topic, and some of us could help you there. This thread might be a little too specific for where you're at. And I still can't do things like the commands they're talking about above, but I'm getting there.

@chumbyfly- I also thought about multiple channels but from my totally non-technical point of view. I could never do anything like you're attempting, so I figured I could just put different channels onto different USB sticks. They're so cheap now, and you don't need anything big or fast to run all of Zurk's widgets, and others I've tried. Zurk's firmware seems to boot up really fast now after changing USB sticks. I've even found the tiny toy-like sticks like the PNY Micro Sleek Attache work fine.

@zurk- my apologies for taking this thread so far off track. I have some more pertinent questions I'll be posting later, but thanks again for your work and for sticking around the forums.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Zurk,

   you are over estimating my ability to make my own versions of your software.
I shall give it a try anyway and see how far I get using the speech directory example you provided.  Thanks.  Meanwhile, how best to capture the multiple channels xml and widgets I have set up in my chumby8?  Please advice.  Thanks in advance.

CF

zurk wrote:

you can only have one channel with the offline firmware at the moment. Having multiple channels is certainly possible but you would need to write code for it and put an executable shell script into the place of show which would then read the input and output the correct profiles as required. its non trivial but doable if you really want to do it. i am happy with one channel so i wont be developing it however until i have a need for it.  for an example of how you can go about doing it look at the shell script thats available in the speech directory. if you do it you can release your own offline firmware and post it up in the forum somewhere.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

best way is to wget the profiles and wget all the widgets contained in em. you can see the profiles by sshing into the chumby, setting putty log on, then running :
stop_control_panel
start_control_panel
and then manually changing channels on the chumby. the urls will show exactly as the offline firmware ones do (i.e. xapis/profile etc with the server xml.chumby.com)
bob: no worries about taking the thread off track. its a development thread and feel free to use it to develop your own versions of the offline firmware.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Thank you so much for all your help!

While I'm certainly not sure of what I'm doing, all your advice makes this a less daunting task.

I'll give it a shot in the next few days.

Thanks, again.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

zurk wrote:

its fairly easy to get 30 or even 100 widgets in your own version of the offline firmware. you need 2 files. the first is your profile file which is an xml file. then you need to download all files referenced by your profile from chumby servers using a tool called wget. the second is the offline firmware profile located in your usb disk once you have downloaded it. both profiles need to match so you have to modify the second to match the first. once you have done that its over and the chumby will play your widgets.
learning about xml and linux is your best starting point. see here: http://www.ibm.com/developerworks/linux … -v3-103-1/ and http://www.xmlfiles.com/xml/
i suggest you first download the c8 offline fw and get used to it though. it has a few very basic widgets you can start playing with. unzip to usb, boot up your c8 and ssh into it and modify weather widget locations for starting.

So your a saying that I can create my own xml profile, dowload the widget (say the "tea timer" widget) from the chumby.com servers, put them on my usb drive, and reference them from the xml profile. And if so, what about the cache? Is it stored in ram or the usb drive?

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

ignore the cache. its supposed to make life easy if you dont want to download with wget but in reality all it does is cause confusion. download directly with wget by reading the URLs from the XML profile file you download.
the cache also slows down the chumby and does all sorts of silly things like breaking widgets by caching data. Thats the reason the offline firmware completely disables the cache functionality and gets a performance boost and reduces cpu/memory load at the same time. the chumby cache system is hopelessly buggy and broken regardless of what anyone tells you.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Excuse me, what *exactly* is the bug you're talking about in the widget caching?  All it does is download and store widgets on the SD card (or RAM on Classic).  It either works or it doesn't.  I don't see how it can possibly "slow down the chumby" once the widget has been downloaded.

I did benchmarks to show a substantial increase in the performance of the device once widget caching was enabled - they no longer used up the valuable HTTP cache or consumed network bandwidth.

What did you discover that doesn't match my findings? If you've found a bug in the caching, I'd be happy to fix it.  It sounds like you have enough information to help me locate it.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

bunch of problems with caching -- i'll detail a few, mostly long term ones :
1. caching of stale data -- a good example is accuweather. run it for about 2-3 weeks and you will start to see repeat days (wed wed thu instead of wed thu fri)
2. bad caching of updated widgets -- a good example is all in one clock which is loaded from an offsite location as well as chumby.com - causes unnecessary HTTP requests and the widget loaded over and over again.
3. bad caching of image data - long term loading of webcam images causes the last image to be reloaded over and over.
4. semi restarts on large data in cache -- when there are a bunch of widgets (>100) playing for over a week the control panel with randomly restart itself. not reboot just restart. it goes thru auth again.
5. crash to offline mode -- seems to happen 1x every 2-3 months. the chumby just dumps to offline mode clock and shows fully connected (network is green) but stays on offline mode. i caught a trace that indicated it might have been tied to the cache when it did a retrieval.

note that all of these issues were solved by dumping the cache system completely and switching to offline mode (load from usb disk, shut off flashplayer cache). so far i have over 1 month uptime on some of my offline chumbies with no issues.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

1) Accuweather sends all of the data for a week in a chunk, not individual days - if you see "wed wed thu", then that's because they sent the data that way in the first place.

2) I'm not sure what you're saying here - if the user set up his server to disable caching, then the widget will not be cached.  The app follows the caching directive in the HTTP header - a missing header is treated as "cache for 2 hours".  Chumbys *used* to cached widgets in the HTTP cache for up to 12 hours, but the new cache code detects updated widgets and will recache new ones once it receives a new channel that shows a new version number.

3) The webcam must set up a header to prevent caching, or the widget needs to use a cache buster URL.

4) Some third-party widgets have memory leaks that eventually cause the app to restart - that has nothing to do with caching except that caching consumes memory, in which case the device will run out of memory sooner than without cache.  The Control Panel actually *intentionally* restarts on a regular basis if it thinks it can do so safely (ie, not in Night Mode, not playing music, etc), typically in the middle of the night, in order to alleviate the issues with widget leaks.

5) The network code is not sensitive to caching behavior.

We've recently seen a large server load uptick in redundant requests for widget parameter information by widgets that would cache under normal circumstances - if you disabled the flashplayer cache, and a significant number of people are using your image in an online mode, that would explain it.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

1. would not explain why a reboot would cure the behavior.
2. ok makes sense.
3. or disable the cache which works fine. unless you somehow can reverse engineer proprietary webcam/ip security cam firmware.
4.  ugh. can you tell me how to eliminate this behavior from the control panel ?? is there a nice switch in the flash player config file which will disable the control panel restarts ?? i would rather fix a memory leaking widget than randomly have the control panel auto restarting.
5. unlikely since all access to chumby servers (*.chumby.com) is blocked through a hosts file and there is no online mode. more likely something broke in the new control panel that was pushed out to remove NYT podcasts. the 500+ chumbies running offline firmware are unlikely to be more than a blip if they somehow managed to get to chumby.com anyway.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

1) I can't explain that.  Some weather services (NOAA is one) will sometimes skip the forecast for the next day if the current day is almost over.  We didn't write the Accuweatther widget, so I can't give you a definitive answer.  A reboot may be flushing out bad data that the widget is choosing to cache, not mitigating a bug in the cache itself.

3) The trouble with this is that you're changing this behavior for *every* widget, which means that servers are getting much higher loads than necessary.

4) I can see if I can add a preference for that.  Not sure how you're going to fix leaks in third party widgets.  We added this behavior because we could not enforce this on the part of external devs.

5) Literally, the *only* change to the Control Panel for that release was to change the result of the "sourceAvailable()" function to "false" for the NYT widgets music source.  Before it returned true if the network was reachable.  We're still trying to determine whats causing the increase in load, and i'm glad to be able to rule out your image as a possible culprit.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Zurk,

   I followed your instructions and found that I must wait for the selected channel to play through all the widgets in that channel for the URLs that the wget needed to show up in the log/trace.  I also found that the widgetcache will only cache the widget one at a time (i.e. when the channel move to the next widget, it will then download the next widget and remove the previous widget from the cache even though the widget cache is no where near the 2MB limit).  Also some widgets will not cache at all (I don't know where the system kept them, if it is not in the widgetcache). For multiple channel offline mode, I wonder if we can do it with multiple profiles using this Chumby trick I found in the wiki:

Between widgets, the Control Panel monitors the existence of /tmp/change_profile - if present, will
attempt to change channels to the channel name in the file. The file is then deleted. This allows external
scripts to initiate a channel change.
You can also send a Control Panel event


Regards,

CF

zurk wrote:

best way is to wget the profiles and wget all the widgets contained in em. you can see the profiles by sshing into the chumby, setting putty log on, then running :
stop_control_panel
start_control_panel
and then manually changing channels on the chumby. the urls will show exactly as the offline firmware ones do (i.e. xapis/profile etc with the server xml.chumby.com)
bob: no worries about taking the thread off track. its a development thread and feel free to use it to develop your own versions of the offline firmware.

49 (edited by zurk 2012-04-11 10:59:30)

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

yup. told ya the caching is broken. for the widgets it didnt cache they are just downloaded over and over again pointlessly. its easier to just download the channel profile file and wget the URLs contained inside it by reading the xml file and dumping everything you find out to wget one at a time.
basically all the chumby does is download the XML profile, go through it to find the widget urls contained in the xml profile and then load those urls (with some weird caching going on which can be ignored) and play em in the order it finds. its just a simple loop.
you can probably rig something up if you get the timing right or fudge it with a cpanel event. its probably easier just to have a shell script which returns the right profile to the control panel the way it thinks is supposed to work. either way should be ok, my preference would be for doing the shell script since that is likely to be the easier way.

edit: heres the semi - automated way i use for extracting URLs from profiles -
cat profile_filename | grep -o 'http://[^"]*'
It doesnt get everything (i.e. relative urls like /xapis/catalog/index/....) but it gets most. The others you need to get by appending http://xml.chumby.com/xapis.... to them with search and replace using nano (also available on the offline fw) or any other text editor.
if you have a linux box the following is better :
perl -ne 'if (/href="([^"]*)"/) { print "wget $1\n"; }' profile_filename
it will get everything. you just need to modify relative urls by adding http://xml.chumby.com in front of them.

Re: Zurk's Chumby 8 Offline Firmware ( Insignia Infocast / chumby8 only! )

Zurk,

   Thanks for the scriptlets for extracting the hrefs.  I appreciate it.
   This is a stupid question but where is the path of the xml profile of the channel located in the online chumby?
   I am currently relying on the start_control_panel output; which is not exactly the profile but it contains enough info to locate the widgets.  If the channel xml profile can be found, I don't have to wait for the channel play to cycle through all the widgets to collect the widget urls.


Regards,

CF

zurk wrote:

yup. told ya the caching is broken. for the widgets it didnt cache they are just downloaded over and over again pointlessly. its easier to just download the channel profile file and wget the URLs contained inside it by reading the xml file and dumping everything you find out to wget one at a time.
basically all the chumby does is download the XML profile, go through it to find the widget urls contained in the xml profile and then load those urls (with some weird caching going on which can be ignored) and play em in the order it finds. its just a simple loop.
you can probably rig something up if you get the timing right or fudge it with a cpanel event. its probably easier just to have a shell script which returns the right profile to the control panel the way it thinks is supposed to work. either way should be ok, my preference would be for doing the shell script since that is likely to be the easier way.

edit: heres the semi - automated way i use for extracting URLs from profiles -
cat profile_filename | grep -o 'http://[^"]*'
It doesnt get everything (i.e. relative urls like /xapis/catalog/index/....) but it gets most. The others you need to get by appending http://xml.chumby.com/xapis.... to them with search and replace using nano (also available on the offline fw) or any other text editor.
if you have a linux box the following is better :
perl -ne 'if (/href="([^"]*)"/) { print "wget $1\n"; }' profile_filename
it will get everything. you just need to modify relative urls by adding http://xml.chumby.com in front of them.