Re: 50 Widgets per channel limit?

The thought is that you'd create a file called something like "chumby_cache_widgets" on the usb drive.  The presence of this file would cause the CP to make a "widgetcache" directory on that drive and store the widgets within that.

Re: 50 Widgets per channel limit?

Okay, I would let the user directly create the "widgetcache" directory. However, that's not such an important detail (although for Windows users it's probably easier to create a directory instead of a file without a known extension).

28

Re: 50 Widgets per channel limit?

1ras wrote:

@Xav: The web interface can't know if the Chumby has an USB Stick for caching available or not when you create or change your channels. The Chumby is not even required to be online while you change the channels and at any time you can remove the USB stick. Therefore the limit need to be dynamic depening on the current situation and not on what the user configured while he had an USB stick plugged in.

Just to clarify, I was suggesting that there's still a line in the web interface with 50 widgets (or 80 points - though Duane's comments make that an unlikely approach) above it, and the rest below. It's more of an advisory to the user: "If you're using a CC and you don't have a USB cache, you'll only get the widgets up to this line, otherwise you get all of them".


Would it be possible to have a named file of a particular fixed size (say, 200MB) that is then mounted via a loopback interface so the kernel treats it as a block device? That way the cached files are hidden away inside a single file, and the user has allocated a fixed size from the outset so there's less likelihood of running out of space if using the USB drive for something else as well.

A suitable pre-created file could be made available for download. Becuase it starts off empty, it would zip down to almost nothing for distribution.

Re: 50 Widgets per channel limit?

You can store all widgets available for the CC on a 128 MB stick. Yes, 128 megabyte for all of them! Even if you buy the cheapest USB stick available you have plenty of space for the cache.

30

Re: 50 Widgets per channel limit?

1ras wrote:

You can store all widgets available for the CC on a 128 MB stick. Yes, 128 megabyte for all of them! Even if you buy the cheapest USB stick available you have plenty of space for the cache.

Yes, you mentioned that on the previous page, which is why I suggested a loopback file of 200MB: big enough to hold all the widgets, plus any new ones that might come along, plus a bit of overhead in case there's a future benefit to caching more than just the widgets.

But whether the cache size is 200MB or 20MB wasn't the point. If all you have on the stick is a folder then there's the possibility of running out of space. Consider someone who creates their cache folder, then decides to load the USB stick up with MP3 files for local playback before putting it into their CC; they could inadvertantly leave no practical space on the stick for the cache even though they've created the folder.

By using a loopback file the cache size is set in stone from the outset, so trying to copy those MP3 files over would result in there being insufficient space for the last one, but the cache is still the right size.

This approach would simplify the web interface. By knowing that the cache size is the same for all the Classics the UI can be as simple as a cutoff line which says "widgets past the line will only appear if you have a USB cache". With just a cache folder the *actual* amount of cache space is variable, so the web end would have to be more ambiguous - "widgets after this line may appear if you've got a USB cache, but exactly which ones is impossible to say because it depends on the size of the cache" wink


Of course the actual implementation is a moot point because it's down to what Duane chooses to implement - I just wanted to suggest a loopback file in the unlikely event that he hadn't already considered that option, because it could possibly make some aspects o fthe implementation a little more robust.

31 (edited by 1ras 2014-07-19 04:05:24)

Re: 50 Widgets per channel limit?

It makes the implementation more complicated which is usually the opposite of robust. KISS - keep it simple stupid is what I would do.

There is no need to tell the user on the web interface what can be cached and what not. And if I remember correctly it is not even easily possible as you can run the same channel on different devices at the same time. On the other side the CP need to be anyway modified, it is more than enough to tell the user on the CP that some widgets are skipped when it happens.

The CP does anyway need to know the cache size, independent where it is cached, on /tmp/widgetcache, /usr/mnt/widgetcache or on /usr/mnt/widgetcache.img. So the actual cache size can be fully dynamic and whenever something does not fit in the cache it is skipped.

I would choose the easiest possible implementation which allows the user to have more than 50 widgets per channel by not overloading the server.

Re: 50 Widgets per channel limit?

Well, honestly I think a cache file would be a better (and easier to implement) solution. Duane would just host the blank file on chumby.com, maybe link to it on the sidebar (Something like "Earth to Chumby Classic Owners! Want to have more than 50 widgets, decreased loading times, and help Blue Octy out? Just download this file to a USB flash drive (minimum size: 128 MB) and plug it in to your Chumby's USB port!). Simply detecting a file and cacheing there would be easier than the control panel having to look at the flash drive, detect how much space is left, and have to cache widgets in that limited space. Plus, what if there's no space left on the flash drive from the presence of other files? Does the device still display more than 50 widgets solely based on the existence of the "cache" folder?

33

Re: 50 Widgets per channel limit?

1ras wrote:

It makes the implementation more complicated which is usually the opposite of robust. KISS - keep it simple stupid is what I would do.

Nathanm's comments more succintly make the point I was trying to get across, that it should actually make the implementation simpler. A fixed cache of known size (big enough to hold all the widgets and more) is likely to be simpler than dealing with a dynamic cache that could potentially be too small to hold even one widget.


1ras wrote:

There is no need to tell the user on the web interface what can be cached and what not. And if I remember correctly it is not even easily possible as you can run the same channel on different devices at the same time.

Did you even read what I said about the web interface? Currently there's a cut-off line at 50 widgets which applies across all devices. I simply suggested that the text by the line could be modified to indicate that any widgets after it would only appear on a CC if it had the cache added, but would appear on all other devices. Exactly the same as the current implementation, but lifting the 50 widget limit for those cases where it doesn't need to apply.


1ras wrote:

I would choose the easiest possible implementation which allows the user to have more than 50 widgets per channel by not overloading the server.

So would I - and to my mind that's a cache file of sufficient size that all the widgets can fit in with room to spare. Anything dynamic raises extra possibilities that need to be dealt with. But as I said, it's really down to Duane as to how (and even if) he chooses to implement it.

34 (edited by 1ras 2014-07-20 07:20:03)

Re: 50 Widgets per channel limit?

Lets compare just the very first step to enable the USB cache and have a look on complexity and possible user errors:

1. My approach:
User to create a widgetcache directory on the USB stick.

Possible issues:
- none unique to this approach, allmost every user is aware how to create a folder, this is an easy task on every OS

2. Duane's approach:
User to create a enable_cache file (similar to the known start_sshd file) on the USB stick.

Possible issues:
- Windows users may end up with a enable_cache.txt file, they are unable to see the txt extention nor are they able to remove it

3. Your approach: User to download a zipped cache image and unzip it to the USB stick.

Possible issues:
- user may have issues with downloading the file
- user may have issues with extracting the file
- user may copy the zip file to the USB stick instead of extracting it
- user may rename the file instead of extracting it
- virus scanner may block the zip file or the extractet image as he detected a virus (false positive)
- the extracted image file may get truncated and damaged as it may not fit on the USB stick (USB stick could be full with other files)

When I learned something then that users unfortunately run into every issue possible :-(

35 (edited by Xav 2014-07-20 08:33:06)

Re: 50 Widgets per channel limit?

1ras wrote:

1. My approach:
User to create a widgetcache directory on the USB stick.

Possible issues:
- none unique to this approach, allmost every user is aware how to create a folder, this is an easy task on every OS

I agree that this is simple for the user, but it complicates the Chumby code as the cache is an indeterminate size. At this point I would favour whichever approach that makes Duane's task easier, which I suspect is to have a known size of cache.

I notice how my approach includes "USB stick could be full with other files", but that's conveniently left out of the issues with your approach - even though it's the biggest potential problem.

You also forgot to mention that most users are using Windows, with its case-insensitive filenames, but the Chumby is based on Linux where they're case-sensitive. So although the user may be able to create a folder, it could still easily have the wrong name - it's not necessarily the simple task you suggest.

With this approach there's no way for the web interface to know how many widgets will be cacheable, so there's no clean way to show which ones will or will not appear on the device.


1ras wrote:

3. Your approach: User to download a zipped cache image and unzip it to the USB stick.

Possible issues:
- user may have issues with downloading the file
- user may have issues with extracting the file
- user may copy the zip file to the USB stick instead of extracting it
- user may rename the file instead of extracting it
- virus scanner may block the zip file or the extractet image as he detected a virus (false positive)
- the extracted image file may get truncated and damaged as it may not fit on the USB stick (USB stick could be full with other files)


My suggested approach was to download an *unzipped* file. Zipping was offered as an option to save bandwidth, but wouldn't be my preference as it does slightly complicate matters.

Furthermore "the extracted image file may get truncated and damaged as it may not fit on the USB stick" is just rubbish. Any sane OS or zip program simply won't copy the file to the USB stick if there's insufficient room.

That leaves the following potential issues:

- user may have issues with downloading the file
- virus scanner may block the zip file or the extractet image as he detected a virus (false positive)

Which are both valid concerns, but likely to affect only a small minority of people.

The advantage, however, is that the cache is of a known size:

- The code only has to check that it's the expected size at boot time, or whenever the USB stick is inserted/removed, not every time a widget is downloaded.
- The web interface can clearly and simply show which widgets will appear with no cache used, and which will appear when the cache is enabled.


Anyway, as you've not yet picked up on the hints I've been dropping, I'll spell it out: I don't see any point continuing this conversation. Duane will do what's best for him, and our opinions won't have any real effect on the outcome. You want to make it trivial for the user, but in doing so put extra burden on the Chumby and web interface code. I want to keep it simple from a programming perspective (as Duane has more than enough on his plate already) which ever so slightly complicates it for the user - but only to the extent that they have to download a single file and put it on a USB drive. It's clear we won't agree on this, so let's agree to differ, and leave Duane to do whatever he feels is best - a decision that could easily be based on a whole load of variables that we know nothing about anyway, so speculation on our part about the "best" approach is uninformed at best.

36 (edited by 1ras 2014-07-20 09:59:50)

Re: 50 Widgets per channel limit?

Xav wrote:

I agree that this is simple for the user, but it complicates the Chumby code as the cache is an indeterminate size.

I am not a flash developer but I guess flash is - like every other programing language - able to check the free space of a device before storing something on it.

Xav wrote:

At this point I would favour whichever approach that makes Duane's task easier, which I suspect is to have a known size of cache.

I suspect it isn't, as properly handling a loop-mounted file system inside an image stored on a removable media isn't such a straightforward task as you may think. While the USB stick is auto-mounted when removed and added you have to detect this situation and make sure that you remount your image as well. Basically you have to check it prior to each access and take the required actions. At the same time you could simply check the free space on the USB stick as I suggested.

Xav wrote:

I notice how my approach includes "USB stick could be full with other files", but that's conveniently left out of the issues with your approach - even though it's the biggest potential problem.

It isn't on my approach as you use the cache as long as there is space, independent where the cache is stored and how much total space is available for the cache.

Xav wrote:

You also forgot to mention that most users are using Windows, with its case-insensitive filenames, but the Chumby is based on Linux where they're case-sensitive. So although the user may be able to create a folder, it could still easily have the wrong name - it's not necessarily the simple task you suggest.

This isn't true as a FAT file system (and that's the only one the CC supports on the USB stick) is case-insensitive on Linux as well.

Xav wrote:

With this approach there's no way for the web interface to know how many widgets will be cacheable, so there's no clean way to show which ones will or will not appear on the device.

Which simplifies the implementation as there are no modifications required on the web interface (beside a small configuration change to allow an higher amount of widgets per channle for all Chumby devices).

Xav wrote:

My suggested approach was to download an *unzipped* file. Zipping was offered as an option to save bandwidth, but wouldn't be my preference as it does slightly complicate matters.

Furthermore "the extracted image file may get truncated and damaged as it may not fit on the USB stick" is just rubbish. Any sane OS or zip program simply won't copy the file to the USB stick if there's insufficient room.

Agreed. As we no longer zipping the file replace it with:
- the file get truncated during download and damaged

This is one of the main issues why people end up with damaged downloads and why people offer checksum files to check file integrity. Unfortunately especially Windows users are not familiar how checking file integrity works.

Xav wrote:

The advantage, however, is that the cache is of a known size:

Which isn't needed on my approach so it can't be an advantage :-)

Xav wrote:

- The code only has to check that it's the expected size at boot time, or whenever the USB stick is inserted/removed, not every time a widget is downloaded.

Nope, as explained above you have to check basically prior to every access if your image is still mounted and remount if needed.

Xav wrote:

- The web interface can clearly and simply show which widgets will appear with no cache used, and which will appear when the cache is enabled.

Unnecessary, I can't see a reason to bother the majority of people who run anyway not into the 50 widgets per channel limit.

Xav wrote:

Anyway, as you've not yet picked up on the hints I've been dropping, I'll spell it out: I don't see any point continuing this conversation. Duane will do what's best for him, and our opinions won't have any real effect on the outcome.

I guess the goal is to find pros and cons of different approaches to see which one fits best.

Re: 50 Widgets per channel limit?

I think a healthy, friendly debate is good.

Duane likely knows more about what's going on in the control panel, and what would have to be changed, however, there may be aspects he hasn't thought of, or new ideas that haven't occured to him.

the end result may not be exactly any of the ideas here, but may incorporate aspects of them.

so, I think letting the ideas flow a good thing smile  (just don't get TOO vigorous defending your turf wink)

Cleaning up any loose bits and bytes.

38

Re: 50 Widgets per channel limit?

Okay, as Diamaunt wants us to continue, I'll bite.

1ras wrote:

[I am not a flash developer but I guess flash is - like every other programing language - able to check the free space of a device before storing something on it.

I never claimed it couldn't.

1ras wrote:

I suspect it isn't, as properly handling a loop-mounted file system inside an image stored on a removable media isn't such a straightforward task as you may think. While the USB stick is auto-mounted when removed and added you have to detect this situation and make sure that you remount your image as well.

This isn't as tricky as you're suggesting. A simple shell script that fires on USB insert/removal can trivially check for a file of a known name on the device, then attempt to mount it as a loopback device - assuming the Chumby has the necessary kernel modules and userspace tools. From there it's just a case of setting a flag to indicate to the control panel whether or not the USB cache is available.


1ras wrote:

This isn't true as a FAT file system (and that's the only one the CC supports on the USB stick) is case-insensitive on Linux as well.

I'll gladly concede this point - I'd forgotten we were dealing with FAT. That said, this does limit your cache to being FAT also, whereas using a loopback file allows for other filesystems. It may be that the existing cache code assumes EXT3, for example, so allowing for FAT might require additional changes. This is speculative, however, and unlikely to be a major issue, so I'm not using this as an argument either way, but just noting that this may need to be considered.


Let's talk algorithms, and maybe you'll see why I think a loopback device of known size is a simpler thing to implement from a coding perspective. In these examples I'll refer to the existing 2MB cache as the "internal" cache, and the cache on the USB stick as the "external" cache:


1) Supplied file mounted as a loopback device

* A shell script, or executable, runs at boot, and on insertion/removal of a USB stick. This initially sets two variables to 0: a boolean (USE_EXTERNAL_CACHE) and an integer (EXTERNAL_CACHE_ERROR).
* If a file of the correct name is present on the USB stick, check its size and try to mount it as a loopback device.
* If it cannot be mounted, set EXTERNAL_CACHE_ERROR to a non-zero error number and exit.
* If it can be mounted, run some rudimentary checks on its structure - mainly that the mounted filesystem is of a minimum defined size, and optionally that it's the right format. Other tests could be conceived if necessary. If any fail, set EXTERNAL_CACHE_ERROR to a non-zero error number and exit.
* Assuming everything was okay, set USE_EXTERNAL_CACHE to 1.
* NOTE: All the above happens at the Linux level, and is asynchronous to the control panel code.

* The control panel is ideally modified to display a suitable error if EXTERNAL_CACHE_ERROR is non-zero.
* The caching code is modified to check for the presence and value of USE_EXTERNAL_CACHE. If it's absent or 0 then the current code applies - a 50 widget limit using the internal cache.
* If USE_EXTERNAL_CACHE is 1 then the path to the cache is changed to use the loopback device. All the other cache code remains the same, it's just the location of the cache that is different.
* In this scenario the internal cache is used as a fallback option, but if the external cache is present that is used exclusively. Only one cache is used at a time.
* There's no change to the cache algorithm itself, although the external cache is large enough to hold every widget, so in practice nothing will ever be thrown out of that cache.


2) Use folder on the USB stick.

* At programming time, the cache is an indeterminate size. It could be 1 byte, 1MB or 1GB.
* In case it's only 1 byte, we can't use the external cache exclusively. We therefore have to consider the external cache to be an *addition* to the internal cache, so the caching code needs to be modified to deal with two caches.
* If it's 1MB then the caching algorithm, when it needs to cache a widget, has to check the free space on the internal cache and put it there if it can, otherwise it needs to check the external cache and put it there if it can.
* If it won't fit in either cache, the code needs to start removing items from the caches until one of them has enough space for the new widget.

Let's consider a worst-case scenario: a 1MB cache filled with 40K widgets, and the next one along is the licky dog 400K widget...

* Check both caches to see which widget is least recently used, and throw it away. We've gained 40K of space in one cache, still not enough for the licky dog.
* Repeat the above, throwing away the next least recently used. As this is a worst case scenario, we've ended up with our widgets perfectly interleaved between the two caches, so we throw the first out of external, the second out of internal, the third out of external, and so on.
* Repeat the above, throwing out widget from both caches until eventually one of them has 400K of free space. But to get there we've thrown out 19 widgets, 9 of which didn't need to be thrown out at all!

An alternative cache invalidation algorithm could reduce this issue, but that's introducing a lot more coding work, and throwing out tried and tested code to do so.

* In the best case scenario of 1GB of free space (actually anything >128MB) there's no need to throw anything out of the cache so this issue doesn't arise. This is also the most likely scenario, but you can't rely on that and have to write the code accordingly.


So with option (1) there's a small shell script to write, plus a minor change to the caching code. If the path to the cache is stored in a variable, it could be as simple as changing that one variable declaration into an "if" statement.

With option (2) the caching code has to be changed to check two caches when looking for a cached widget, to check two caches when trying to find space for a new one, to check two caches when trying to remove a widget to make space. As I've shown, this can lead to inefficiencies in the caching mechanism - or the need to replace the existing caching algorithm entirely. As a large part of this excercise is trying to reduce bandwidth use, a system that can throw away widgets unnecessarily is also a poor choice.

From a coding perspective, option (1) looks a lot simpler to me. It uses the existing caching code with minimal modifications, simply switching wholesale to use a larger cache if it's available. Option (2) requires changes to the caching code to deal with two simultaneous caches, and possibly a major change to the caching algorithm if inefficiencies are to be avoided.



1ras wrote:
Xav wrote:

[Referring to a cache directory on the USB stick]
With this approach there's no way for the web interface to know how many widgets will be cacheable, so there's no clean way to show which ones will or will not appear on the device.

Which simplifies the implementation as there are no modifications required on the web interface (beside a small configuration change to allow an higher amount of widgets per channle for all Chumby devices).

I don't understand. Are you really suggesting that the limit can be removed entirely, or at least raised across all devices? What about the CC users who never add a cache? How about those who add one, but only leave 1 byte of space? I can't see how this is meant to work in practice, so please explain if I've misunderstood what you mean.

39 (edited by 1ras 2014-07-21 13:40:54)

Re: 50 Widgets per channel limit?

Xav wrote:
1ras wrote:

I suspect it isn't, as properly handling a loop-mounted file system inside an image stored on a removable media isn't such a straightforward task as you may think. While the USB stick is auto-mounted when removed and added you have to detect this situation and make sure that you remount your image as well.

This isn't as tricky as you're suggesting. A simple shell script that fires on USB insert/removal can trivially check for a file of a known name on the device, then attempt to mount it as a loopback device - assuming the Chumby has the necessary kernel modules and userspace tools. From there it's just a case of setting a flag to indicate to the control panel whether or not the USB cache is available.

And now we are talking about a firmware upgrade instead of simply delivering an updated controlpanel.swf. There are of course several reasons while a firmware upgrade would be nice (e.g. the buggy and outdated WLAN driver is one which would also require a kernel update) but this feature isn't a reason in my opinion. We are then also talking about a completely different layer of complexity. Therefore I would like to skip everything which would require a firmware upgrade for now.

Xav wrote:
1ras wrote:

This isn't true as a FAT file system (and that's the only one the CC supports on the USB stick) is case-insensitive on Linux as well.

I'll gladly concede this point - I'd forgotten we were dealing with FAT. That said, this does limit your cache to being FAT also, whereas using a loopback file allows for other filesystems. It may be that the existing cache code assumes EXT3, for example, so allowing for FAT might require additional changes. This is speculative, however, and unlikely to be a major issue, so I'm not using this as an argument either way, but just noting that this may need to be considered.

The CC kernel does simply not support ext* file systems and there is no such kernel module available either. The widgetcache implementation which already exists on the CC would also not require an ext* file system, FAT is just fine.

Xav wrote:

Let's talk algorithms, and maybe you'll see why I think a loopback device of known size is a simpler thing to implement from a coding perspective. In these examples I'll refer to the existing 2MB cache as the "internal" cache, and the cache on the USB stick as the "external" cache:

1) Supplied file mounted as a loopback device

I have read it but I am not further commenting on it as it would require a firmware upgrade.

Xav wrote:

2) Use folder on the USB stick.

* At programming time, the cache is an indeterminate size. It could be 1 byte, 1MB or 1GB.
* In case it's only 1 byte, we can't use the external cache exclusively. We therefore have to consider the external cache to be an *addition* to the internal cache, so the caching code needs to be modified to deal with two caches.
* If it's 1MB then the caching algorithm, when it needs to cache a widget, has to check the free space on the internal cache and put it there if it can, otherwise it needs to check the external cache and put it there if it can.

It's checking the maximum size of each cache. Maximum size of the cache is used cache + free space (just to make sure we're talking about the same thing).

Checking the free space of a device is as simple as: df /mnt/usb/widgetcache | awk 'NR==2{print $4}'
Checking the used size of a cache is as simple as: du /mnt/usb/widgetcache | awk '{print $1}'
Both values added is the maximum cache size of each cache.

As said I'm not a flash developer, you may get the values easily within flash or you may get it from somewhere else, maybe just by calling the above commands.

The cache with the biggest maximum cache size is used for caching. With other words if the cache on the USB stick isn't bigger as the one on the RAMdisk then the USB stick is not used for caching. Thinking about an implementation which could simultaneously cache on both is IMHO not worth the time.

Xav wrote:

* If it won't fit in either cache, the code needs to start removing items from the caches until one of them has enough space for the new widget.

When the free space of the selected cache location is below a pre-defined value then the cache is detected as full. You definitely have to leave a small amount of free space on the RAMdisk as it is used for other stuff as well and you should do the same on the USB stick. A similar code is probably already in place for the RAMdisk cache (or Duane is crossing the fingers and hoping that the RAMdisk is never running full with caching 50 widgets of unknown sizes, who knows ;-) ) and can be used for the USB cache as well.
You also would like to keep some free space because as Duane pointed out you don't know the size of the widget in advance that you would like to download. You would like to keep at least enough free space to store the next widget.
It's anyway a good practice to not fill up a file system to 100%.

When the cache is detected as full then no more widgets are downloaded from the Chumby server and the Chumby is skipping any widgets which are not in the cache. There should be of course a small message e.g. displayed on the CP to make the user aware why some widgets are not displayed. E.g. the space where the CP shows the widget preview image can be used for this message.

Xav wrote:
1ras wrote:
Xav wrote:

[Referring to a cache directory on the USB stick]
With this approach there's no way for the web interface to know how many widgets will be cacheable, so there's no clean way to show which ones will or will not appear on the device.

Which simplifies the implementation as there are no modifications required on the web interface (beside a small configuration change to allow an higher amount of widgets per channle for all Chumby devices).

I don't understand. Are you really suggesting that the limit can be removed entirely

I can only talk for the caching topic: Yes, you can remove the entire limit.
But of course, there may be another limitation in place on the Chumby devices (I think Duane mentioned earlier that the limit is 100 widgets per channel) which you still need.

Xav wrote:

, or at least raised across all devices?

Raising it across all devices is probably what you would like to do if you still need a limit for a different purpose not covered by the caching topic.

Xav wrote:

What about the CC users who never add a cache? How about those who add one, but only leave 1 byte of space? I can't see how this is meant to work in practice, so please explain if I've misunderstood what you mean.

There is always a cache on the CC which is the one on the RAMdisk. If you don't add a sufficient cache which is bigger as the existing one then you have not added a cache and the RAMdisk is used for caching as usual.

If the cache is full then the Chumby stops downloading new widgets and only plays the one available in the cache. To make the user aware the CP displays a suitable message.

40

Re: 50 Widgets per channel limit?

1ras wrote:

And now we are talking about a firmware upgrade instead of simply delivering an updated controlpanel.swf.

As it's just a simple shell script, I can think of at least a couple of ways that it could be deployed without requiring a full firmware upgrade. That said, I started this whole point with a question: "Would it be possible to have a named file that is then mounted via a loopback interface...?" If the answer to that question is a categorical "not without a firmware upgrade" then I agree it's not worth the extra effort.


1ras wrote:

The cache with the biggest maximum cache size is used for caching. With other words if the cache on the USB stick isn't bigger as the one on the RAMdisk then the USB stick is not used for caching. Thinking about an implementation which could simultaneously cache on both is IMHO not worth the time.

This was something you hadn't made clear earlier. We agree then, that dealing with two caches is not worth the effort. But if you plan to just use whichever cache is biggest, you still have the same issue that the USB cache could be as small as 2.01MB but still get used.

My concern is that you could end up raising the widget limit based on the existence of a larger cache when it turns out the larger cache has barely any more space, ultimately resulting in a higher burden on the server than if there was no extra cache.


1ras wrote:

When the cache is detected as full then no more widgets are downloaded from the Chumby server and the Chumby is skipping any widgets which are not in the cache. There should be of course a small message e.g. displayed on the CP to make the user aware why some widgets are not displayed. E.g. the space where the CP shows the widget preview image can be used for this message.

My understanding is that this is *not* what's happening currently. At the moment the CC will cycle through up to 50 widgets, and this limit was set based on an average widget being 40K and the cache being 2MB. If, however, your channel uses all the largest widgets then you will still cycle through 50, it's just that older ones will be thrown out to make room for the next one.

The current algorithm is *not* "keep loading widgets 'til the cache is full, then skip any others", as I understand it.

My sole concern about just using a directory for the cache is that you could raise the limit based on the existence of a cache directory, only to find that it hits the servers hard because the cache is barely any larger than the built-in one. That's the only reason I speculated about an approach that creates a fixed size cache; so that Duane could be certain that the presence of the cache makes it safe to remove the 50 widget limit for that device.


1ras wrote:

There is always a cache on the CC which is the one on the RAMdisk. If you don't add a sufficient cache which is bigger as the existing one then you have not added a cache and the RAMdisk is used for caching as usual.

If the cache is full then the Chumby stops downloading new widgets and only plays the one available in the cache. To make the user aware the CP displays a suitable message.

As I said, I don't think that's what's happening at the moment. Making it work like that would further restrict the device for CC users with no extra cache - a few "licky dog" sized widgets and there's not much room for anything else. There would be no indication on the website as to what your limit is, potentially leading to users adding widgets that will never be seen.

Sticking with the current caching algorithm and simply raising the limit for CC users with a cache is another option. But if it's possible for the cache to be only marginally larger than the existing one, you could end up with an extra burden on the server end. With no way for the web end to know about your CCs cache size, it's not possible to display a cut-off at the point of setting up the channels.


I'm not particularly attached to the notion of a loopback device. I only mentioned it as a way to have a cache of known size, so that the web interface can show something sensible. That's where all this started: would a USB cache allow the limit to be raised? The answer has to be "yes, but only if the cache is large enough". My concern is simply that a folder on a USB stick full of other files may not be "large enough".


I'll finish by quoting Duane's comments from the start of this thread about the current caching algorithm:

Duane wrote:

Yes, a round robin of channels would help - by the very nature of this simple cache, once you go past the limit, *every* app gets reloaded, *every* time.  Using a round robin, it would at least cache for the duration a given channel is showing.

One thing I'm looking into is using a better strategy for the caching on these devices.  Right now, its the least recent app that gets purged to make room, but there's very likely a better algorithm.

41 (edited by 1ras 2014-07-22 01:53:38)

Re: 50 Widgets per channel limit?

Xav wrote:
1ras wrote:

And now we are talking about a firmware upgrade instead of simply delivering an updated controlpanel.swf.

As it's just a simple shell script, I can think of at least a couple of ways that it could be deployed without requiring a full firmware upgrade. That said, I started this whole point with a question: "Would it be possible to have a named file that is then mounted via a loopback interface...?" If the answer to that question is a categorical "not without a firmware upgrade" then I agree it's not worth the extra effort.

Not with a loop-mount but with a bind-mount, which I would consider a workaround, not a solution. Anyway, who should bind-mount your script over an existing one? The CP? The CP is started when the USB stick is already mounted, that's too late for your udev script.

Xav wrote:
1ras wrote:

The cache with the biggest maximum cache size is used for caching. With other words if the cache on the USB stick isn't bigger as the one on the RAMdisk then the USB stick is not used for caching. Thinking about an implementation which could simultaneously cache on both is IMHO not worth the time.

This was something you hadn't made clear earlier. We agree then, that dealing with two caches is not worth the effort. But if you plan to just use whichever cache is biggest, you still have the same issue that the USB cache could be as small as 2.01MB but still get used.

My concern is that you could end up raising the widget limit based on the existence of a larger cache when it turns out the larger cache has barely any more space, ultimately resulting in a higher burden on the server than if there was no extra cache.

I'm raising the widget limit for everybody independent if he has a USB cache in place or not. I do not even know at web interface level if he has one in place or not. That's not an issue as widgets which can't be cached are skipped and user get a small message on the CP. In no way I am causing high traffic load on the Chumby server.

Xav wrote:
1ras wrote:

When the cache is detected as full then no more widgets are downloaded from the Chumby server and the Chumby is skipping any widgets which are not in the cache. There should be of course a small message e.g. displayed on the CP to make the user aware why some widgets are not displayed. E.g. the space where the CP shows the widget preview image can be used for this message.

My understanding is that this is *not* what's happening currently. At the moment the CC will cycle through up to 50 widgets, and this limit was set based on an average widget being 40K and the cache being 2MB. If, however, your channel uses all the largest widgets then you will still cycle through 50, it's just that older ones will be thrown out to make room for the next one.

The current algorithm is *not* "keep loading widgets 'til the cache is full, then skip any others", as I understand it.

My sole concern about just using a directory for the cache is that you could raise the limit based on the existence of a cache directory, only to find that it hits the servers hard because the cache is barely any larger than the built-in one. That's the only reason I speculated about an approach that creates a fixed size cache; so that Duane could be certain that the presence of the cache makes it safe to remove the 50 widget limit for that device.

The current implementation could cycle thru any amount of widgets as only the newest ones are stored in the cache. But this could cause high server load if someone has too many widgets in the channel. That's why Duane limited the amount of widgets to 50 per channel so that all widgets of a channel can be usually stored in the cache.

On Duane's implementation when the user just has the big widgets in the cache it may happen that the cache can't store 50 widgets and the server is still loaded with high traffic. This cannot happen with my implementation as I only use the amount of widgets which can be really stored in the cache. The user may end up with a smaller amount of visible widgets per channel but is not causing high traffic load to the Chumby server. This is an advantage as it lowers the traffic costs.

On the other side when the user has mainly small widgets in the channel or when he uses the same widget several times in the channel then with my implementation he may not even need a USB cache as the RAMdisk cache can already store a channel with e.g. 100 widgets.

Xav wrote:
1ras wrote:

There is always a cache on the CC which is the one on the RAMdisk. If you don't add a sufficient cache which is bigger as the existing one then you have not added a cache and the RAMdisk is used for caching as usual.

If the cache is full then the Chumby stops downloading new widgets and only plays the one available in the cache. To make the user aware the CP displays a suitable message.

As I said, I don't think that's what's happening at the moment.

Sure, I'm talking about my implementation and not about the current one.

Xav wrote:

Making it work like that would further restrict the device for CC users with no extra cache - a few "licky dog" sized widgets and there's not much room for anything else.

Which is an advantage as my implementation is never causing high traffic load to the Chumby server, independent what type of widgets you put into the channel.

Xav wrote:

There would be no indication on the website as to what your limit is, potentially leading to users adding widgets that will never be seen.

The user is notified by the CP when the available cache is insufficient for the amount of widgets he has in his channel. This simplifies the implementation as the whole logic is at the CP side.

Xav wrote:

Sticking with the current caching algorithm and simply raising the limit for CC users with a cache is another option. But if it's possible for the cache to be only marginally larger than the existing one, you could end up with an extra burden on the server end.

That's not possible with my implementation as widgets which can't be cached are skipped. Independent of the type of the cache (RAMdisk or USB stick). I only require one cache implementation and just change the storage location. This simplifies the implementation.

Xav wrote:

With no way for the web end to know about your CCs cache size, it's not possible to display a cut-off at the point of setting up the channels.

My implementation displays a notification on the CP only when the situation really happens. As explained above you could have 100 widgets in a channel and no USB stick plugged in and maybe not running into the situation that something get skipped. Since my implementation is fully dynamic you always get the best possible in your situation.

Xav wrote:

I'm not particularly attached to the notion of a loopback device. I only mentioned it as a way to have a cache of known size, so that the web interface can show something sensible. That's where all this started: would a USB cache allow the limit to be raised? The answer has to be "yes, but only if the cache is large enough". My concern is simply that a folder on a USB stick full of other files may not be "large enough".

My implementation would not need any limitation as the only limitation is the cache size which is fully under control of the user.

42

Re: 50 Widgets per channel limit?

1ras wrote:

On Duane's implementation when the user just has the big widgets in the cache it may happen that the cache can't store 50 widgets and the server is still loaded with high traffic. This cannot happen with my implementation as I only use the amount of widgets which can be really stored in the cache. The user may end up with a smaller amount of visible widgets per channel but is not causing high traffic load to the Chumby server. This is an advantage as it lowers the traffic costs.

Here we finally get to the core of the matter. I had incorrectly assumed that you were proposing something to extend the current functionality - i.e. by doing nothing existing CC users would still get 50 widgets per channel. Instead you are actually proposing to replace the existing algorithm with one that just fills the cache then issues a warning message and stops getting more widgets.

For some - possibly the majority - of CC users this might actually result in more than 50 widgets. But for some it could result in a reduction. That's what I was trying to avoid: Duane has been under fire enough for "breaking" the free service by using a different clock; I wouldn't want to be in his position when some paying customers are suddenly reduced from 50 widgets down to 30 and presented with a warning on their screens.


As far as I'm concerned any "solution" which doesn't *at least* preserve the status quo for those who don't implement the additional cache is not a practical solution at all.

43 (edited by 1ras 2014-07-22 06:37:35)

Re: 50 Widgets per channel limit?

Which status quo? The status quo for about a year was 1 widget ;-)

I understand the 50 widgets per channel limitation as a quick workaround to not further delay the go live date. I don't think it's intended that some users are allowed to cause high server load and some not just based on the type of widgets they have chosen. In my opinion it's just fair that all users are treated the same way and all are responsible the same way to keep the costs low.

You just have to invest a few bucks (about 4 Euro in Germany) to get a small USB stick e.g. from the Intenso Micro Line and plug it into your CC. The stick is very small and barely visible. My CC has a very similar one from almost the beginning which contains my own modifications. That can't be a show-stopper in my opinion.

44

Re: 50 Widgets per channel limit?

1ras wrote:

Which status quo? The status quo for about a year was 1 widget ;-)

True, but you only have to look at the reaction of some non-subscribers to the fact that the service they're not paying for changed the 1 widget they were using, to realise that any perceived reduction in service - especially one they're paying for - is likely to result in complaints.


1ras wrote:

In my opinion it's just fair that all users are treated the same way and all are responsible the same way to keep the costs low.

I agree with you in principle, but that's not the way the service has run in the past, nor the way it currently runs in its resurrected form. I actually think the 50 widget limit is too lenient, but that's what Duane chose, and any approach that potentially reduces that number risks a backlash from some subscribers who might feel that they've been subjected to a "bait & switch" scam.


1ras wrote:

You just have to invest a few bucks (about 4 Euro in Germany) to get a small USB stick. That can't be a show-stopper in my opinion.

It's not about the cost. Some users won't add a USB stick to their CC regardless of price. A warning appearing on their control panel that they can't have all their widgets unless they buy extra hardware - no matter how cheap it is - will be seen as extortion by some. "Those are some very nice widgets you've got there, it would be a shame if you couldn't see them all. Just spend another 4 Euro to get them back..."


I also think the UI you're proposing for the web interface would be a bad idea (i.e. removing the limit entirely). Consider a CC user owner who subscribes to the service, but never comes anywhere near the forums: knowing that there are over 1,000 widgets to choose from he spends an hour or two trawling through them all, adding loads of them to his newly resurrected Chumby. The only time he gets told that they won't all fit is once he looks at the control panel on the Chumby - by which time he's probably rather irate that the web interface let him waste time adding far more widgets than he could use. The current system isn't ideal, but at least there's a clear indication of which widgets will appear on your Chumby and which ones won't.


In an ideal world every CC user would add a USB cache to their Chumby and would ensure that they manage the files on it so that there's always enough free space, they could add all the widgets they want via the web interface, and they would never see the warning on the control panel. But we don't live in an ideal world.

Re: 50 Widgets per channel limit?

Xav wrote:
1ras wrote:

Which status quo? The status quo for about a year was 1 widget ;-)

True, but you only have to look at the reaction of some non-subscribers to the fact that the service they're not paying for changed the 1 widget they were using, to realise that any perceived reduction in service - especially one they're paying for - is likely to result in complaints.

It seams that you already learned that it's impossible to make it right for everybody. That's an important lesson probably everyone come across earlier or later in his life. When do you start utilizing the knowledge you learned? ;-) Might sound arrogant but it isn't meant. So much time is lost by trying to make it right for a very few people without any advantage for the majority. At the end of the day you still fail because there is always one who dislike what you did. I would just try it and see what happens and not worry about what others may think before even tried.

But of course, you could also just ignore efficient caching for channels with 50 widgets or less like it is done now and only start with efficient caching for bigger channels. I would personally not do it if I have the efficient caching algorithm in place, but I'm also not the one who have to make the decision and pay the bills.

Xav wrote:

I also think the UI you're proposing for the web interface would be a bad idea (i.e. removing the limit entirely). Consider a CC user owner who subscribes to the service, but never comes anywhere near the forums: knowing that there are over 1,000 widgets to choose from he spends an hour or two trawling through them all, adding loads of them to his newly resurrected Chumby. The only time he gets told that they won't all fit is once he looks at the control panel on the Chumby - by which time he's probably rather irate that the web interface let him waste time adding far more widgets than he could use. The current system isn't ideal, but at least there's a clear indication of which widgets will appear on your Chumby and which ones won't.

Duane wrote earlier in this thread that Chumby had a 100 widget limit. Assuming that we could further rise the limit this would be an advantage nobody had before.

46

Re: 50 Widgets per channel limit?

1ras wrote:

Might sound arrogant but it isn't meant.

No, it doesn't sound arrogant. Just patronising, sarcastic and unnecessary. As the thread has descended to this level, and stopped being productive quite some time ago, this will likely be my last post on the subject unless there is a genuinely useful contribution made by someone.

1ras wrote:

I would just try it and see what happens and not worry about what others may think before even tried.

Whereas I would choose to gather some data to make an informed decision, rather than "just try it and see what happens". Without adequate metrics we don't know if your proposed change would reduce the number of widgets for 1 user or 100. I suspect the figure is quite low, but without inside knowledge there's no way to be sure. As I said 10 posts ago: "[let's] leave Duane to do whatever he feels is best - a decision that could easily be based on a whole load of variables that we know nothing about anyway".


1ras wrote:

But of course, you could also just ignore efficient caching for channels with 50 widgets or less like it is done now and only start with efficient caching for bigger channels.

I think that's a better suggestion from the perspective of not removing existing functionality from paying customers. But it still leaves the problem of an unclear web UI as there can be no indication of the number of widgets you *can* have (considering the possibility of a 2.01MB USB cache). I doubt this is a solvable problem without a fixed - or at least a minimum - cache size, though.

Do note, however, that "efficient" is a subjective term. Yes, your suggestion might be a more "efficient" use of server bandwidth, but a CC user might consider the algorithm that lets him have 50 widgets, regardless of their size, to be a more "efficient" use of their Chumby. There are two ends to the connection, and you need to consider both of them.



Anyway, reply if you wish but it's unlikely I'll respond. I see little point in continuing a conversation that's just going round in circles and has descended into sarcasm. Neither of us has the figures to know how many Classics are connecting to the service, how many widgets they typically use, or how much of an effect it's having on the bandwidth and costs. Without that information any further speculation seems pointless to me.

Re: 50 Widgets per channel limit?

Yeah, it might be time to stop this thread, since there's nothing really new coming from it.

It's been very interesting reading, and I've made a decision, which I'll be incorporating into the CP and site.

There is no perfect solution to the problem, and no matter what I do there will be some issue.

I did a random sampling of the database to find out how much of an issue this really is.

It turns out that the number of channels with 50 widgets is quite small, 0.05%, so the number of people affected by the current limit is just a handful.  I haven't correlated these large channels with the presence of CCs.

The most common channel size is 2 widgets (10.4%), followed by 31 widgets (6.34%), 3 widgets (5.95%) and 10 widgets (5.46%).  There are very few channels above 40 widgets.

The graph is pretty much an inverse distribution, with little bumps around 16 and 32 widgets.

Checking the old database, the number of channels with more than 50 widgets that weren't the result of widget subscriptions was similarly small.

Re: 50 Widgets per channel limit?

16 and 32 is interesting.  Do you see a small bump for all multiples of 4 in the distribution since there are 4 columns when setting up apps on the web interface?

Linux Guy - Occasional Chumby Hacker

Re: 50 Widgets per channel limit?

Xav wrote:

No, it doesn't sound arrogant. Just patronising, sarcastic and unnecessary.

It's not meant patronising or sarcastic. There was a time where I was also thinking that I have to make it right for everyone. It took me some time to realize that it's not possible. I would therefore name it experience of life. What I'm saying is: Listen to others, take the good stuff with you but don't overstate the negative part.


@Duane: The numbers are very interesting, especially the most common channel size of 2 widgets is surprising.

Re: 50 Widgets per channel limit?

Materdaddy wrote:

16 and 32 is interesting.  Do you see a small bump for all multiples of 4 in the distribution since there are 4 columns when setting up apps on the web interface?

I don't see anything particularly interesting around the other multiples of four.

Note that some of these numbers are probably driven by channels that were automatically created when the accounts are created or certain types of devices are added to the account.