Okay, as Diamaunt wants us to continue, I'll bite.
1ras wrote:[I am not a flash developer but I guess flash is - like every other programing language - able to check the free space of a device before storing something on it.
I never claimed it couldn't.
1ras wrote:I suspect it isn't, as properly handling a loop-mounted file system inside an image stored on a removable media isn't such a straightforward task as you may think. While the USB stick is auto-mounted when removed and added you have to detect this situation and make sure that you remount your image as well.
This isn't as tricky as you're suggesting. A simple shell script that fires on USB insert/removal can trivially check for a file of a known name on the device, then attempt to mount it as a loopback device - assuming the Chumby has the necessary kernel modules and userspace tools. From there it's just a case of setting a flag to indicate to the control panel whether or not the USB cache is available.
1ras wrote:This isn't true as a FAT file system (and that's the only one the CC supports on the USB stick) is case-insensitive on Linux as well.
I'll gladly concede this point - I'd forgotten we were dealing with FAT. That said, this does limit your cache to being FAT also, whereas using a loopback file allows for other filesystems. It may be that the existing cache code assumes EXT3, for example, so allowing for FAT might require additional changes. This is speculative, however, and unlikely to be a major issue, so I'm not using this as an argument either way, but just noting that this may need to be considered.
Let's talk algorithms, and maybe you'll see why I think a loopback device of known size is a simpler thing to implement from a coding perspective. In these examples I'll refer to the existing 2MB cache as the "internal" cache, and the cache on the USB stick as the "external" cache:
1) Supplied file mounted as a loopback device
* A shell script, or executable, runs at boot, and on insertion/removal of a USB stick. This initially sets two variables to 0: a boolean (USE_EXTERNAL_CACHE) and an integer (EXTERNAL_CACHE_ERROR).
* If a file of the correct name is present on the USB stick, check its size and try to mount it as a loopback device.
* If it cannot be mounted, set EXTERNAL_CACHE_ERROR to a non-zero error number and exit.
* If it can be mounted, run some rudimentary checks on its structure - mainly that the mounted filesystem is of a minimum defined size, and optionally that it's the right format. Other tests could be conceived if necessary. If any fail, set EXTERNAL_CACHE_ERROR to a non-zero error number and exit.
* Assuming everything was okay, set USE_EXTERNAL_CACHE to 1.
* NOTE: All the above happens at the Linux level, and is asynchronous to the control panel code.
* The control panel is ideally modified to display a suitable error if EXTERNAL_CACHE_ERROR is non-zero.
* The caching code is modified to check for the presence and value of USE_EXTERNAL_CACHE. If it's absent or 0 then the current code applies - a 50 widget limit using the internal cache.
* If USE_EXTERNAL_CACHE is 1 then the path to the cache is changed to use the loopback device. All the other cache code remains the same, it's just the location of the cache that is different.
* In this scenario the internal cache is used as a fallback option, but if the external cache is present that is used exclusively. Only one cache is used at a time.
* There's no change to the cache algorithm itself, although the external cache is large enough to hold every widget, so in practice nothing will ever be thrown out of that cache.
2) Use folder on the USB stick.
* At programming time, the cache is an indeterminate size. It could be 1 byte, 1MB or 1GB.
* In case it's only 1 byte, we can't use the external cache exclusively. We therefore have to consider the external cache to be an *addition* to the internal cache, so the caching code needs to be modified to deal with two caches.
* If it's 1MB then the caching algorithm, when it needs to cache a widget, has to check the free space on the internal cache and put it there if it can, otherwise it needs to check the external cache and put it there if it can.
* If it won't fit in either cache, the code needs to start removing items from the caches until one of them has enough space for the new widget.
Let's consider a worst-case scenario: a 1MB cache filled with 40K widgets, and the next one along is the licky dog 400K widget...
* Check both caches to see which widget is least recently used, and throw it away. We've gained 40K of space in one cache, still not enough for the licky dog.
* Repeat the above, throwing away the next least recently used. As this is a worst case scenario, we've ended up with our widgets perfectly interleaved between the two caches, so we throw the first out of external, the second out of internal, the third out of external, and so on.
* Repeat the above, throwing out widget from both caches until eventually one of them has 400K of free space. But to get there we've thrown out 19 widgets, 9 of which didn't need to be thrown out at all!
An alternative cache invalidation algorithm could reduce this issue, but that's introducing a lot more coding work, and throwing out tried and tested code to do so.
* In the best case scenario of 1GB of free space (actually anything >128MB) there's no need to throw anything out of the cache so this issue doesn't arise. This is also the most likely scenario, but you can't rely on that and have to write the code accordingly.
So with option (1) there's a small shell script to write, plus a minor change to the caching code. If the path to the cache is stored in a variable, it could be as simple as changing that one variable declaration into an "if" statement.
With option (2) the caching code has to be changed to check two caches when looking for a cached widget, to check two caches when trying to find space for a new one, to check two caches when trying to remove a widget to make space. As I've shown, this can lead to inefficiencies in the caching mechanism - or the need to replace the existing caching algorithm entirely. As a large part of this excercise is trying to reduce bandwidth use, a system that can throw away widgets unnecessarily is also a poor choice.
From a coding perspective, option (1) looks a lot simpler to me. It uses the existing caching code with minimal modifications, simply switching wholesale to use a larger cache if it's available. Option (2) requires changes to the caching code to deal with two simultaneous caches, and possibly a major change to the caching algorithm if inefficiencies are to be avoided.
1ras wrote:Xav wrote:[Referring to a cache directory on the USB stick]
With this approach there's no way for the web interface to know how many widgets will be cacheable, so there's no clean way to show which ones will or will not appear on the device.
Which simplifies the implementation as there are no modifications required on the web interface (beside a small configuration change to allow an higher amount of widgets per channle for all Chumby devices).
I don't understand. Are you really suggesting that the limit can be removed entirely, or at least raised across all devices? What about the CC users who never add a cache? How about those who add one, but only leave 1 byte of space? I can't see how this is meant to work in practice, so please explain if I've misunderstood what you mean.