Topic: Simulating a read/write filesystem on chumby one
Have you every wanted to try putting some software into /usr/bin but you don't want to modify your firmware until you know it works?
Here's the trick:
1) (insert a USB dongle, preferably formatted as ext3)
2) mkdir /mnt/usb/overlay
3) chmod 777 /mnt/usb/overlay
4) mkdir /mnt/overlay
5) mount -t unionfs -o dirs=/mnt/usb/overlay:/=ro unionfs /mnt/overlay
6) chroot /mnt/overlay
At this point, you'll find yourself in what appears to be a writable version of the chumby firmware - you can add, edit and remove files in directories that *appear* to be in the firmware. Your changes are actually recorded in /mnt/usb/overlay.
To get out, simply exit, then "umount /mnt/overlay"
There are some caveats - some parts of the filesystem such as /proc and /sys are not available. It's also possible to "bind" /dev to this filesystem with a few extra commands.
Thanks to help from ChumbyLurker, I've built an overlay that has a native gcc toolchain, with everything in the right place - no mucking with PATH, etc. - so one can actually compile and run C and C++ programs on the device itself.
Note that unionfs allows the chaining of more than two file systems - with some cleverness, you could create a read/write overlay on top of a read-only tools overlay, on top of the firmware, use the tools to build something, then remount without the tools overlay. This would let you use the results of the build without polluting it with the build tools. Pretty cool.
None of this is black magic - it's a pretty common trick with Live CD distributions of Linux.