Not very hard if you know what you're doing. The Chumby's open source software is enough to get started, though you might want to look at graphical toolkits to speed up your adventure.

On my Katamari Chumby, I have played around with several different little graphical diddies. The first graphical program you can look at is simply drawing something to the Linux kernel framebuffer ([Katamari] Chumby creates /dev/fb and is able to be drawn to after you kill the Flash player). Next, you can look at different graphical toolkits (and there are hundreds of them to choose from). Just remember that you're running on a 350MHz computer with no hardware graphics and no hardware floating point and program accordingly.