It's actually not an issue of buffering, it's more that these videos are optimized for multiGHz desktop-class processors, or embedded devices with hardware-based decoders or DSPs.  We have neither.

We're doing some optimization work to improve this over time, however, there's only so much you can do with such a lightweight processor.