Good catch! Let me do a bit of testing on it and merge it in for the next firmware release.
If you're seriously considering it, grab the new version I just put up. I added a lot more logic to handle rarer cases, and discovered a bug (it was assuming state 3->state 0 meant rotate up 2, when it could mean either direction).
All said, this is really a horrible hardware rotary decoder - I knew something was wrong when they had debounce circuitry in a gray code decoder . They should have either made it count on each state transition, or up and down across symmetric state edges (10->00 and 00->10 instead of 10->00 and 01->00)
edit: on futher testing it still seems to be tracking rather inconsistently.
edit2: I think the problem was the sampling was too slow. I turned up the sampling rate of the decoder hardware by a factor of 4 and it seems to be working better. It still seems to trend upwards a bit if you rotate the knob back and forth rapidly, but at least a given rotation results in about the same change in volume. New version uploaded.