1 Topic by ChumbyLurker (edited by ChumbyLurker 2009-03-11 08:25:35)

  • Registered: 2008-06-27
  • Posts: 518

Topic: Fix for drifting clock

We've come up with a possible fix for the drifting clock problem, where Chumby starts losing time.  You'll need to install a new kernel, which is just like installing an update, by following these steps:

1) Make sure you're running the latest firmware, 1.6.0.
2) Download http://files.chumby.com/resources/clock-skew-fix-02.zip
3) Extract it to the root of a USB drive that's formatted as FAT32.  You should see a directory called "update2" on your USB drive.
4) Turn off your chumby.
5) Insert the drive into your Chumby, and make sure it's the only drive inserted.
6) While holding your finger on the screen, turn your Chumby back on.
7) Continue holding the screen until it says "Loading Special Options"
8) Once you get to the Special Options screen, select Install Updates, then Install From USB Flash Drive.

If you want to uninstall this fix, you'll want to perform a manual upgrade back to version 1.6.0.  Steps to do that are located on the Firmware 1.6 upgrade page, but the steps are identical to those above, except you'll want to download http://files.chumby.com/resources/chumby_fw_1-6-0.zip

What seems to be happening is that the system timer, TIMER1, is sometimes firing while interrupts are disabled, or fires multiple times but only gets called once.  In any case, I used a second timer (oddly enough TIMER3) to look for missed ticks, and call timer_tick().  If this fix does end up working, it'll be part of the next firmware build, but I've posted my modified time.c.

You can get a running count of how many missed ticks it's caught by looking at the file /proc/missed_ticks.  This is also the only way to tell if a Chumby is running this modified kernel.

EDIT: I seem to get a much more stable clock if I don't subtract LATCH/2 from TIMER3.  I've updated the link to point to a second version of the fix that doesn't do this.  See if it turns out to be more accurate.

  • From: Kansas
  • Registered: 2008-01-28
  • Posts: 231

Re: Fix for drifting clock

How will this change or installation of the next firmware build affect those of us whose Chumby's haven't lost and don't lose time? I am concerned about this and the Pandora issue, because mine has been fine all along (time and Pandora) yet, when others report problems, a chumbian comes on to say they have located the issue or problem and are taking steps to fix it. Meanwhile, I am fine, so I am slightly concerned that changes will affect me adversely. Not that it shouldn't be fixed for those who are having issues, but I am just wondering why I am not and others are. So, basically, what's different from mine to theirs? Thanks, B

  • Duane
  • Duane
  • Blue Octy
  • Offline
  • From: San Diego, CA
  • Registered: 2006-08-24
  • Posts: 8,211

Re: Fix for drifting clock

It will not affect future updates - devices with this fix installed will update normally.

Duane's Website

  • Registered: 2008-10-10
  • Posts: 56

Re: Fix for drifting clock

Should everyone apply this fix then?

Thanks

  • Duane
  • Duane
  • Blue Octy
  • Offline
  • From: San Diego, CA
  • Registered: 2006-08-24
  • Posts: 8,211

Re: Fix for drifting clock

No - this is a test for those people that are having the problem in order to confirm the fix.  If you're not experiencing the clock drift (which is most people), then there's no reason to install this since we won't know if it made any difference.

The drifting clock only happens with a particular network configuration, which is why so few people have seen it, and why it was so tough to uncover.

Duane's Website

  • From: Kansas
  • Registered: 2008-01-28
  • Posts: 231

Re: Fix for drifting clock

Duane wrote:

The drifting clock only happens with a particular network configuration, which is why so few people have seen it, and why it was so tough to uncover.

See, I couldn't tell from the posts that that was the case, and I thought mine was weird since it worked so well. Glad to know that it is most of who are doing fine. I'm glad I don't have to do anything differently because I am pretty pleased with the way it is working for me now and for the past year and a half. 24/7/

  • Registered: 2008-10-30
  • Posts: 8

Re: Fix for drifting clock

OK, I've installed the fix and switched back to WPA2.  I'll let you know tomorrow!

  • Duane
  • Duane
  • Blue Octy
  • Offline
  • From: San Diego, CA
  • Registered: 2006-08-24
  • Posts: 8,211

Re: Fix for drifting clock

People generally don't post when "everything's fine", so it's often an illusion that some problem is much more common than it actually is.  Based on this forum and other Customer Service feedback, I think we have maybe a dozen or so folks that have been vocal on this issue, which is a extremely small fraction of the total installed base of units.  There may be others that are having the problem, but don't care (perhaps they use their chumby for non-alarm purposes), or aren't using their chumbys in a manner that exacerbates the problem.

If it had been a more common problem, we would have been able to diagnose it much sooner.  As it was, the cause was so far removed from the symptom that it was hard to come up with the connection.  Once a few users had diagnosed that certain network configurations affected it, we were able to reproduce it.

Duane's Website

  • Registered: 2008-10-30
  • Posts: 8

Re: Fix for drifting clock

Duane wrote:

People generally don't post when "everything's fine", so it's often an illusion that some problem is much more common than it actually is.  Based on this forum and other Customer Service feedback, I think we have maybe a dozen or so folks that have been vocal on this issue, which is a extremely small fraction of the total installed base of units.  There may be others that are having the problem, but don't care (perhaps they use their chumby for non-alarm purposes), or aren't using their chumbys in a manner that exacerbates the problem.

If it had been a more common problem, we would have been able to diagnose it much sooner.  As it was, the cause was so far removed from the symptom that it was hard to come up with the connection.  Once a few users had diagnosed that certain network configurations affected it, we were able to reproduce it.

It's a pretty obscure problem to be sure.  Good for you guys for tracking it down.  I'll post my results either way.

  • From: Kansas
  • Registered: 2008-01-28
  • Posts: 231

Re: Fix for drifting clock

And if MY time STARTS drifting, you'll all be in big trouble! :-)

  • From: Austria
  • Registered: 2007-10-12
  • Posts: 179

Re: Fix for drifting clock

Thanks for this fix - that softens the problem very well. Before the fix I had a loss of about 5 secs in 10 minutes, now I need an hour to lose the same amount of time. After an hour /proc/missed_ticks reports "Corrected 1285 missed ticks".

  • Registered: 2008-06-27
  • Posts: 518

Re: Fix for drifting clock

Thank you for trying it out, Christian.  Could you try version 2 that I just posted, and see if it helps any more?

13 Reply by Christian (edited by Christian 2009-03-11 10:55:54)

  • From: Austria
  • Registered: 2007-10-12
  • Posts: 179

Re: Fix for drifting clock

Yes, that works better. After one and a half hour I have no measurable time loss. Seems to fix the problem completely, at least for me wink

  • From: Brooklyn, NY (Brian)
  • Registered: 2008-06-21
  • Posts: 176

Re: Fix for drifting clock

Christian wrote:

Thanks for this fix - that softens the problem very well. Before the fix I had a loss of about 5 secs in 10 minutes, now I need an hour to lose the same amount of time. After an hour /proc/missed_ticks reports "Corrected 1285 missed ticks".

Just wondering. Is ntpd or something similar running on the chumby? If it was, it could be configured to accept a time broadcast from an ntp server on a home network every so often. I guess 30 seconds an hour was quite severe though....

I have a Reciva radio that only picks up the time from Reciva's server once each time the radio is turned on (off standby). So if it is running or off for several days then the clock will creep up over 1 minute per day. However it is configured to accept time broadcast so I just have my PC ping it with an ntp broadcast twice a day. Even when the radio is off 'Standby' the clock will update. I realize though that a chumby is either on or off. No real middle ground, right?

How often does a chumby pick up the time from the company server when it is on??

Brian, #1 Joan Jett Fan
  • Duane
  • Duane
  • Blue Octy
  • Offline
  • From: San Diego, CA
  • Registered: 2006-08-24
  • Posts: 8,211

Re: Fix for drifting clock

The chumby uses rdate to sync with network time on power up and every 24 hours thereafter (actually, at 3 am) - however, this particular drift was large enough that the network time sync at this frequency wasn't very helpful.  One of the workarounds was to alter the cron job to run more frequently.

Duane's Website

  • Registered: 2008-10-30
  • Posts: 8

Re: Fix for drifting clock

The fix worked perfectly for me.  Thanks!

  • Duane
  • Duane
  • Blue Octy
  • Offline
  • From: San Diego, CA
  • Registered: 2006-08-24
  • Posts: 8,211

Re: Fix for drifting clock

Note that this patch is built into firmware versions 1.7 and later.  We consider this issue closed.

Duane's Website

  • Registered: 2008-12-02
  • Posts: 5

Re: Fix for drifting clock

Has this regressed by any chance? Our Classic started losing time a few weeks ago. I was futzing with the wifi during this time. 

Today, after reading many of the time related threads, I put Chumby onto a wired ethernet adaptor, and how time seems to be rock-solid.

I did the upgrade to 1.7 firmware (was on 1.6).

WiFi was config'd for WPA2/Personal.  I'm going to switch it back to WPA/WPA2 and after I confirm that Wired is solid, switch back to WiFi to see what happens.

  • andom
  • New member
  • Offline
  • Registered: 2009-06-20
  • Posts: 3

Re: Fix for drifting clock

travisd wrote:

Has this regressed by any chance? Our Classic started losing time a few weeks ago. I was futzing with the wifi during this time. 

Today, after reading many of the time related threads, I put Chumby onto a wired ethernet adaptor, and how time seems to be rock-solid.

I did the upgrade to 1.7 firmware (was on 1.6).

WiFi was config'd for WPA2/Personal.  I'm going to switch it back to WPA/WPA2 and after I confirm that Wired is solid, switch back to WiFi to see what happens.


I tried different things and ended up changing the time sync interval in the crontab of root from running once a day to every 5 minutes and I also changed the script doing the sync :
chumby# crontab -e

#* * * * * /usr/chumby/scripts/sync_time.sh
*/5 * * * * /psp/myscripts/fix_time.sh

here is the script:
chumby# vi /psp/myscripts/fix_time.sh

#!/bin/sh
# ntpdate -- set the date and time via NTP

if ps | grep -v grep | grep -q ntpd
then
      killall ntpd
      fi

#ntpdate at.pool.ntp.org
ntpdate time.euro.apple.com
----------------------------------------------
this script just kills the ntp demon if it is running (was originally like this) and then uses ntpdate to set the correct time.
now the time is always correct. Before it went slow one hour within 24 hours on both of my chumbys.

  • geoff
  • New member
  • Offline
  • Registered: 2015-09-09
  • Posts: 7

Re: Fix for drifting clock

A recent router change means I had to switch from Mac Address wireless security to WPA. Both my Chumbys now consistently lose time.

The above looks like a surefire way of fixing the issue but as a non-programmer I have no idea how to go about making the edits/changes needed to implement. I can just about SSH into my Chumbys but that's all. A little steer in the right direction would be much appreciated. TIA.

  • Duane
  • Duane
  • Blue Octy
  • Offline
  • From: San Diego, CA
  • Registered: 2006-08-24
  • Posts: 8,211

Re: Fix for drifting clock

geoff - what type of chumby is it?

I have a Control Panel for the 3.5" devices in testing that seems to address this issue, and if you're interested, I can get it to you to verify that it fixes it for you.

Duane's Website

22 Reply by geoff (edited by geoff 2016-09-06 10:45:53)

  • geoff
  • New member
  • Offline
  • Registered: 2015-09-09
  • Posts: 7

Re: Fix for drifting clock

Duane wrote:

geoff - what type of chumby is it?

Thanks for the reply Duane. Stupid of me not to state which model. It's a Classic with the padded leather wrapping. Would love to try the new version BTW.

  • From: Toronto, Ontario
  • Registered: 2014-03-12
  • Posts: 36

Re: Fix for drifting clock

Hi I too have a classic with this problem. Would love to know how to fix.