NTP is wrong by almost 3 minutes.


  • Banned

    I've noticed that my NTP is wrong recently, by a lot, nearly 3 minutes.

    I've tried using the pfsense pool NTP servers and the NIST servers. I'm comparing the time from pfsense to the time on both my cell phone, and also the times when I go to the NIST website, and the US Navy time website. NIST, US Navy, and my cell phone all agree, and my pfsense box and desktop (connected to pfsense) both agree, but pfsense is almost three minutes faster than the US Navy, NIST & my cellphone.

    I've tried restarting the service, just form looking at the RRD graphs is appears that the statistics have tightened up somewhat over the last month.

    Any suggestions?


  • Rebel Alliance Global Moderator

    So what does pfsense show for its status on its ntp?

    status, ntp - should show you what pfsense is using.  Does it show reach of 377?  Post up this screenshot.

    Keep in mind ntp doesn't instantly change a devices clock.  It adjusts the time of the device to sync up with its timesource and once in sync to stay there.  If you graphing ntp stuff - lets look at just what its reporting for offset for a say the last day.. You can undo all the other stuff and just look at the offset.. You can see if its getting closer or just all over the board, etc..



  • Banned

    OK thanks, here are the graphs and status. I'm also looking at the NTP widget on my pfsense dashboard, it does match up with my desktop clock and they have been for well over 24 hours since I first noticed this.





  • Rebel Alliance Global Moderator

    Other than not having 377 on your active server.. That all looks up and up.. Means your dropping packets to that server.. Also that your poll time is higher then your 256 setting.. Would point to having trouble talking to that server.

    376 points to you have dropped the last packet, you will see reach cycle down to 177 until that no answer cycles through the 8 bit buffer..

    So your saying this time is 3 minutes off??  Your ref ID is nist and your syncing off a stratum 1.. I find it very unlikely that time is off by 3 minutes from these servers..  Your currently showing offset of .033 millisecons.. or .000033 Seconds..

    Where are you think pfsense time is off?  The web gui status page?  The time shown there??

    edit: One thing I would suggest is you find some closer servers to sync off of..  Your delay is 70 some ms..  Look here for stratum 1's that are closer to you..  Find something in your state or region, etc.

    http://support.ntp.org/bin/view/Servers/StratumOneTimeServers

    Pay attention to their rules - some don't like to be queried unless your providing time for large number of clients, etc.  There is also the stratum 2 list
    http://support.ntp.org/bin/view/Servers/StratumTwoTimeServers

    Which are also fine to use..


  • Banned

    Well this is what I noticed. I actually noticed it because my desktops time was off from my cellphones time, so then I went into pfsense and compared pfsense time to government time servers.

    Maybe this isn't a valid comparison, but I don't see why this much of a difference would be showing?



  • Rebel Alliance Global Moderator

    is your ntp widget counting.. Or is it stuck there?

    I just added the widget, and then called up that nist page in another window and put them side by side and they show exact same time

    I assume that page is pulling time from the ntp server and pfsense time.. But its a browser thing - maybe its pulling time from your local machine?  Clearly what you posted is that the time is insync with its source only off by a few ms..

    Off the top not sure would could account for the difference other than browser stuck not refreshing, etc.?

    If you want to do a live check to see if pfsense clock is off you could use ntpdate in test mode with the -d and point it to the IP of any ntp server you want to check it against

    example

    
    [2.3.2-RELEASE][root@pfsense.local.lan]/root: ntpdate -d 192.168.9.32
    21 Dec 14:06:02 ntpdate[45505]: ntpdate 4.2.8p8@1.3265-o Fri Jun 10 15:40:54 UTC 2016 (1)
    transmit(192.168.9.32)
    receive(192.168.9.32)
    transmit(192.168.9.32)
    receive(192.168.9.32)
    transmit(192.168.9.32)
    receive(192.168.9.32)
    transmit(192.168.9.32)
    receive(192.168.9.32)
    server 192.168.9.32, port 123
    stratum 1, precision -20, leap 00, trust 000
    refid [PPS], delay 0.02609, dispersion 0.00006
    transmitted 4, in filter 4
    reference time:    dc055f21.f87dc034  Wed, Dec 21 2016 14:05:53.970
    originate timestamp: dc055f30.d2941d7d  Wed, Dec 21 2016 14:06:08.822
    transmit timestamp:  dc055f30.d28388c7  Wed, Dec 21 2016 14:06:08.822
    filter delay:  0.02675  0.02632  0.02617  0.02609
             0.00000  0.00000  0.00000  0.00000
    filter offset: -0.00018 -0.00018 -0.00009 -0.00007
             0.000000 0.000000 0.000000 0.000000
    delay 0.02609, dispersion 0.00006
    offset -0.000070
    
    21 Dec 14:06:08 ntpdate[45505]: adjust time server 192.168.9.32 offset -0.000070 sec
    [2.3.2-RELEASE][root@pfsense.local.lan]/root:
    
    

    21 Dec 14:06:08 ntpdate[45505]: adjust time server 192.168.9.32 offset -0.000070 sec

    so you see there my pfsense says its off by 70 us from my ntp server..



  • Banned

    ok thanks ill run that check, and no it isn't stuck, all three of the times pulled up in webpages on that screenshot were counting seconds.


  • Banned

    I ran ti twice, here is the output, I noticed at the end of the first run, after it gives me the offset time it says "325 sec", whats taht about?

    ntpdate -d time.nist.gov
    21 Dec 12:27:10 ntpdate[90838]: ntpdate 4.2.8p8@1.3265-o Tue Jul 19 16:25:10 UTC                                                                                                                                                              2016 (1)
    transmit(128.138.141.172)
    receive(128.138.141.172)
    transmit(128.138.141.172)
    receive(128.138.141.172)
    transmit(128.138.141.172)
    receive(128.138.141.172)
    transmit(128.138.141.172)
    receive(128.138.141.172)
    server 128.138.141.172, port 123
    stratum 1, precision -29, leap 00, trust 000
    refid [NIST], delay 0.09991, dispersion 0.00021
    transmitted 4, in filter 4
    reference time:    dc0563c9.00000000  Wed, Dec 21 2016 12:25:45.000
    originate timestamp: dc056424.63b52a21  Wed, Dec 21 2016 12:27:16.389
    transmit timestamp:  dc056424.5befb402  Wed, Dec 21 2016 12:27:16.359
    filter delay:  0.09991  0.09993  0.10071  0.10135
             0.00000  0.00000  0.00000  0.00000
    filter offset: -0.00832 -0.00838 -0.00796 -0.00751
             0.000000 0.000000 0.000000 0.000000
    delay 0.09991, dispersion 0.00021
    offset -0.008325
    
    21 Dec 12:27:16 ntpdate[90838]: adjust time server 128.138.141.172 offset -0.008   sec  325 sec  
    ``` 
    
    

    ntpdate -d time.nist.gov
    21 Dec 12:32:39 ntpdate[16340]: ntpdate 4.2.8p8@1.3265-o Tue Jul 19 16:25:10 UTC                                                                                                                                                              2016 (1)
    transmit(129.6.15.30)
    receive(129.6.15.30)
    transmit(129.6.15.30)
    transmit(129.6.15.30)
    receive(129.6.15.30)
    transmit(129.6.15.30)
    transmit(129.6.15.30)
    server 129.6.15.30, port 123
    stratum 1, precision -29, leap 00, trust 000
    refid [ACTS], delay 0.15990, dispersion 24.00020
    transmitted 4, in filter 4
    reference time:    dc056561.1ebacc2a  Wed, Dec 21 2016 12:32:33.120
    originate timestamp: dc05656b.6a52fae8  Wed, Dec 21 2016 12:32:43.415
    transmit timestamp:  dc05656d.512dbb90  Wed, Dec 21 2016 12:32:45.317
    filter delay:  0.15997  0.00000  0.15990  0.00000
            0.00000  0.00000  0.00000  0.00000
    filter offset: 0.030677 0.000000 0.031073 0.000000
            0.000000 0.000000 0.000000 0.000000
    delay 0.15990, dispersion 24.00020
    offset 0.031073

    21 Dec 12:32:47 ntpdate[16340]: adjust time server 129.6.15.30 offset 0.031073 s                                                                                                                                                            ec


  • Rebel Alliance Global Moderator

    Not sure, don't think ever seen anything like that before.

    You can see it sends 4 queries, and shows your offset from all 4 off them

    filter offset: -0.00832 -0.00838 -0.00796 -0.00751

    So not sure where that odd 325 display would come from??


  • Banned

    Any guesses as to why I am seeing such a big difference?



  • Grasping at straws, but is the hardware clock on the unit itself correct?


  • Banned

    I dont know, how could I check that?



  • If it's a PC of some type then I would look in the BIOS.  If it's something else then I don't know.



  • I hate to ask stupid questions, but there are some basics…

    1. Ignoring the NTP widget for the moment, what is shown as the current date/time in the System Information widget? Does it agree or disagree with www.time.gov?

    2. Have you cleared your browser's cache?



  • To eliminate all possible browser cache problems just go to    Diagnostics ->Command Prompt
    type "date" without quotes and wait for the answer.
    Or do the same in shell…

    I don't think it's related to any cache just because in logs we already have

    "21 Dec 12:27:16 ntpdate[90838]: adjust time server 128.138.141.172 offset -0.008  sec  325 sec"

    To check what hardware clocks you have run
    dmesg | grep timer

    and/or

    sysctl kern.timecounter

    You should see all possible timers and counters found and/or used by system. You can also change it to desired, tuning kern.timecounter.hardware.
    Read https://www.freebsd.org/doc/en/books/faq/troubleshoot.html


  • Banned

    Thanks all, the System Information Widget was also wrong.

    w0w's suggestion seems to have fixed system time.

    : dmesg | grep Timecounter
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Timecounter "HPET" frequency 14318180 Hz quality 950
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    Timecounters tick every 1.000 msec
    Timecounter "TSC-low" frequency 1546529202 Hz quality 1000
    
    
    : sysctl kern.timecounter.hardware=HPET
    kern.timecounter.hardware: TSC-low -> HPET
    
    

    So now System time matches time.gov, however NTP time remains the same although I don't know if this will fix itself given a little time? I restarted the service but the NTP offset remains the same.



  • I'm seeing the same issue with the NTP time being off by 5mins. Never noticed that till now. My system clock matches the NIST time though.

    Running 'date' from the command prompt returns the system time which is correct to the NIST time. The NTP widget is 5 mins off



  • I just polled the ntp server pool I'm using (pool.ntp.org) and it is off by the same 5 mins.

    sntp 0.pool.ntp.org

    When I poll nist it shows the same 5 minute discrepancy

    sntp 128.138.141.172



  • If anyone is interested I switched my ntp servers from ntp.org to the NIST servers and the discrepancy has resolved itself..


  • Banned

    That is interesting.

    I have been using time.nist.gov for a few days now and my NTP time is still off by a few minutes. System time now matches up but NTP has not resolved itself, I restarted the service over a day ago and it is still off by a few minutes.

    Anyone know what the root problem is here, especially now that I am apparently not the only one seeing some issues with time?



  • I'm sure you do, but just to check you do have your machines set to use your pfsense as the ntp time server?


  • Banned

    Some of them are and some aren't what I'm looking at is that the NTP time widget on my pfSense box is not syncing up with time.gov, even though it is using NIST for NTP. My system clock is synced up.

    Also, my machines that are using pfsense for time match the incorrect pfsense time widget exactly.

    My machines that are not using pfsense for time (cellphone, laptop) match time.gov exactly.



  • I just successfully tested time.nist.gov sync without any issue.
    Please provide the output of
    more /var/etc/ntpd.conf



  • I was having the same problem, running on 2.4 i would select to run NTP on the Lan side and it would never sync.  If i manually sync'ed then it would work.



  • Some of the NTP tools included with pfSense can be a big help in figuring out what is going wrong. The place to start is likely ntpq:

    [2.3.2-RELEASE][root@pfSense.home]/root: ntpq -p
         remote           refid      st t when poll reach   delay   offset  jitter
    ==============================================================================
    *pi-v1.home      .GPS0.           1 u   12   64  377    0.420   -0.034   0.008
    +hp400.home      172.16.0.4       2 u   44   64  377    0.412    5.662   0.782
    +ntp.cox.net     .GPS.            1 u   14   64  377   51.577    1.667   0.452
    

    The above shows that I am syncing with my Pi while my hp server and Cox Cable NTP servers are available as fallbacks.

    The first column is the sync status and is covered in the http://www.ntp.org/ documentation. If you are not selecting servers and syncing to one of them you have a starting place to look for your problem.

    I'd go look at at least the quick start at ntp.org and select either a pool or at least three servers to sync to before going on to look for other issues.



  • I finally got around to having a look at the code for the NTP widget. The "Server Time" field is created using some cute dynamic update code that plays games between the http server (aka pfSense) time and the http client browser time. The approach was likely state of the art… in 2003 when it was last updated. Seriously, the code lists compatibility as IE 4.x/5.0, Netscape 4.x/6.0, and Mozilla 1.0.

    The short version is that this code is badly need of replacing, and until it is replaced the Server Time field in the NTP widget should simply be ignored.

    One more "as time permits" activity. :)


  • Banned

    @dennypage:

    I finally got around to having a look at the code for the NTP widget. The "Server Time" field is created using some cute dynamic update code that plays games between the http server (aka pfSense) time and the http client browser time. The approach was likely state of the art… in 2003 when it was last updated. Seriously, the code lists compatibility as IE 4.x/5.0, Netscape 4.x/6.0, and Mozilla 1.0.

    The short version is that this code is badly need of replacing, and until it is replaced the Server Time field in the NTP widget should simply be ignored.

    One more "as time permits" activity. :)

    Wow, thank you so much for looking into this! I'm glad that this could resolve into something that will eventually be an improvement to pfSense!


  • Banned

    Craptastic JS gone from the widget: https://github.com/pfsense/pfsense/pull/3553



  • @doktornotor:

    Craptastic JS gone from the widget: https://github.com/pfsense/pfsense/pull/3553

    Thank you very much!

    My NTP widget time was lagging by about 8 minutes.
    Your script works fine, and now the NTP widget shows the correct time.


  • Rebel Alliance Global Moderator

    that patch was pushed to master back in feb of 2017… What version of pfsense are you running that you would manually put in that patch?