Ntpd / gps need some love part II

pete

Thank-you Robi!

Updated files. Sync to 11 satellites took less than 5 seconds. gps0 and pps0 are there under devs.

I can see output now just fine with gps0.

[2.1.3-RELEASE][rootat]/dev(5): ntpq -c clockvar
assID=0 status=00f2 clk_okay, last_clk_242,
device="NMEA GPS Clock",
timecode="$GPGGA,142307.000,4134.4394,N,08800.6301,W,2,9,1.03,223.7,M,-34.0,M,0000,0000*67",
poll=17, noreply=0, badformat=111, baddata=0, fudgetime2=400.000,
stratum=0, refid=GPS, flags=5

You have made my day today Friday, 16th of May 2014 a great day!

Geeze; its like Christmas near Chicago today; snow and all.

robi

Glad to help. But I've got no clue why you couln't apply the patch… I did on 3 different boxes so far, and had no problems...

pete

Thank you Robi.

Yup for whatever reason new application cannot get to internet NTP servers. I can ping them just fine though.

Playing with this stuff for years did originally block the NTP ports on my firewall and just used my GPS NTP time server for home network time.

Ideally I am fine with this scenario of not having to utilize NTP on the internet.

Is there a way to disable the use of internet NTP servers with this application?

The patch did work fine on 2.1.2. I updated to 2.1.3 and then had issues with the patches (didn't work)

It was probably me mucking up the 2.1.3 build.

On a lark will rebuild box #2 with current 2.1.3 from scratch and will try patch and ….

Going to try this with another box (well #3) which is just a faster CPU and more memory.

May 17 06:19:46 ntpd[52732]: Listen and drop on 0 v6wildcard [::]:123
May 17 06:19:46 ntpd[52732]: proto: precision = 1.676 usec (-19)
May 17 06:19:46 ntpd[52710]: Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
May 17 06:19:46 ntpd[52710]: ntpd 4.2.7p411@1.2483-o Fri Mar 28 00:14:21 UTC 2014 (1): Starting
May 17 06:19:45 ntpd[4636]: ntpd exiting on signal 15 (Terminated: 15)

I found a related thread here about putting in static routes to the IP's of the NTP servers in Internetlandia which didn't work for me though.

ntp.jpg_thumb

pete

Well found another issue or concern. I don't know if this is the right place to post.

It appears that I cannot sync my NTP from the PFSense box for whatever reason.

I did a quickie test from a Wintel enterprise server and an Ubuntu 14.04 server and both are not getting NTP from PFSense and do get it fine from the internet NTP servers. I found this to be happening by looking at my IP HD CCTV cameras which were syncing to the PFSense box and the time was way off on these devices.

Here is the "test" on the Wintel box using Tardis. NTP / 123 is not blocked as it's getting it directly from the internet which is odd to me.

This is indicatory though that NTP from the internet does work; but doesn't work with PFSense / GPS NTP update stuff.

This sort of defeats the purpose of using NTP with a GPS/PPS on the PFSense box for me.

On a Wintel enterprise server I see:

C:\WINDOWS\system32>tardisnt debug
2014/05/19 08:24:55.39,Debug: RFC868 TCP Server started
2014/05/19 08:24:55.39,Debug: RFC868 UDP Server started
2014/05/19 08:24:55.39,Debug: RFC867 TCP Server started
2014/05/19 08:24:55.39,Debug: RFC867 UDP Server started
2014/05/19 08:24:55.45,Debug: SNTP Server started
2014/05/19 08:24:55.45,Debug: SNTP Client started
2014/05/19 08:24:55.45,Info : SNTP Client connecting to IP_OF_PFSense_Gateway
2014/05/19 08:25:10.47,Warn : SNTP Client No reply
2014/05/19 08:25:12.47,Debug: SNTP Client Stopping
2014/05/19 08:25:12.47,Debug: SNTP Client started
2014/05/19 08:25:12.47,Info : SNTP Client connecting to 2.pool.ntp.org

On an Ubuntu 14.04 box I see:

19 May 08:43:41 ntpdate[17929]: no servers can be used, exiting
root@ICS-ZM2:~# ntpdate IP_OF_PFSENSE_BOX
19 May 08:43:59 ntpdate[17937]: no server suitable for synchronization found
root@ICS-ZM2:~#
root@ICS-ZM2:~# ntpdate 1.pool.ntp.org
19 May 08:45:27 ntpdate[17979]: step time server 162.210.196.6 offset -5.106127 sec
root@ICS-ZM2:~#

NTP Log shows:

May 19 23:04:12 ntpd[86951]: ntpd 4.2.7p411@1.2483-o Fri Mar 28 00:14:21 UTC 2014 (1): Starting
May 19 16:28:55 ntpd[86135]: unable to bind to wildcard address :: - another process may be running - EXITING
May 19 16:28:55 ntpd[86135]: proto: precision = 1.676 usec (-19)
May 19 16:28:55 ntpd[85984]: Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
May 19 16:28:55 ntpd[85984]: ntpd 4.2.7p411@1.2483-o Fri Mar 28 00:14:21 UTC 2014 (1): Starting
May 19 16:28:40 ntpd[81621]: unable to bind to wildcard address :: - another process may be running - EXITING
May 19 16:28:40 ntpd[81621]: proto: precision = 1.676 usec (-19)

Looking at above relating to NTPD in general found something relating to having NTPDATE on startup.

Something called a race condition between NTPD and NTPDate.

I didn't see it anywhere though.

pete

Apologies guys.

The issue was related to the Access restrictions. I had left it at the defaults. I changed it and everything is now working.

Is there a way to add a second NTP connection (well GPSd) to this configuration?

GPS-1.jpg_thumb

GPS-2.jpg_thumb

robi

@pete:

Is there a way to disable the use of internet NTP servers with this application?

I think you can either delete all external NTP servers from your config, or add only one pointing to localhost. Or just tick "noselect" for them, and NTP service will not use them for timing.

If you want to use only your GPS and PPS as a time source, don't forget to set "All" at the "NMEA sentences" option. This will allow to extract the relative time from the NMEA sentences in text format. PPS will be used as usual to keep the time exact within the second.
I tested this by explicitly moving my test pfSese box's date and time to some random value in the past. Pulled the WAN cable, and let the box boot. NTPd just took the correct date and time from the GPS's NMEA data coming in though the serial port, and set the system date and time accordingly. Plus PPS accuracy included. No WAN access at all.

pete

I think you can either delete all external NTP servers from your config, or add only one pointing to localhost. Or just tick "noselect" for them, and NTP service will not use them for timing.

Thank you Robi. Yup have another GPS in a different part of the house such that I have it connected to a Wintel Server which I can point PFSense to.

charliem

Just a heads-up here for serial GPS users.

I'm running 2.2 alpha with an Adafruit GPS, and noticed a number spikes in the logs, with corresponding spikes in the ntp rrd graphs:

May 20 22:19:50 pfsense ntpd[16682]: 0.0.0.0 c615 05 clock_sync
May 20 22:19:51 pfsense ntpd[16682]: 0.0.0.0 0413 03 spike_detect -0.175410 s
May 20 22:19:57 pfsense ntpd[16682]: 0.0.0.0 041b 0b leap_event
May 20 22:20:07 pfsense ntpd[16682]: 0.0.0.0 0415 05 clock_sync
May 20 22:24:55 pfsense ntpd[16682]: 0.0.0.0 041d 0d kern PPS enabled
May 23 05:32:39 pfsense ntpd[16682]: 0.0.0.0 0413 03 spike_detect -0.249206 s
May 23 05:32:55 pfsense ntpd[16682]: 0.0.0.0 0415 05 clock_sync
May 23 21:49:59 pfsense ntpd[16682]: 0.0.0.0 0413 03 spike_detect -0.142089 s
May 23 21:50:15 pfsense ntpd[16682]: 0.0.0.0 0415 05 clock_sync
May 24 05:20:07 pfsense ntpd[16682]: 0.0.0.0 0413 03 spike_detect -0.138790 s
May 24 05:20:23 pfsense ntpd[16682]: 0.0.0.0 0415 05 clock_sync
May 25 20:47:35 pfsense ntpd[16682]: 0.0.0.0 0413 03 spike_detect -0.397899 s
May 25 20:48:07 pfsense ntpd[16682]: 0.0.0.0 0415 05 clock_sync

Then I noticed a significant number of replies from the GPS couldn't be parsed properly by ntpd (badformat). Here, 166 of 34901 responses have had errors:

[2.2-ALPHA][root@pfsense.localdomain]/var/log(11): ntpq
ntpq> cv
associd=0 status=00f2 15 events, clk_bad_format,
device="NMEA GPS Clock",
timecode="$GPGGA,132646.000,3666.4384,N,08999.4250,W,2,8,0.98,224.9,M,-32.5,M,0000,0000*61",
poll=34901, noreply=0, badformat=166, baddata=0, fudgetime2=400.000,
stratum=0, refid=pps, flags=5
ntpq> q

Looking further, I saw that my gps.init strings did not properly turn off the sentences as I had intended. Turns out I had the wrong checksum in the command I entered :) Once I got the GPS sending only the $GPGGA sentence, I haven't had any spikes (so far). If I was ambitious, I'd go back in my clockstats file, and see what the strings looked like around the times the spikes were detected. Hopefully it's fixed though.

So, moral of the story:

Verify your checksums if you enter init commands by hand, and
Turn off extra NMEA sentences!

robi

I suggest to use ZDA or ZDG option instead of GGA when considering sentences in case you have problems like that, as $GPZDA is less than half in size than $GPGGA, it contains only Date & Time. In order to lessen even more the stress for ntpd to process the string every second.

pete

Thank-you Robi!

Did a quickie check this morning and noticed that it wasn't running. Enabled ZDA or ZDG option and all is well this morning.

I wasn't paying attention. Google location stuff was eye candy but I liked the number of satellites it was getting.

ntpq> cv
localhost: timed out, nothing received
***Request timed out
ntpq>

<changed and="" saved="" settings="" here="">ntpq> cv
assID=0 status=0000 clk_okay, last_clk_okay,
device="NMEA GPS Clock", timecode="$GPZDA,120846.000,28,05,2014,,*57",
poll=2, noreply=0, badformat=0, baddata=0, fudgetime2=400.000,
stratum=0, refid=GPS, flags=5
ntpq></changed>

So I calculated the checksum and edited the command text saving the correct check sum for each of the command lines.

SureGPS-Checksum.jpg_thumb

robi

I just found a couple of small issues, like typos and html display stuff, fine-tuned a bit SureGPS defaults and noticed that RRD graphs were not plotting on 2.1.3, fixed that.
If applied patch using system patches package, revert the old patch, and apply this new one, or simply overwrite the files on the system, and press "Save" on the Serial GPS page.
Pushed the fixes to GitHub also, for 2.2.

(Edit: attachments removed, read further for updates)

pete

Thanks Rob. For the update posted do I need to recalculate the checksum stuff?

I did notice while using FF after the save on the serial GPS page; the page looked a bit weird.

Here is the adjusted checksums for the SureGPS.

$PMTK225,025
$PMTK314,1,1,0,1,0,5,0,0,0,0,0,0,0,0,0,0,0,1,023
$PMTK301,220
$PMTK397,02D
$PMTK1023F
$PMTK313,120
$PMTK513,126
$PMTK319,02B
$PMTK527,0.000E
$PMTK251,960019

GPS-Wierdness.jpg_thumb

stephenw10

Love the annotation. ;D

Steve

robi

@pete:

Thanks Rob. For the update posted do I need to recalculate the checksum stuff?

I did notice while using FF after the save on the serial GPS page; the page looked a bit weird.

Here is the adjusted checksums for the SureGPS.

$PMTK225,025
$PMTK314,1,1,0,1,0,5,0,0,0,0,0,0,0,0,0,0,0,1,023
$PMTK301,220
$PMTK397,02D
$PMTK1023F
$PMTK313,120
$PMTK513,126
$PMTK319,02B
$PMTK527,0.000E
$PMTK251,960019

What adjustments do these commands do? Why did you have to change them? There's a mistake for sure for example on the second line, checksum for that is not 23 but 2D. Didn't check all of them, but PMTK301,2 has the correct checksum of 2E, not 20.

Actually I use default settings (the ones which come when you select the model in the GPS pulldown), they all have already correct checksums precalculated. I'd say you should delete the whole thing, select "Generic" from the pulldown, then select "SureGPS" again, to re-load the defaults and press Save.

Don't know about the php error you see on that page, have you done any other modifications to your pfSense system? I use this patch on several machines in production, none of them shows this.

charliem

@charliem:

Once I got the GPS sending only the $GPGGA sentence, I haven't had any spikes (so far). If I was ambitious, I'd go back in my clockstats file, and see what the strings looked like around the times the spikes were detected. Hopefully it's fixed though.

I spoke (wrote?) too soon! My spikes are back, so I've got to do more digging; will update if I find anything.

charliem

@robi:

I just found a couple of small issues, like typos and html display stuff, fine-tuned a bit SureGPS defaults and noticed that RRD graphs were not plotting on 2.1.3, fixed that.

Thanks for keeping this up! I'm testing 2.2, and I've noticed that the ntp RRD data does not survive an update, unlike the other RRD groups like system, traffic, packets, etc. Note that I mean update from one 2.2 snapshot to the next, not the update from 2.1.3 to 2.2 (which I didn't try).

Sorry I haven't looked into it any deeper; I hate posting a problem without a solution … but your post reminded me.

robi

I also noticed that a restart of service sometimes, but a reconfiguration of NTP settings also deletes the entire RRD data, have no clue why.

charliem

@robi:

I also noticed that a restart of service sometimes, but a reconfiguration of NTP settings also deletes the entire RRD data, have no clue why.

Found it, I think … in /etc/inc/rrd.inc:

 /* NTP, set up the ntpd rrd file */
                if (isset($config['ntpd']['statsgraph'])) {
                        /* set up the ntpd rrd file */
                        if (!file_exists("$rrddbpath$ifname$ntpd")) {
                                $rrdcreate = "$rrdtool create $rrddbpath$ntpd --step $rrdntpdinterval ";
                                $rrdcreate .= "DS:offset:GAUGE:$ntpdvalid:0:1000 ";
...

There shouldn't be the '$ifname' in the check for the existing file, since ntpd stats are not per interface. From the previous captive portal stanza, $ifname is set to 'captiveportal' at this point anyway. So '/var/db/rrd/captiveportalntpd.rrd' is of course not found, and a new ntpd.rrd file is created every time through this code. Seems to be the case for both 2.2 and your 2.1.3 patch.

charliem

Also, I believe that UNKNOWN values need to be written in more data set fields during boot:

 /* set up the ntpd rrd file */
                        if (!file_exists("$rrddbpath$ntpd")) {
                                $rrdcreate = "$rrdtool create $rrddbpath$ntpd --step $rrdntpdinterval ";
                                $rrdcreate .= "DS:offset:GAUGE:$ntpdvalid:0:1000 ";
                                $rrdcreate .= "DS:sjit:GAUGE:$ntpdvalid:0:1000 ";
                                $rrdcreate .= "DS:cjit:GAUGE:$ntpdvalid:0:1000 ";
                                $rrdcreate .= "DS:wander:GAUGE:$ntpdvalid:0:1000 ";
                                $rrdcreate .= "DS:freq:GAUGE:$ntpdvalid:0:1000 ";
                                $rrdcreate .= "DS:disp:GAUGE:$ntpdvalid:0:1000 ";
                                $rrdcreate .= "RRA:MIN:0.5:1:1200 ";
                                $rrdcreate .= "RRA:MIN:0.5:5:720 ";
                                $rrdcreate .= "RRA:MIN:0.5:60:1860 ";
                                $rrdcreate .= "RRA:MIN:0.5:1440:2284 ";
                                $rrdcreate .= "RRA:AVERAGE:0.5:1:1200 ";
                                $rrdcreate .= "RRA:AVERAGE:0.5:5:720 ";
                                $rrdcreate .= "RRA:AVERAGE:0.5:60:1860 ";
                                $rrdcreate .= "RRA:AVERAGE:0.5:1440:2284 ";
                                $rrdcreate .= "RRA:MAX:0.5:1:1200 ";
                                $rrdcreate .= "RRA:MAX:0.5:5:720 ";
                                $rrdcreate .= "RRA:MAX:0.5:60:1860 ";
                                $rrdcreate .= "RRA:MAX:0.5:1440:2284 ";

                                create_new_rrd($rrdcreate);
                                unset($rrdcreate);
                        }

                        /* enter UNKNOWN values in the RRD so it knows we rebooted. */
                        if($g['booting']) {
                                mwexec("$rrdtool update $rrddbpath$ntpd N:U");
                        }

The update line should be:

mwexec("$rrdtool update $rrddbpath$ntpd N:U:U:U:U:U:U");

robi

Can you please attach your modified /etc/inc/rrd.inc?