NTP troubles post upgrade.

[miken58b]

Hi Steve

I've done a bit of delving into the Status>NTP 'hang'.  It also affects the Dashboard NTP widget.  I'm using a SureGPS evaluation board so had set GPS Type in Service>NTP>Serial GPS to 'SureGPS'.  Looking a the php script this seems to invoke some special code to process the $GPGSV sentence.  I copied the initialization settings to GPS Type 'Custom' and the hang disappears.  So my conclusion is there is some problem with processing $GPGSV and as thi only displays the number of satellites in view it's hardly that important.

I'm not 100% sure what the php script is actually doing - could it be it's looking for the $GPGSV sentancxe and then not finding it when expected?

Over to you!

Cheers

Mike  :)
'

[robi]

GSV - number of satellites in view - is important to know for troubleshooting antenna or location issues. If one day your number of satellites suddenly drops, you may have a problem with the antenna, or its cable, or somebody on the roof may have limited your view to the sky near the antenna.

[miken58b]

Just taking stock and I realise I have hijacked the original query from Paul.  Paul - have you resolved your original problem?

I have tried to delve a bit deeper into the $GPGSV issue.  The code in ntp_status.widget.php includes:

//GSV message is only enabled by init commands in services_ntpd_gps.php for SureGPS board

So that took me to look at the Init settings for the SureGPS.  This includes the following:

$PMTK314,1,1,0,1,0,5,0,0,0,0,0,0,0,0,0,0,0,1,0*2D

which controls the sentences output.  Now such documentation as I can find gives the following:

Packet Type: 314 PMTK_API_SET_NMEA_OUTPUT
Support Chip Type:
MT3318、MT3329、MT3339
Packet Meaning：
API_Set_NMEA_Out
Set NMEA sentence output frequencies
Data Field：
There are totally 19 data fields that present output frequencies for the 19 supported
NMEA sentences individually.
Supported NMEA Sentences
0 NMEA_SEN_GLL, // GPGLL interval - Geographic Position - Latitude longitude
1 NMEA_SEN_RMC, // GPRMC interval - Recommended Minimum Specific GNSS Sentence
2 NMEA_SEN_VTG, // GPVTG interval - Course over Ground and Ground Speed
3 NMEA_SEN_GGA, // GPGGA interval - GPS Fix Data
4 NMEA_SEN_GSA, // GPGSA interval - GNSS DOPS and Active Satellites
5 NMEA_SEN_GSV, // GPGSV interval - GNSS Satellites in View
6 //Reserved
7 //Reserved
13 //Reserved
14 //Reserved
15 //Reserved
16 //Reserved
17 //Reserved
18 NMEA_SEN_MCHN, // PMTKCHN interval – GPS channel status
Supported Frequency Setting
0 - Disabled or not supported sentence
1 - Output once every one position fix

and that rather implies each parameter should be 0 or 1 and not 5!  It may be that the GPS chip interprets anything other than '0' to mean enable but that's a guess.  I changed the setting in the code and reworked the checksum and updated the Base64 string but this doesn't make any difference.

Back to Paul's problem I note his configuration includes both PPS and GPS devices.  Paul - do you have two time sources or one?  My interpretation of pfSense's NTP implementation is that the PPS service was not required for serial GPS devices that supported both NMEA and PPS.  The SureGPS supports both once the wiring fix has been made - can't speak about the Garmin.  Anyway, while 'experimenting' I thought I would enable the PPS service - I just checked the enable kernel PPS driver.  It did fire up and logged exactly the same time offset as the GPS driver.  The PPS interface says:

Devices with a Pulse Per Second output such as radios that receive a time signal from DCF77 (DE), JJY (JP), MSF (GB) or WWVB (US) may be used as a PPS reference for NTP. A serial GPS may also be used, but the serial GPS driver would usually be the better option. A PPS signal only provides a reference to the change of a second, so at least one other source to number the seconds is required.

so I concluded PPS wasn't appropriate.  However, once 'touched' there's no obvious way of removing it.  :(  I tracked down the ntpd.conf file and manually removed the PPS entry but each time the ntpd service is restarted the ntpd.conf file is recreated and the PPS settings come back!  Any advice on how to resolve this (short of factory reset) would be appreciated.

Finally after much random experimentation occasionally the satellites in view appears but more often the status display hangs so no further forward with that.  I'm now running as a Custom setting and not SureGPS.

In reply to Robi my original rather dismissive comment about the importance of 'satellites in view' was more in relation to having a working pfSense system than intrinsic value.  When it's all working the offset keeps within +-6 micro seconds for much of the day but I do get larger excursions which are almost certainly satellite visibility related.  The main problem for me is positioning the GPS receiver higher to see more the the southern sky.

All endlessly entertaining and for us Brits a diversion from the sterile EU in/out debate…  ;D

Cheers

Mike

[Steve_B]

Mike,

Thanks for your experimenting. I hope it leads to some conclusions we can at least include in the documentation, if not the code base.

In general, pfSense stores all configuration data in /cf/conf/config.xml When a service is started, the config file required by the service is created from this central XML repository.

Edit that file and search for "<gps>" and you may find you can make the changes you need persistent. Make a back-up copy first though. XML files are fairly fragile.

I'm afraid we will need some much more powerful medicine than this to divert us from the absurdities of the US presidential election cycle :(</gps>

[charliem]

I have a sure GPS board, and will try to find it and do some verification this weekend.

You really want to use PPS, if your device has it available.  Note that the sure GPS board requires some manual wiring intervention to bring the PPS signal to the RS-232 connector. [edit ..] I see Mike has made that mod to his board.

That said, however, I don't think pfSense should enable both the PPS driver and the NMEA driver, but instead use the pps processing that's built into NMEA.  That's the recommended approach from the ntpd developer(s) (I know … citation needed).  AFAIK the PPS driver is better suited to devices that put out only PPS at the change of second, and still need other clocks to know the time.

I also noticed the inability to remove the PPS driver once enabled, that's clearly a pfSense bug.  It's harmless though, as it simply gets marked as a false ticker and the NMEA_GPS driver carries on using PPS.  If you want to disable the PPS driver manually, you can edit /etc/inc/system.inc, and add a "#" to the lines before the PPS driver in two places, like this:

$ntpcfg .= '#server 127.127.22.0';
$ntpcfg .= '#fudge 127.127.22.0';

[miken58b]

Hi CharlieM

Thanks for your post - I have commented out the PPS lines in ntpd.conf as per your recommendations.  I feel I'm beginning to find out how pfSense is put together and constructing a mental road map.

The only problem I'm left with is setting the GPS Type to SureGPS still hangs the status display and from what I can see in the php script this can only be a problem with capturing the $GPGSV sentence.  I thought about experimenting with the code - perhaps it's stuck in the while loop for some strange reason.  I would guess there isn't actually much wrong with it.  All I would say is having set GPS Type to SureGPS and getting the status script to hang, if I then reset GPS Type to Custom and Save the hang clears AND the number of satellites in view is displayed.  That seems to confirm my GPS is at least generating the sentence.  If I was going to debug it further I would want to build a test bed for fear of finally breaking pfSense in some way and giving myself a bigger problem!!

So if you are willing to look into it at least one guy on the planet would be appreciative!

Of course I don't actually need a time source accurate to a few microseconds but that's not the point.  ;D

Mike

[charliem]

@miken58b:

So that took me to look at the Init settings for the SureGPS.  This includes the following:

$PMTK314,1,1,0,1,0,5,0,0,0,0,0,0,0,0,0,0,0,1,0*2D

which controls the sentences output.  Now such documentation as I can find gives the following:

Packet Type: 314 PMTK_API_SET_NMEA_OUTPUT
Support Chip Type:
MT3318、MT3329、MT3339
Packet Meaning：
API_Set_NMEA_Out
Set NMEA sentence output frequencies
Data Field：
There are totally 19 data fields that present output frequencies for the 19 supported
NMEA sentences individually.
Supported NMEA Sentences
0 NMEA_SEN_GLL, // GPGLL interval - Geographic Position - Latitude longitude
1 NMEA_SEN_RMC, // GPRMC interval - Recommended Minimum Specific GNSS Sentence
2 NMEA_SEN_VTG, // GPVTG interval - Course over Ground and Ground Speed
3 NMEA_SEN_GGA, // GPGGA interval - GPS Fix Data
4 NMEA_SEN_GSA, // GPGSA interval - GNSS DOPS and Active Satellites
5 NMEA_SEN_GSV, // GPGSV interval - GNSS Satellites in View
6 //Reserved
7 //Reserved
13 //Reserved
14 //Reserved
15 //Reserved
16 //Reserved
17 //Reserved
18 NMEA_SEN_MCHN, // PMTKCHN interval – GPS channel status
Supported Frequency Setting
0 - Disabled or not supported sentence
1 - Output once every one position fix

and that rather implies each parameter should be 0 or 1 and not 5!  It may be that the GPS chip interprets anything other than '0' to mean enable but that's a guess.  I changed the setting in the code and reworked the checksum and updated the Base64 string but this doesn't make any difference.

The PMTK chips accept integers 0 thru 5 for this command, which indicate the interval or period at which they are emitted.  So, a 0 implies never, a 1 implies 'include that sentence every transmission' and a 5 implies 'include that sentence only every 5 transmissions'.  Not sure why that was chosen.

That command you quoted also seems to enable one of the reserved outputs, #17; I'd change that to zero just in case.

It's good ntp practice to minimize the number of sentences emitted by the GPS (with the init command), and then tell ntpd to use only that sentence (you can do this via services -> ntp -> serial GPS -> NMEA sentences).  This keeps the transmission length consistent and short, so you get no overruns at low baud rate and low jitter if you have to run w/o PPS.

Here is link to a post where I listed some useful GPS init commands for a few chipsets (long thread where pfSense ntp improvements were discussed & developed, somehow degenerated into flames …):
https://forum.pfsense.org/index.php?topic=67189.msg373885#msg373885

[miken58b]

Hi Charlie

I think I can see what the problem is but I don't know enough to be able to fix it.

The script status_ntpd.php contains the code:

if (isset($config['ntpd']['gps']['type']) && ($config['ntpd']['gps']['type'] == 'SureGPS') && (isset($gps_ok))) {
	//GSV message is only enabled by init commands in services_ntpd_gps.php for SureGPS board
	$gpsport = fopen("/dev/gps0", "r+");
	while ($gpsport) {
		$buffer = fgets($gpsport);
		if (substr($buffer, 0, 6) == '$GPGSV') {
			//echo $buffer."\n";
			$gpgsv = explode(',', $buffer);
			$gps_satview = $gpgsv[3];
			break;
		}
	}
}

so is trying to read the $GPGSV sentence direct from the serial port.  The serial port is also read by ntpd.  When ntpd is running and the GPS Type is set to SureGPS the script status_ntpd.php sometimes runs (and displays the satellites in view corrrectly) but mostly hangs.  By logging in using ssh I can kill the ntpd process and this allows status_ntpd.php to complete.  It seems there is conflict between ntpd and status_ntpd.php (but only when it's a SureGPS) over access to the serial port /dev/gps0.

I don't know enough about FreeBSD to know if this can never work or if there is some permission or locking setting which can simply be changed.  I guess status_ntpd.php will 'consume' the serial traffic until it's got what it needs and then completes.  If ntpd misses a cycle that's hardly significant.  It seems to me it would be better if ntpd had exclusive responsibility of handling the GPS serial traffic and passed the sentences required to other applications but, of course, $GPGSV may not feature in ntpd's world.

Hope this helps pin-point the problem.  Doesn't explain how (AFAIR) it used to work so I suspect something else has changed or it may be a timing issue and used to work by luck.

Cheers

Mike

[LabDog]

Back again. Job took me away.
Thanks for all the input on this problem(Mike).
I haven't had time to do any testing but did finally get the GPS to be seen through trial and error with many reboots.
The issue with triggering the PPS signal in the PPS tab and not being able to disable it with out editing a conf. file does need to fixed.
I should have some time this week to play with the suggestions Mike made and see what happens with the Garmin GPS I am using.
Will be back after I get some testing done.
Thanks
Paul
P.S.
One other thing that needs to be looked at is the resolution on the NPT Graph. Have micro second times and there is freq. in seconds the other lines turn into a straight line at the bottom of graph.

[robi]

@miken58b:

Hi Charlie

I think I can see what the problem is but I don't know enough to be able to fix it.

The script status_ntpd.php contains the code:

if (isset($config['ntpd']['gps']['type']) && ($config['ntpd']['gps']['type'] == 'SureGPS') && (isset($gps_ok))) {
	//GSV message is only enabled by init commands in services_ntpd_gps.php for SureGPS board
	$gpsport = fopen("/dev/gps0", "r+");
	while ($gpsport) {
		$buffer = fgets($gpsport);
		if (substr($buffer, 0, 6) == '$GPGSV') {
			//echo $buffer."\n";
			$gpgsv = explode(',', $buffer);
			$gps_satview = $gpgsv[3];
			break;
		}
	}
}

so is trying to read the $GPGSV sentence direct from the serial port.  The serial port is also read by ntpd.  When ntpd is running and the GPS Type is set to SureGPS the script status_ntpd.php sometimes runs (and displays the satellites in view corrrectly) but mostly hangs.  By logging in using ssh I can kill the ntpd process and this allows status_ntpd.php to complete.  It seems there is conflict between ntpd and status_ntpd.php (but only when it's a SureGPS) over access to the serial port /dev/gps0.

I don't know enough about FreeBSD to know if this can never work or if there is some permission or locking setting which can simply be changed.  I guess status_ntpd.php will 'consume' the serial traffic until it's got what it needs and then completes.  If ntpd misses a cycle that's hardly significant.  It seems to me it would be better if ntpd had exclusive responsibility of handling the GPS serial traffic and passed the sentences required to other applications but, of course, $GPGSV may not feature in ntpd's world.

Hope this helps pin-point the problem.  Doesn't explain how (AFAIR) it used to work so I suspect something else has changed or it may be a timing issue and used to work by luck.

Cheers

Mike

Good catch. I think all code related to GSV should be removed.

[wkd2639]

I've been going round and round in circles trying to get my Sure GPS to work.

No matter how many times I set the speed to 9600 in the config screen the system logs always say it opened the serial port at 4800, which obviously doesn't work.

If I tell it to connect at 19200 then the system logs say it opened the serial port at 19200. If I go to Custom GPS and set it to 9600 and save when the page reloads it will still be showing 4800.

I even tried setting it to speed 16 directly in the config.xml and reloading the page but when the page saves it still resets the serial port to 4800.

If it helps - the page shows 4800 has a value of 0, and 9600 has a value of 15, whenever the page is saved with 9600 selected it saves the value 15 not 16 in the config.xml file.

I hope I have provided enough information.

System Version:
2.3.1-RELEASE-p1 (i386)
built on Wed May 25 14:53:12 CDT 2016
FreeBSD 10.3-RELEASE-p3

Andy

EDIT:
OK I managed to get it working by forcing it to post speed value 16 by editing the page live in Chrome.

services_ntpd_gps.php needs line 281 fixing.

		[0 => '4800', 15 => '9600', 32 => '19200', 48 => '38400', 64 => '57600', 80 => '115200']

should be

		[0 => '4800', 16 => '9600', 32 => '19200', 48 => '38400', 64 => '57600', 80 => '115200']