NTP troubles post upgrade.



  • Hi need help on fixing couple NTP issues.
    Did the jump to 2.3 all went rather smooth couple small easy fixes.
    Checking things out tonight noticed my GPS(Garmin 18x/LVC(serial)) used for time  sync. is not functioning any more.
    NTP log show connected with no kernel PPS.

    GPS_NMEA(0) serial /dev/gps0 open at 4800 bps
    0.0.0.0 061d 0d kern PPS enabled status: 2001 -> 0007
    0.0.0.0 062d 0d kern no PPS signal
    kernel reports TIME_ERROR: 0x2007: PPS Frequency Sync wanted but no PPS; PPS Time Sync wanted but no PPS signal

    Statue shows:

    Network Time Protocol Status
    Status	Server	Ref ID	Stratum	Type	When	Poll	Reach	Delay	Offset	Jitter
    Unreach/Pending	127.127.22.0	.PPS.	0	l	-	16	0	0.000	0.000	0.000
    Unreach/Pending	127.127.20.0	.GPS.	0	l	-	16	0	0.000	0.000	0.000
    Active Peer	24.56.178.140	.ACTS.	1	u	33	64	377	67.499	1.963	4.288
    

    Did some poking around didn't find any thing that looked out of place.
    Hoping somebody else might have an idea
    Thanks
    Paul



  • Hi.

    I'm having serial GPS problems post upgrade.  I'm using a SureGPS connecting via the serial port.  It looks like the serial port is not set to the correct baud rate (9600) or is not connecting to the ntpd application.  It's always been a 'fiddle' to get the link working on pfSense reboot (won't connect up unattended) but I can't get it to work at all with post 2.3 update which I installed yesterday.  I'll try and conduct some basic diagnostics and report any further findings.

    If it helps the system log reports:

    /services_ntpd_gps.php: The command 'cat /tmp/gps.init > /dev/cuau0' returned exit code '2', the output was ''

    but this doesn't mean much to me at the moment.

    Cheers

    Mike (generally a big pfSense fan).



  • Having spent a couple of hours trying to double check the ntp.conf settings and scratching by head and other parts of my anatomy for inspiration I restarted the NTP service for the umpteenth time and it suddenly decided to work.  This would possibly imply some kind of issue with the ntp process not getting access to the serial port when it starts.  When the GPS is connected and powered up it sends its NMEA string every second.  The board has a serial port activity light which, when it's not working correctly, shows pretty much continuous activity.  I'm guessing but thinking this could well be OpenBSD saying 'invalid username' to the NMEA input.  However, rebooting the PC with the GPS disconnected (and not connecting it until everything is up and running) doesn't make a difference.  I don't have a serial line monitor to see what the serial traffic actually is.

    My conclusion is there is a problem with the management of the serial port and this makes 'unattended' reboot impossible.  What has unfolded in the last few hours with the version 2.3 just seems to have made matters worse.  A fix would be good because I'm reluctant to restart the firewall because it's such a hassle.  I could move the GPS/NTP service to another computer I guess.  However a fix would be great!  ;D.

    Does anyone else with a serial GPS/NTP config find it 'just works' - could always be a hardware related condition and my firewall PC is nothing special at all.

    Cheers

    Mike



  • miken58b
    Ditto on all the issues you have been having. I had two almost identical systems with 2.2.6 they would not bring up the GPS on reboot. Plug it in after boot up it would work most of the time.
    Upgraded to 2.3 on my home system now the GPS will not come on line. Tested and verified the GPS on a windows machine all is functioning like it should. Will be doing some more testing tonight.
    Paul



  • Just spotted this:

    Linux Quirk

    Hal Murray notes:

    I'm in the habit of using things like:  cat /dev/ttyUSB0  when checking out GPS that I expect to send ASCII.  When I tried that with my new toy, it went bonkers.  In particular, the firmware LED went on, and the NMEA LED switched from blinking to solid on.  The data included the stuff I expected, but it also included a lot of garbage.  I (finally) tracked that down to "echo" being set on the tty parameters.  That echoes a copy of each input character back to where it came from.  The cure is as simple as:  stty -F /dev/ttyUSB0 -echo

    It's from David Taylor's excellent site: http://www.satsignal.eu/ntp/Sure-GPS.htm and describes precisely what I experience with pfSense.

    All I have to do is work out how to try it!

    MIke



  • OK - I think I have finally worked out what's wrong and fixed it to a point.  Unattended reboot still doesn't work.  Given Serial GPS support is a product feature I have to declare this as a pfSense bug.

    The serial port (in my case cuau0) seemed to be set a) to accept logins and b) echo input characters to output.  Given the connected GPS device is outputting a text string every second the traffic on the interface was continuous as described in the previous post.

    I am not a Unix or FreeBSD expert so if I have made inappropriate changes than I will gladly accept corrections.

    In the first instance I edited /etc/rc.d/serial to disable echo on all cua ports [[i]/dev/cuauN.lock]:

    # Change some defaults for serial devices.
    # Standard defaults are:
    #	dtrwait 300 drainwait `sysctl -n kern.drainwait`
    #	initial cflag from <sys ttydefaults.h=""> = cread cs8 hupcl
    #	initial iflag, lflag and oflag all 0
    #	speed 9600
    #	special chars from <sys ttydefaults.h="">
    #	nothing locked
    # except for serial consoles the initial iflag, lflag and oflag are from
    # <sys ttydefaults.h=""> and clocal is locked on.
    
    default() {
    	# Reset everything changed by the other functions to initial defaults.
    
    	dc=$1; shift	# device name character
    	drainwait=`sysctl -n kern.drainwait`
    
    	for i in $*
    	do
    		comcontrol /dev/tty${dc}${i} dtrwait 300 drainwait $drainwait
    		stty < /dev/tty${dc}${i}.init -clocal crtscts hupcl 9600 reprint ^R
    		stty < /dev/tty${dc}${i}.lock -clocal -crtscts -hupcl 0
    		stty < /dev/cua${dc}${i}.init -clocal crtscts hupcl 9600 reprint ^R
    		stty < /dev/cua${dc}${i}.lock -echo -clocal -crtscts -hupcl 0
    	done
    }</sys></sys></sys>
    

    This did not fix the problems but it made the serial traffic from the GPS to the PC less.  I then looked at the processes running and found a lot oaf activity associated with device u0 so concluded the serial traffic was being handled by the login process.  Referring to FreeBSD documentation I commented out device ttyu0 in /etc/ttys:

    # Serial terminals
    # The 'dialup' keyword identifies dialin lines to login, fingerd etc.
    # ttyu0	"/usr/libexec/getty al.9600"	cons25	onifconsole	secure
    ttyu1	"/usr/libexec/getty al.9600"	cons25	onifconsole	secure
    ttyu2	"/usr/libexec/getty al.9600"	cons25	onifconsole	secure
    ttyu3	"/usr/libexec/getty al.9600"	cons25	onifconsole	secure
    

    If I attempt to boot up with the GPS connected the boot fails and some audible error warning is emitted.  I don't know what the problem is.  If I disconnect the GPS the boot is clean.  Reconnecting the GPS and restarting the ntp service then reliably starts the GPS time synchronisation and the GPS serial traffic lights behave normally.

    It would seem some part of the configuration management is broken.  My fixes may not entirely correct but at least appear to work.

    Enough fun for the day!

    Cheers

    Mike  :)



  • OK - games up.  This has turned into a major case of 'insufficient user intelligence'  :-[.  It suddenly occurred to me to check if there were any serial port settings under System Set-up and under System>Advanced there is indeed a check box to enable a serial console on the first serial port.  I removed the tick mark and reversed the changes described in my previous posts.  My firewall will now reboot with GPS connected (on the first and only serial port) and the GPS will sync up without the need to disconnect just as one would expect.  ;D

    I'm left with one issue.  The Status>NTP php script now won't run at all and just hangs,  Previously it has run for a bit after reboot and then stopped working.  I don't think it does a lot more than display results from ntpq which I can run remotely and does work.  I have no experience of php or how to go about debugging it least of all in a 'live' environment - luckily it's not a show stopper.

    Onwards and upwards!

    Mike


  • Developer Netgate

    Glad you were able to figure it out :)

    status_ntp.php was recently updated to refresh dynamically via AJAX, but I did only minimal testing on the GPS portion.

    I'll test more thoroughly this week and see if I can shed any light on the problem you are encountering.



  • Hi Steve

    I've done a bit of delving into the Status>NTP 'hang'.  It also affects the Dashboard NTP widget.  I'm using a SureGPS evaluation board so had set GPS Type in Service>NTP>Serial GPS to 'SureGPS'.  Looking a the php script this seems to invoke some special code to process the $GPGSV sentence.  I copied the initialization settings to GPS Type 'Custom' and the hang disappears.  So my conclusion is there is some problem with processing $GPGSV and as thi only displays the number of satellites in view it's hardly that important.

    I'm not 100% sure what the php script is actually doing - could it be it's looking for the $GPGSV sentancxe and then not finding it when expected?

    Over to you!

    Cheers

    Mike  :)
    '



  • GSV - number of satellites in view - is important to know for troubleshooting antenna or location issues. If one day your number of satellites suddenly drops, you may have a problem with the antenna, or its cable, or somebody on the roof may have limited your view to the sky near the antenna.



  • Just taking stock and I realise I have hijacked the original query from Paul.  Paul - have you resolved your original problem?

    I have tried to delve a bit deeper into the $GPGSV issue.  The code in ntp_status.widget.php includes:

    //GSV message is only enabled by init commands in services_ntpd_gps.php for SureGPS board

    So that took me to look at the Init settings for the SureGPS.  This includes the following:

    $PMTK314,1,1,0,1,0,5,0,0,0,0,0,0,0,0,0,0,0,1,0*2D

    which controls the sentences output.  Now such documentation as I can find gives the following:

    Packet Type: 314 PMTK_API_SET_NMEA_OUTPUT
    Support Chip Type:
    MT3318、MT3329、MT3339
    Packet Meaning:
    API_Set_NMEA_Out
    Set NMEA sentence output frequencies
    Data Field:
    There are totally 19 data fields that present output frequencies for the 19 supported
    NMEA sentences individually.
    Supported NMEA Sentences
    0 NMEA_SEN_GLL, // GPGLL interval - Geographic Position - Latitude longitude
    1 NMEA_SEN_RMC, // GPRMC interval - Recommended Minimum Specific GNSS Sentence
    2 NMEA_SEN_VTG, // GPVTG interval - Course over Ground and Ground Speed
    3 NMEA_SEN_GGA, // GPGGA interval - GPS Fix Data
    4 NMEA_SEN_GSA, // GPGSA interval - GNSS DOPS and Active Satellites
    5 NMEA_SEN_GSV, // GPGSV interval - GNSS Satellites in View
    6 //Reserved
    7 //Reserved
    13 //Reserved
    14 //Reserved
    15 //Reserved
    16 //Reserved
    17 //Reserved
    18 NMEA_SEN_MCHN, // PMTKCHN interval – GPS channel status
    Supported Frequency Setting
    0 - Disabled or not supported sentence
    1 - Output once every one position fix

    and that rather implies each parameter should be 0 or 1 and not 5!  It may be that the GPS chip interprets anything other than '0' to mean enable but that's a guess.  I changed the setting in the code and reworked the checksum and updated the Base64 string but this doesn't make any difference.

    Back to Paul's problem I note his configuration includes both PPS and GPS devices.  Paul - do you have two time sources or one?  My interpretation of pfSense's NTP implementation is that the PPS service was not required for serial GPS devices that supported both NMEA and PPS.  The SureGPS supports both once the wiring fix has been made - can't speak about the Garmin.  Anyway, while 'experimenting' I thought I would enable the PPS service - I just checked the enable kernel PPS driver.  It did fire up and logged exactly the same time offset as the GPS driver.  The PPS interface says:

    Devices with a Pulse Per Second output such as radios that receive a time signal from DCF77 (DE), JJY (JP), MSF (GB) or WWVB (US) may be used as a PPS reference for NTP. A serial GPS may also be used, but the serial GPS driver would usually be the better option. A PPS signal only provides a reference to the change of a second, so at least one other source to number the seconds is required.

    so I concluded PPS wasn't appropriate.  However, once 'touched' there's no obvious way of removing it.  :(  I tracked down the ntpd.conf file and manually removed the PPS entry but each time the ntpd service is restarted the ntpd.conf file is recreated and the PPS settings come back!  Any advice on how to resolve this (short of factory reset) would be appreciated.

    Finally after much random experimentation occasionally the satellites in view appears but more often the status display hangs so no further forward with that.  I'm now running as a Custom setting and not SureGPS.

    In reply to Robi my original rather dismissive comment about the importance of 'satellites in view' was more in relation to having a working pfSense system than intrinsic value.  When it's all working the offset keeps within +-6 micro seconds for much of the day but I do get larger excursions which are almost certainly satellite visibility related.  The main problem for me is positioning the GPS receiver higher to see more the the southern sky.

    All endlessly entertaining and for us Brits a diversion from the sterile EU in/out debate…  ;D

    Cheers

    Mike


  • Developer Netgate

    Mike,

    Thanks for your experimenting. I hope it leads to some conclusions we can at least include in the documentation, if not the code base.

    In general, pfSense stores all configuration data in /cf/conf/config.xml When a service is started, the config file required by the service is created from this central XML repository.

    Edit that file and search for "<gps>" and you may find you can make the changes you need persistent. Make a back-up copy first though. XML files are fairly fragile.

    I'm afraid we will need some much more powerful medicine than this to divert us from the absurdities of the US presidential election cycle :(</gps>



  • I have a sure GPS board, and will try to find it and do some verification this weekend.

    You really want to use PPS, if your device has it available.  Note that the sure GPS board requires some manual wiring intervention to bring the PPS signal to the RS-232 connector. [edit ..] I see Mike has made that mod to his board.

    That said, however, I don't think pfSense should enable both the PPS driver and the NMEA driver, but instead use the pps processing that's built into NMEA.  That's the recommended approach from the ntpd developer(s) (I know … citation needed).  AFAIK the PPS driver is better suited to devices that put out only PPS at the change of second, and still need other clocks to know the time.

    I also noticed the inability to remove the PPS driver once enabled, that's clearly a pfSense bug.  It's harmless though, as it simply gets marked as a false ticker and the NMEA_GPS driver carries on using PPS.  If you want to disable the PPS driver manually, you can edit /etc/inc/system.inc, and add a "#" to the lines before the PPS driver in two places, like this:

    $ntpcfg .= '#server 127.127.22.0';
    $ntpcfg .= '#fudge 127.127.22.0';
    


  • Hi CharlieM

    Thanks for your post - I have commented out the PPS lines in ntpd.conf as per your recommendations.  I feel I'm beginning to find out how pfSense is put together and constructing a mental road map.

    The only problem I'm left with is setting the GPS Type to SureGPS still hangs the status display and from what I can see in the php script this can only be a problem with capturing the $GPGSV sentence.  I thought about experimenting with the code - perhaps it's stuck in the while loop for some strange reason.  I would guess there isn't actually much wrong with it.  All I would say is having set GPS Type to SureGPS and getting the status script to hang, if I then reset GPS Type to Custom and Save the hang clears AND the number of satellites in view is displayed.  That seems to confirm my GPS is at least generating the sentence.  If I was going to debug it further I would want to build a test bed for fear of finally breaking pfSense in some way and giving myself a bigger problem!!

    So if you are willing to look into it at least one guy on the planet would be appreciative!

    Of course I don't actually need a time source accurate to a few microseconds but that's not the point.  ;D

    Mike



  • @miken58b:

    So that took me to look at the Init settings for the SureGPS.  This includes the following:

    $PMTK314,1,1,0,1,0,5,0,0,0,0,0,0,0,0,0,0,0,1,0*2D

    which controls the sentences output.  Now such documentation as I can find gives the following:

    Packet Type: 314 PMTK_API_SET_NMEA_OUTPUT
    Support Chip Type:
    MT3318、MT3329、MT3339
    Packet Meaning:
    API_Set_NMEA_Out
    Set NMEA sentence output frequencies
    Data Field:
    There are totally 19 data fields that present output frequencies for the 19 supported
    NMEA sentences individually.
    Supported NMEA Sentences
    0 NMEA_SEN_GLL, // GPGLL interval - Geographic Position - Latitude longitude
    1 NMEA_SEN_RMC, // GPRMC interval - Recommended Minimum Specific GNSS Sentence
    2 NMEA_SEN_VTG, // GPVTG interval - Course over Ground and Ground Speed
    3 NMEA_SEN_GGA, // GPGGA interval - GPS Fix Data
    4 NMEA_SEN_GSA, // GPGSA interval - GNSS DOPS and Active Satellites
    5 NMEA_SEN_GSV, // GPGSV interval - GNSS Satellites in View
    6 //Reserved
    7 //Reserved
    13 //Reserved
    14 //Reserved
    15 //Reserved
    16 //Reserved
    17 //Reserved
    18 NMEA_SEN_MCHN, // PMTKCHN interval – GPS channel status
    Supported Frequency Setting
    0 - Disabled or not supported sentence
    1 - Output once every one position fix

    and that rather implies each parameter should be 0 or 1 and not 5!  It may be that the GPS chip interprets anything other than '0' to mean enable but that's a guess.  I changed the setting in the code and reworked the checksum and updated the Base64 string but this doesn't make any difference.

    The PMTK chips accept integers 0 thru 5 for this command, which indicate the interval or period at which they are emitted.  So, a 0 implies never, a 1 implies 'include that sentence every transmission' and a 5 implies 'include that sentence only every 5 transmissions'.  Not sure why that was chosen.

    That command you quoted also seems to enable one of the reserved outputs, #17; I'd change that to zero just in case.

    It's good ntp practice to minimize the number of sentences emitted by the GPS (with the init command), and then tell ntpd to use only that sentence (you can do this via services -> ntp -> serial GPS -> NMEA sentences).  This keeps the transmission length consistent and short, so you get no overruns at low baud rate and low jitter if you have to run w/o PPS.

    Here is link to a post where I listed some useful GPS init commands for a few chipsets (long thread where pfSense ntp improvements were discussed & developed, somehow degenerated into flames …):
    https://forum.pfsense.org/index.php?topic=67189.msg373885#msg373885



  • Hi Charlie

    I think I can see what the problem is but I don't know enough to be able to fix it.

    The script status_ntpd.php contains the code:

    if (isset($config['ntpd']['gps']['type']) && ($config['ntpd']['gps']['type'] == 'SureGPS') && (isset($gps_ok))) {
    	//GSV message is only enabled by init commands in services_ntpd_gps.php for SureGPS board
    	$gpsport = fopen("/dev/gps0", "r+");
    	while ($gpsport) {
    		$buffer = fgets($gpsport);
    		if (substr($buffer, 0, 6) == '$GPGSV') {
    			//echo $buffer."\n";
    			$gpgsv = explode(',', $buffer);
    			$gps_satview = $gpgsv[3];
    			break;
    		}
    	}
    }
    

    so is trying to read the $GPGSV sentence direct from the serial port.  The serial port is also read by ntpd.  When ntpd is running and the GPS Type is set to SureGPS the script status_ntpd.php sometimes runs (and displays the satellites in view corrrectly) but mostly hangs.  By logging in using ssh I can kill the ntpd process and this allows status_ntpd.php to complete.  It seems there is conflict between ntpd and status_ntpd.php (but only when it's a SureGPS) over access to the serial port /dev/gps0.

    I don't know enough about FreeBSD to know if this can never work or if there is some permission or locking setting which can simply be changed.  I guess status_ntpd.php will 'consume' the serial traffic until it's got what it needs and then completes.  If ntpd misses a cycle that's hardly significant.  It seems to me it would be better if ntpd had exclusive responsibility of handling the GPS serial traffic and passed the sentences required to other applications but, of course, $GPGSV may not feature in ntpd's world.

    Hope this helps pin-point the problem.  Doesn't explain how (AFAIR) it used to work so I suspect something else has changed or it may be a timing issue and used to work by luck.

    Cheers

    Mike



  • Back again. Job took me away.
    Thanks for all the input on this problem(Mike).
    I haven't had time to do any testing but did finally get the GPS to be seen through trial and error with many reboots.
    The issue with triggering the PPS signal in the PPS tab and not being able to disable it with out editing a conf. file does need to fixed.
    I should have some time this week to play with the suggestions Mike made and see what happens with the Garmin GPS I am using.
    Will be back after I get some testing done.
    Thanks
    Paul
    P.S.
    One other thing that needs to be looked at is the resolution on the NPT Graph. Have micro second times and there is freq. in seconds the other lines turn into a straight line at the bottom of graph.



  • @miken58b:

    Hi Charlie

    I think I can see what the problem is but I don't know enough to be able to fix it.

    The script status_ntpd.php contains the code:

    if (isset($config['ntpd']['gps']['type']) && ($config['ntpd']['gps']['type'] == 'SureGPS') && (isset($gps_ok))) {
    	//GSV message is only enabled by init commands in services_ntpd_gps.php for SureGPS board
    	$gpsport = fopen("/dev/gps0", "r+");
    	while ($gpsport) {
    		$buffer = fgets($gpsport);
    		if (substr($buffer, 0, 6) == '$GPGSV') {
    			//echo $buffer."\n";
    			$gpgsv = explode(',', $buffer);
    			$gps_satview = $gpgsv[3];
    			break;
    		}
    	}
    }
    

    so is trying to read the $GPGSV sentence direct from the serial port.  The serial port is also read by ntpd.  When ntpd is running and the GPS Type is set to SureGPS the script status_ntpd.php sometimes runs (and displays the satellites in view corrrectly) but mostly hangs.  By logging in using ssh I can kill the ntpd process and this allows status_ntpd.php to complete.  It seems there is conflict between ntpd and status_ntpd.php (but only when it's a SureGPS) over access to the serial port /dev/gps0.

    I don't know enough about FreeBSD to know if this can never work or if there is some permission or locking setting which can simply be changed.  I guess status_ntpd.php will 'consume' the serial traffic until it's got what it needs and then completes.  If ntpd misses a cycle that's hardly significant.  It seems to me it would be better if ntpd had exclusive responsibility of handling the GPS serial traffic and passed the sentences required to other applications but, of course, $GPGSV may not feature in ntpd's world.

    Hope this helps pin-point the problem.  Doesn't explain how (AFAIR) it used to work so I suspect something else has changed or it may be a timing issue and used to work by luck.

    Cheers

    Mike

    Good catch. I think all code related to GSV should be removed.



  • I've been going round and round in circles trying to get my Sure GPS to work.

    No matter how many times I set the speed to 9600 in the config screen the system logs always say it opened the serial port at 4800, which obviously doesn't work.

    If I tell it to connect at 19200 then the system logs say it opened the serial port at 19200. If I go to Custom GPS and set it to 9600 and save when the page reloads it will still be showing 4800.

    I even tried setting it to speed 16 directly in the config.xml and reloading the page but when the page saves it still resets the serial port to 4800.

    If it helps - the page shows 4800 has a value of 0, and 9600 has a value of 15, whenever the page is saved with 9600 selected it saves the value 15 not 16 in the config.xml file.

    I hope I have provided enough information.

    System Version:
    2.3.1-RELEASE-p1 (i386)
    built on Wed May 25 14:53:12 CDT 2016
    FreeBSD 10.3-RELEASE-p3

    Andy

    EDIT:
    OK I managed to get it working by forcing it to post speed value 16 by editing the page live in Chrome.

    services_ntpd_gps.php needs line 281 fixing.

    		[0 => '4800', 15 => '9600', 32 => '19200', 48 => '38400', 64 => '57600', 80 => '115200']
    

    should be

    		[0 => '4800', 16 => '9600', 32 => '19200', 48 => '38400', 64 => '57600', 80 => '115200']