NTP Not Working [SOLVED (totally)]

beremonavabi · Jun 1, 2018, 5:56 PM

@gertjan Thanks for the reply. I don't know if you intended to just provide me with a link to a Google Search for "pfsense ntp", but that's what the URL you used:

https://www.google.com/search?client=firefox-b&ei=hX8RW97XCsu7UZCHqdgI&q=pfsense+ntp&oq=pfsense+ntp&gs_l=psy-ab.3..0j0i22i30k1l9.9122.9610.0.9810.3.3.0.0.0.0.107.288.2j1.3.0....0...1c.1.64.psy-ab..0.3.285....0.6QCTW78GVd8

resolves to. Regardless, I've already done searches and come up with nothing useful. Was there a specific article you were trying to point me to?

johnpoz · Jun 1, 2018, 6:33 PM

Ntp server on pfsense is not going to serve up time to clients if it can not sync time. All the ntp servers your pointing to have 0 for reach.. So no ntp on pfsense is not going to work.

Looks like you can not reach these servers, even though your resolving them to IP. Maybe your ISP is blocking ntp now?

Do a simple ntp query to one of the ntp servers in question..

example

[2.4.3-RELEASE][root@sg4860.local.lan]/root: ntpdate -q 0.pfsense.pool.ntp.org
server 108.61.56.35, stratum 2, offset 0.003551, delay 0.05855
server 50.116.52.97, stratum 2, offset -0.000013, delay 0.05957
server 69.36.182.57, stratum 2, offset 0.003915, delay 0.05742
server 141.193.21.6, stratum 2, offset 0.013477, delay 0.09850
1 Jun 13:32:51 ntpdate[52350]: adjust time server 69.36.182.57 offset 0.003915 sec
[2.4.3-RELEASE][root@sg4860.local.lan]/root: ntpdate -q pool.ntp.org
server 192.138.210.214, stratum 2, offset -0.002151, delay 0.06967
server 45.79.111.114, stratum 2, offset 0.009330, delay 0.09789
server 74.120.81.219, stratum 3, offset 0.005477, delay 0.06345
server 34.225.6.20, stratum 2, offset 0.003509, delay 0.05782
1 Jun 13:33:02 ntpdate[52651]: adjust time server 34.225.6.20 offset 0.003509 sec
[2.4.3-RELEASE][root@sg4860.local.lan]/root:

If that is not working then yeah your having an issue talking to the nameservers.

beremonavabi · Jun 1, 2018, 7:09 PM

@beremonavabi A few more data points. I can ping the pools just fine. Here's one for 0.pfsense.pool.ntp.org:

PING 0.pfsense.pool.ntp.org (108.61.73.244): 56 data bytes
64 bytes from 108.61.73.244: icmp_seq=0 ttl=55 time=71.750 ms
64 bytes from 108.61.73.244: icmp_seq=1 ttl=55 time=71.785 ms
64 bytes from 108.61.73.244: icmp_seq=2 ttl=55 time=70.744 ms

--- 0.pfsense.pool.ntp.org ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 70.744/71.426/71.785/0.483 ms

And, here's a tracert:

 1  10.4.0.1  12.457 ms  14.250 ms  11.751 ms
 2  199.241.146.177  13.710 ms  12.849 ms  13.194 ms
 3  199.244.116.37  14.318 ms  14.021 ms  46.231 ms
 4  199.244.116.1  12.544 ms  13.587 ms  14.409 ms
 5  173.205.61.21  16.018 ms  15.139 ms  12.568 ms
 6  213.254.230.154  71.582 ms  69.599 ms  70.534 ms
 7  69.31.34.62  73.353 ms  71.083 ms  72.618 ms
 8  * * *
 9  108.61.93.178  72.594 ms  72.036 ms  71.717 ms
10  108.61.56.35  71.262 ms  70.703 ms  71.341 ms

DNS Lookup also gives a result:

DNS Lookup
Hostname 
0.pfsense.pool.ntp.org
Results
Result	Record type
50.116.52.97	A
72.5.72.15	A
154.16.245.246	A
45.33.48.4	A
Timings
Name server	Query time
127.0.0.1	0 msec

and, DIG:

Shell Output - dig 0.pfsense.pool.ntp.org
; <<>> DiG 9.11.2-P1 <<>> 0.pfsense.pool.ntp.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26967
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;0.pfsense.pool.ntp.org.		IN	A

;; ANSWER SECTION:
0.pfsense.pool.ntp.org.	47	IN	A	192.138.210.214
0.pfsense.pool.ntp.org.	47	IN	A	108.59.2.24
0.pfsense.pool.ntp.org.	47	IN	A	45.33.84.208
0.pfsense.pool.ntp.org.	47	IN	A	96.244.96.19

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Jun 01 11:42:48 PDT 2018
;; MSG SIZE  rcvd: 115

So, it looks like routing-wise, pfSense can handle the traffic to/from these NTP servers. It's just the actual pfSense NTP server that's giving me issues.

EDIT: I rebooted pfSense again and grabbed the NTP log for just after it started up:


Jun 1 12:05:28	ntpd	47401	Soliciting pool server 69.36.182.57
Jun 1 12:04:29	ntpd	47401	Soliciting pool server 2001:470:e66b:10::13
Jun 1 12:04:24	ntpd	47401	Soliciting pool server 216.229.4.69
Jun 1 12:04:23	ntpd	47401	Soliciting pool server 138.236.128.112
Jun 1 12:03:22	ntpd	47401	Soliciting pool server 2001:470:5:bf4::14
Jun 1 12:03:19	ntpd	47401	Soliciting pool server 23.239.26.89
Jun 1 12:03:17	ntpd	47401	Soliciting pool server 45.76.244.193
Jun 1 12:02:15	ntpd	47401	Soliciting pool server 2600:1f16:7a3:8a22:a922:8e9c:be3:992a
Jun 1 12:02:13	ntpd	47401	Soliciting pool server 74.82.59.150
Jun 1 12:02:13	ntpd	47401	Soliciting pool server 74.207.240.206
Jun 1 12:01:08	ntpd	47401	Soliciting pool server 2620:6:2000:104::48
Jun 1 12:01:07	ntpd	47401	Soliciting pool server 96.245.170.99
Jun 1 12:01:07	ntpd	47401	Soliciting pool server 216.229.0.50
Jun 1 12:01:06	ntpd	47401	DNS time.nist.gov -> 2610:20:6f96:96::4
Jun 1 12:01:05	ntpd	47401	0.0.0.0 c016 06 restart
Jun 1 12:01:05	ntpd	47401	0.0.0.0 c012 02 freq_set kernel -0.581 PPM
Jun 1 12:01:05	ntpd	47401	0.0.0.0 c01d 0d kern kernel time sync enabled
Jun 1 12:01:05	ntpd	47401	Listening on routing socket on fd #32 for interface updates
Jun 1 12:01:05	ntpd	47401	Listen normally on 11 lo0 127.0.0.1:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 10 lo0 [::1]:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 9 igb5 192.168.40.1:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 8 igb5 [fe80::208:a2ff:fe0b:9183%6]:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 7 igb4 192.168.30.1:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 6 igb4 [fe80::208:a2ff:fe0b:9182%5]:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 5 igb3 192.168.20.1:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 4 igb3 [fe80::208:a2ff:fe0b:9181%4]:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 3 igb2 192.168.10.1:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 2 igb2 [fe80::208:a2ff:fe0b:9180%3]:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 1 igb0 192.168.1.1:123
Jun 1 12:01:05	ntpd	47401	Listen normally on 0 igb0 [fe80::208:a2ff:fe0b:9184%1]:123
Jun 1 12:01:05	ntpd	47401	proto: precision = 0.370 usec (-21)
Jun 1 12:01:05	ntpd	47127	Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
Jun 1 12:01:05	ntpd	47127	ntpd 4.2.8p11@1.3728-o Fri Mar 16 18:58:06 UTC 2018 (1): Starting

Nothing leaps out at me as being wrong (except for it soliciting ipv6 addresses when ipv6 is disabled throughout).

beremonavabi · Jun 1, 2018, 6:50 PM

@johnpoz said in NTP Not Working:

ntpdate -q 0.pfsense.pool.ntp.org

Thanks, johnpoz. It looks like our posts crossed in the mist. But, here's the result of the ntpdate command you recommended:

Shell Output - ntpdate -q 0.pfsense.pool.ntp.org
server 171.66.97.126, stratum 1, offset -3.122794, delay 0.04608
server 129.250.35.250, stratum 2, offset -3.123750, delay 0.03802
server 155.94.164.121, stratum 2, offset -3.121402, delay 0.03841
server 107.161.30.25, stratum 2, offset -3.120848, delay 0.08383
 1 Jun 11:46:57 ntpdate[17654]: step time server 171.66.97.126 offset -3.122794 sec

cyberzeus · Jun 1, 2018, 8:16 PM

Is pfSense running on a VM inside of a provider network? If so, did you check with them if they are blocking port 123?

beremonavabi · Jun 1, 2018, 9:41 PM

@cyberzeus No. This is version 2.4.3-Release on my Netgate SG-4860.

EDIT: I've since updated to 2.4.3-RELEASE-p1 (amd64). As expected, it makes no difference.

beremonavabi · Jun 1, 2018, 9:56 PM

@beremonavabi Is there any way to force a time update in pfSense? If I'm reading the result of the ntpdate command correctly, my pfSense box is about 3 seconds off from the NTP server time. Could that be too big and pfSense is just refusing to make the change because of that?

Looking around, I see the following set of commands to do that:

sudo service ntp stop
sudo ntpd -gq
sudo service ntp start

But, I don't know if sudo is needed with pfSense, if it can be done within the GUI, or if it can be done via the console. I don't even know if those are valid things to do in pfSense.

EDIT: Looks like running ntpdate without a parameter does the job. I ran "ntpdate 0.pfsense.pool.ntp.org" from a Command Prompt in the GUI and it updated the time. But, nothing else has changed. The NTP server is still not grabbing the times from those internet NTP pools.

beremonavabi · Jun 1, 2018, 11:42 PM

@beremonavabi I think I found a solution. The following thread:

https://forum.netgate.com/topic/127340/solved-ntpd-on-vlan-sub-interface

talks about the same problem on vlans. He mentioned:

"In case WAN is not among the list of the interfaces to listen on, NTPD picks the source ip for it’s outgoing ntp traffic as follows:

sort the ip addresses where it is configured to listen on (interfaces)
select the first one as source address
This ip should have outgoing nat configured.
As I did not want to have NTPD listen on WAN interface and my vlan sub-interfaces did not have outgoing nat, all ntp traffic leaved the WAN interface with internal ip address as source ip.

Solution1: select WAN interface to listen on (access from outside is blocked by default)
Solution2: make sure you have outgoing nat for the interface with lowest ip address."

I had my WAN un-selected for NTP under Services > NTP. Once I added it, everything started working again. So, something must have changed from when I first configured this (either with the various versions or with my messing around with the settings on my other interfaces).

Anyway, is there any security implication for letting the NTP server use the WAN? Anything I need to firewall?

jahonix · Jun 1, 2018, 10:18 PM

This post is deleted!

johnpoz · Jun 2, 2018, 9:36 AM

You sure do not need to be listening on wan for it to go outbound and talk to the ntp servers.

jahonix · Jun 2, 2018, 1:54 PM

Maybe you need to check one of the entries as "Prefer"?

beremonavabi · Jun 3, 2018, 1:22 PM

@johnpoz said in NTP Not Working [SOLVED]:

You sure do not need to be listening on wan for it to go outbound and talk to the ntp servers.

Well, that's what I thought when I set it up initially (and it worked that way at the time). But, all I can say is that, at this time (with my current configuration on pfSense 2.4.3-Release), without WAN being selected, the NTP service doesn't work. Merely adding the WAN interface under Services > NTP causes it to immediately start working again.

EDIT: Another odd thing is that according to that quote I posted from the other thread, above, the other solution is to have the selected interface with the lowest IP address have outbound NAT configured. Well, for me, that's my LAN interface (192.168.1.0.24). Firewall > NAT > Outbound for that interface looks like:


Interface	Source	Source Port	Destination	Destination Port	NAT Address	NAT Port	Static Port	Description	Actions
		WAN	192.168.1.0/24	*	*	*	WAN address	*		LAN to WAN

So, unless I'm mis-reading that, I do have outbound NAT configured for it. Odd.

kpa · Jun 2, 2018, 2:07 PM

It's not as simple as you would think. NTP service uses UDP only and there is only one listening socket for sending out queries and listening for replies on UDP port 123. If you set the service not to listen on your WAN interface that will also cut off the service's ability to send anything out on the WAN interface. Yes I know, DNS (for example) does the right thing and uses a separate socket for sending out the queries with a randomized source port, unfortunately NTP hasn't caught up with the new developments yet.

jahonix · Jun 2, 2018, 2:08 PM

So you're saying all my setups with NTP servicing all internal networks but WAN don't work? I must have missed something then...

kpa · Jun 2, 2018, 2:31 PM

Look at your outbound NAT rules. There should be a rule for outbound NAT'ing anything sourced from 127.0.0.0/8. That is the thing that makes your set up still work if you happen to set the service not to listen on the WAN, the service will instead use the localhost 127.0.0.1 address for the traffic and it will be properly NAT'ed.

Edit: The service might also select your LAN interface in case WAN is not available and if the LAN interface has a routable IP address it works just fine, also if the LAN IP address is an RFC1918 address you're still good because of the standard outbound NAT.

beremonavabi · Jun 2, 2018, 2:30 PM

@kpa said in NTP Not Working [SOLVED]:

Look at your outbound NAT rules. There should be a rule for outbound NAT'ing anything sourced from 127.0.0.0/8. That is the thing that makes your set up still work if you happen to set the service not to listen on the WAN, the service will instead use the localhost 127.0.0.1 address for the traffic and it will be properly NAT'ed.

Interface	Source	Source Port	Destination	Destination Port	NAT Address	NAT Port	Static Port	Description	Actions
		WAN	127.0.0.0/8	*	*	*	WAN address	*		Localhost to WAN

I've got that rule, too. But, without WAN being selected for NTP, it still didn't work.

johnpoz · Jun 2, 2018, 2:46 PM

I do not have wan selected, and ntp works just fine to outside ntp servers.

And yeah your loopback should be outbound nat.. Unless the user turned off automatic outbound nat then yeah loopback would work and or any of other lan side interfaces. If they turned off loopback nat, pretty sure that would break a bunch of stuff.

cyberzeus · Jun 2, 2018, 10:15 PM

This post is deleted!

beremonavabi · Jun 2, 2018, 10:19 PM

Just to double-check this (in case my forcing a clock update in my troubleshooting attempts "fixed" this), I unselected WAN from the selection list on Services > NTP and restarted the NTP service. I was right back in the same situation I was when I started this thread:

The NTP Status Dashboard widget has the time, but lists the Sync Source as “No active peers available.”

Under Status > NTP, all the pools show a status of “Pool Placeholder” and non-pools show “Unreach/Pending.” They all have “Stratum” equal to 16 instead of the 1 or 2 they should be, and the “When” fields are all blank (just a dash). All the statistics are 0.
...
Under Status > System Logs > NTP, all the log entries are nothing but “Soliciting pool server…” messages.

Re-selecting the WAN interface on the list and restarting the NTP service starts everything up properly and everything works again.

When it's not working, I don't see anything in any of the logs (except for the constant "Soliciting pool server" messages in the NTP log) related to this. Anyone have any suggestions on log settings to try to track this down? I'd really prefer not having NTP listening to the WAN interface if at all possible.

jahonix · Jun 2, 2018, 11:21 PM

So did you check an entry as "Prefer"???