NTP Not Working [SOLVED (totally)]

beremonavabi · Jun 2, 2018, 10:19 PM

Just to double-check this (in case my forcing a clock update in my troubleshooting attempts "fixed" this), I unselected WAN from the selection list on Services > NTP and restarted the NTP service. I was right back in the same situation I was when I started this thread:

The NTP Status Dashboard widget has the time, but lists the Sync Source as “No active peers available.”

Under Status > NTP, all the pools show a status of “Pool Placeholder” and non-pools show “Unreach/Pending.” They all have “Stratum” equal to 16 instead of the 1 or 2 they should be, and the “When” fields are all blank (just a dash). All the statistics are 0.
...
Under Status > System Logs > NTP, all the log entries are nothing but “Soliciting pool server…” messages.

Re-selecting the WAN interface on the list and restarting the NTP service starts everything up properly and everything works again.

When it's not working, I don't see anything in any of the logs (except for the constant "Soliciting pool server" messages in the NTP log) related to this. Anyone have any suggestions on log settings to try to track this down? I'd really prefer not having NTP listening to the WAN interface if at all possible.

jahonix · Jun 2, 2018, 11:21 PM

So did you check an entry as "Prefer"???

beremonavabi · Jun 2, 2018, 11:29 PM

@jahonix said in NTP Not Working [SOLVED]:

So did you check an entry as "Prefer"???

Yes. It made no difference.

Gertjan · Jun 4, 2018, 5:48 AM

@beremonavabi said in NTP Not Working [SOLVED]:

Was there a specific article you were trying to point me to?

Wow - I guess I checked in after the storm.
Indeed, I was using Google myself to find some possible related posts : ntp not being able to contact remote time servers (Firewall and gateway issues).
Clear is now : there is a something that is not 'default'.
NTP settings : I haven't checked the WAN interface - only my LAN interfaces (I thought this ensures that ntp is serving my local devices with a time server, my pfSense box). See image below.

My Outbound NAT is pretty big and pretty default :

My WAN IP is RFC1918 (192.168.1.10.0/24)
LAN and OPT1 are default.

My NTP settings (tghis time with WAN selected - that works also for me ):

Btw : selecting WAN isn't big deal. As long as there is no firewall rule letting in connections, your ok.
You should know that the GUI is also listening on ALL interfaces (WAN included !).

The only thing I changed there was "fr.pool.ntp.org" (I'm living in France) and checked "this is a pool).

Did you change anything on the Services => NTP => ACLs tab ?

beremonavabi · Jun 4, 2018, 2:05 PM

On my WAN interface, I'm allowing IP4 UDP traffic going to port 443 (for OpenVPN clients connecting to my OpenVPN server):

My NTP ACL tab is default: Kiss-o'-death, Modifications, Peer Association, and Trap Service are all checked, while Queries and Service are unchecked.

EDIT: And since so few of us are having this problem, there's got to be something non-default somewhere that's causing this (as you said). I just can't find it.

beremonavabi · Jun 7, 2018, 2:28 PM

According to:

http://support.ntp.org/bin/view/Support/TroubleshootingNTP#Section_9.8.

ntpd requires full bidirectional access to the privileged UDP port 123

I assume that's what kpa was referring to, above. So, for those of you who do have NTP synchronizing time correctly without having the WAN interface selected (apparently, most everyone), do you have a firewall rule opening port 123 to UDP traffic on your WAN?

Also, according to:

http://support.ntp.org/bin/view/Support/TroubleshootingNTP#Section_9.9.

there's supposed to be an ntp.conf file at /etc. I don't see that on my SG-4860. Does pfSense use that file for NTP?

Gertjan · Jun 7, 2018, 3:00 PM

@beremonavabi said in NTP Not Working [SOLVED]:

According to:

http://support.ntp.org/bin/view/Support/TroubleshootingNTP#Section_9.8.

ntpd requires full bidirectional access to the privileged UDP port 123

Yep.
Part "9.8. Check the NTP port" also states :

Bla bla bla bla ...
If this is not possible, you may need to run ntpd on the firewall itself, so that it can have full unrestricted access to UDP port >123 in both directions, and then have it serve time to your internal clients. However, this may also be disallowed.

Of course, our ntpd is running on the firewall pfSense with 'root' privileges, so it can snap to this "123" port, so it can send out requests , and receives the replies.

I assume that's what kpa was referring to, above. So, for those of you who do have NTP synchronizing time correctly without having the WAN interface selected (apparently, most everyone), do you have a firewall rule opening port 123 to UDP traffic on your WAN?

No way !!!!
As seen above, the ntpd process can go outside anytime it wants to do so - connect itself as a "client" to a ntp "server". These connections are outbound.
Firewall rules on the (WAN) interface are for inbound connection only.
I have no (well, yes : one : an incoming VPN rule) firewall rules on WAN, so by default no initial connection comes in.
NTP servers like fr.pool.ntp.org do not connect to my pfSEnse. It's my pfSense that connects to fr.pool.ntp.org.

Also, according to:
http://support.ntp.org/bin/view/Support/TroubleshootingNTP#Section_9.9.
there's supposed to be an ntp.conf file at /etc. I don't see that on my SG-4860. Does pfSense use that file for NTP?

Yes, no ... this is FreeBSD.
See your post number 4 here ( you !) told us were we (you !) can find this file ^^
As you said yourself over there : the file is here : /var/etc/ntpd.conf

beremonavabi · Jun 10, 2018, 9:26 PM

One more attempt at figuring this out.

I saw a bug report that was similar (though it involved CARP):

https://redmine.pfsense.org/issues/5548

The response was:

You're breaking NTP connectivity on the backup by sending the traffic using a CARP IP. It won't, and can't, receive those replies - they go to the primary. When WAN isn't bound, it's probably hitting NAT to a CARP IP because it has a private source IP. NAT it to the WAN IP in that case.

The OP responded:

Added the following NAT rules at the top of the Outbound manual rules list:

Interfc Source Src Pt Dest Dest Pt NAT Addr NAT Pt Static Description 
WAN1 This Fw udp/* * udp/123 WAN1 addr * NO NTP to WAN1 INTFC IP

and that fixed it.

I was wondering if my restricting all outbound traffic to go through my VPN might be causing a similar problem here. So, I added the following outbound NAT rule (even though it ought to have been covered by my existing NAT rules and I've got "redirect-gateway def1; in my VPN client Custom Options to make sure the firewall, itself, can get out the default interface) which, I think, is an equivalent:

It doesn't make any difference. NTP still won't start without the WAN being selected. Can anyone confirm that these situations might be equivalent and that the NAT rule I added makes some kind of sense?

stephenw10 · Jun 11, 2018, 12:00 AM

Can we see your full outbound NAT page?

Steve

beremonavabi · Jun 11, 2018, 12:46 AM

@stephenw10

stephenw10 · Jun 11, 2018, 1:01 PM

Hmm, looks OK. And you're using manual mode there or hybrid?

The the WAN the default gateway on your system?

Is default gateway switching enabled?

I could imagine NTP trying to use the VPN gateway and sourcing from something not NAT'd there. It should always use the default gateway though.

Steve

beremonavabi · Jun 11, 2018, 2:42 PM

@stephenw10 said in NTP Not Working [SOLVED (mostly)]:

Hmm, looks OK. And you're using manual mode there or hybrid?

The the WAN the default gateway on your system?

Is default gateway switching enabled?

I could imagine NTP trying to use the VPN gateway and sourcing from something not NAT'd there. It should always use the default gateway though.

Steve

Manual mode and, yes, the WAN is the default gateway. Default gateway switching is OFF under System > Advanced > Miscellaneous.

In normal operation, everything is on the VPN_LAN interface (192.168.20.0/24). My firewall rules for that are:

The first two rules are special cases that are hardly ever used. The third rule sends local device traffic to other local devices out the default. And the last rule sends traffic to the outside world out via the VPN's Gateway Group. So, in general, everything to the outside world goes out over the VPN via that Gateway Group. But, I've got "redirect-gateway def1;" in my VPN client Custom Options to make sure the firewall, itself, can still get out the default gateway. I assume NTP falls under that and should get out via the WAN.

That special Outbound NAT rule with "This Firewall", above, was just a desperate stab at trying to make sure that NTP traffic could get out the WAN and not get stuck by the VPN. But, it didn't work. Assuming I wrote that rule correctly, it doesn't look like that's the issue.

I wonder if it could be a DNS issue (not being able to resolve the NTP pool names to addresses -- though why adding the WAN to the NTP listening interfaces would "fix" that, I don't know). I'm using DNS Resolver in NON-Forwarding mode. It's active for all my local LAN-type interfaces (not WAN) and sends everything out via the VPNx_WAN interfaces (again, not WAN). I wonder if I should add the WAN to the Outgoing Network Interfaces:

beremonavabi · Jun 11, 2018, 8:18 PM

@beremonavabi
It doesn't look like it's a DNS issue, either. I stuck the actual IP address for a public DNS server

http://support.ntp.org/bin/view/Servers/PublicTimeServer000011

in and removed the WAN interface. Same problem: NTP doesn't start. Put the WAN back in the list and all was well.

beremonavabi · Jun 20, 2018, 9:12 PM

Still not working without WAN selected in Settings > NTP. For posterity, here's some information on my gateways:

RW_VPN is my VPN server, VPNx_WAN are my two VPN clients, IPv6 is off so all the IPV6 gateways are disabled, and I've scribbled over my WAN_DHCP addresses for privacy purposes.

I've also got a Gateway Group:

johnpoz · Jun 21, 2018, 10:12 AM

See that other thread on how dicked up the other guys setup was to why it wasn't working.. A clue to why yours isn't working is prob in there as well.

johnpoz · Jun 21, 2018, 10:17 AM

@beremonavabi said in NTP Not Working [SOLVED (mostly)]:

RW_VPN is my VPN server

Huh? Why would you vpn server your running on pfsense be setup as a gateway?

Looks like he is shooting himself in the foot same way other guy was... Manual mode outbound nat and not natting what the ntp is trying to use as the source IP, etc.

And for what reason are you using manual for your outbound nat?

Gertjan · Jun 21, 2018, 2:12 PM

Just thinking and typing out loud :

Is this not just another case of trying to pass everything through the VPN tunnel ?

Knowing that the ntp deamon probably starts before the VPN server, it will work for a while : the WAN connections works, (NAT) rules are fine.
Then the VPN server starts, the gateways are shot in the back and reconstructed, so that all traffic goes out of the VPN, the new "WAN".
Our ntp isn't informed, and is locked out.

Something like that ?

jimp · Jun 21, 2018, 3:27 PM

Firewall > NAT, Outbound tab. Add rule to top.

Disabled: Unchecked
Do not NAT: Unchecked
Interface: WAN (make one of these rules for each WAN)
Protocol: any
Source: This Firewall (self)
Destination: any
Not: Unchecked
Translation Address: Interface Address
Port or Range: Blank
Description: NAT anything out from the firewall itself

beremonavabi · Jun 21, 2018, 4:06 PM

@jimp said in NTP Not Working [SOLVED (mostly)]:

Firewall > NAT, Outbound tab. Add rule to top.

Disabled: Unchecked
Do not NAT: Unchecked
Interface: WAN (make one of these rules for each WAN)
Protocol: any
Source: This Firewall (self)
Destination: any
Not: Unchecked
Translation Address: Interface Address
Port or Range: Blank
Description: NAT anything out from the firewall itself

Yay! Thanks, jimp. That got it. I had to reboot the pfSense box before it would work, though. I'd tried something similar a couple of weeks ago, but it was limited to UDP on port 123, was at the end of my list of outbound NAT rules, and, probably most importantly, I didn't reboot the box.

jimp · Jun 21, 2018, 4:11 PM

You probably didn't need to reboot, just clear the states table (Diagnostics > States, Reset States tab)