NTP Not Working [SOLVED (totally)]

beremonavabi

@beremonavabi I think I found a solution. The following thread:

https://forum.netgate.com/topic/127340/solved-ntpd-on-vlan-sub-interface

talks about the same problem on vlans. He mentioned:

"In case WAN is not among the list of the interfaces to listen on, NTPD picks the source ip for it’s outgoing ntp traffic as follows:

sort the ip addresses where it is configured to listen on (interfaces)
select the first one as source address
This ip should have outgoing nat configured.
As I did not want to have NTPD listen on WAN interface and my vlan sub-interfaces did not have outgoing nat, all ntp traffic leaved the WAN interface with internal ip address as source ip.

Solution1: select WAN interface to listen on (access from outside is blocked by default)
Solution2: make sure you have outgoing nat for the interface with lowest ip address."

I had my WAN un-selected for NTP under Services > NTP. Once I added it, everything started working again. So, something must have changed from when I first configured this (either with the various versions or with my messing around with the settings on my other interfaces).

Anyway, is there any security implication for letting the NTP server use the WAN? Anything I need to firewall?

jahonix

This post is deleted!

johnpoz

You sure do not need to be listening on wan for it to go outbound and talk to the ntp servers.

jahonix

Maybe you need to check one of the entries as "Prefer"?

beremonavabi

@johnpoz said in NTP Not Working [SOLVED]:

You sure do not need to be listening on wan for it to go outbound and talk to the ntp servers.

Well, that's what I thought when I set it up initially (and it worked that way at the time). But, all I can say is that, at this time (with my current configuration on pfSense 2.4.3-Release), without WAN being selected, the NTP service doesn't work. Merely adding the WAN interface under Services > NTP causes it to immediately start working again.

EDIT: Another odd thing is that according to that quote I posted from the other thread, above, the other solution is to have the selected interface with the lowest IP address have outbound NAT configured. Well, for me, that's my LAN interface (192.168.1.0.24). Firewall > NAT > Outbound for that interface looks like:


Interface	Source	Source Port	Destination	Destination Port	NAT Address	NAT Port	Static Port	Description	Actions
		WAN	192.168.1.0/24	*	*	*	WAN address	*		LAN to WAN

So, unless I'm mis-reading that, I do have outbound NAT configured for it. Odd.

kpa

It's not as simple as you would think. NTP service uses UDP only and there is only one listening socket for sending out queries and listening for replies on UDP port 123. If you set the service not to listen on your WAN interface that will also cut off the service's ability to send anything out on the WAN interface. Yes I know, DNS (for example) does the right thing and uses a separate socket for sending out the queries with a randomized source port, unfortunately NTP hasn't caught up with the new developments yet.

jahonix

So you're saying all my setups with NTP servicing all internal networks but WAN don't work? I must have missed something then...

kpa

Look at your outbound NAT rules. There should be a rule for outbound NAT'ing anything sourced from 127.0.0.0/8. That is the thing that makes your set up still work if you happen to set the service not to listen on the WAN, the service will instead use the localhost 127.0.0.1 address for the traffic and it will be properly NAT'ed.

Edit: The service might also select your LAN interface in case WAN is not available and if the LAN interface has a routable IP address it works just fine, also if the LAN IP address is an RFC1918 address you're still good because of the standard outbound NAT.

beremonavabi

@kpa said in NTP Not Working [SOLVED]:

Look at your outbound NAT rules. There should be a rule for outbound NAT'ing anything sourced from 127.0.0.0/8. That is the thing that makes your set up still work if you happen to set the service not to listen on the WAN, the service will instead use the localhost 127.0.0.1 address for the traffic and it will be properly NAT'ed.

Interface	Source	Source Port	Destination	Destination Port	NAT Address	NAT Port	Static Port	Description	Actions
		WAN	127.0.0.0/8	*	*	*	WAN address	*		Localhost to WAN

I've got that rule, too. But, without WAN being selected for NTP, it still didn't work.

johnpoz

I do not have wan selected, and ntp works just fine to outside ntp servers.

And yeah your loopback should be outbound nat.. Unless the user turned off automatic outbound nat then yeah loopback would work and or any of other lan side interfaces. If they turned off loopback nat, pretty sure that would break a bunch of stuff.

cyberzeus

This post is deleted!

beremonavabi

Just to double-check this (in case my forcing a clock update in my troubleshooting attempts "fixed" this), I unselected WAN from the selection list on Services > NTP and restarted the NTP service. I was right back in the same situation I was when I started this thread:

The NTP Status Dashboard widget has the time, but lists the Sync Source as “No active peers available.”

Under Status > NTP, all the pools show a status of “Pool Placeholder” and non-pools show “Unreach/Pending.” They all have “Stratum” equal to 16 instead of the 1 or 2 they should be, and the “When” fields are all blank (just a dash). All the statistics are 0.
...
Under Status > System Logs > NTP, all the log entries are nothing but “Soliciting pool server…” messages.

Re-selecting the WAN interface on the list and restarting the NTP service starts everything up properly and everything works again.

When it's not working, I don't see anything in any of the logs (except for the constant "Soliciting pool server" messages in the NTP log) related to this. Anyone have any suggestions on log settings to try to track this down? I'd really prefer not having NTP listening to the WAN interface if at all possible.

jahonix

So did you check an entry as "Prefer"???

beremonavabi

@jahonix said in NTP Not Working [SOLVED]:

So did you check an entry as "Prefer"???

Yes. It made no difference.

Gertjan

@beremonavabi said in NTP Not Working [SOLVED]:

Was there a specific article you were trying to point me to?

Wow - I guess I checked in after the storm.
Indeed, I was using Google myself to find some possible related posts : ntp not being able to contact remote time servers (Firewall and gateway issues).
Clear is now : there is a something that is not 'default'.
NTP settings : I haven't checked the WAN interface - only my LAN interfaces (I thought this ensures that ntp is serving my local devices with a time server, my pfSense box). See image below.

My Outbound NAT is pretty big and pretty default :
0_1528090393664_7395ac1f-e0ed-427b-b1cb-a93807886403-image.png

My WAN IP is RFC1918 (192.168.1.10.0/24)
LAN and OPT1 are default.

My NTP settings (tghis time with WAN selected - that works also for me ):

0_1528090777859_2f9b83b6-304e-477f-b01e-e1bffe3b3f37-image.png

Btw : selecting WAN isn't big deal. As long as there is no firewall rule letting in connections, your ok.
You should know that the GUI is also listening on ALL interfaces (WAN included !).

The only thing I changed there was "fr.pool.ntp.org" (I'm living in France) and checked "this is a pool).

Did you change anything on the Services => NTP => ACLs tab ?

beremonavabi

On my WAN interface, I'm allowing IP4 UDP traffic going to port 443 (for OpenVPN clients connecting to my OpenVPN server):
0_1528120985005_20180604 -- Firewall Rules WAN.PNG
My NTP ACL tab is default: Kiss-o'-death, Modifications, Peer Association, and Trap Service are all checked, while Queries and Service are unchecked.

EDIT: And since so few of us are having this problem, there's got to be something non-default somewhere that's causing this (as you said). I just can't find it.

beremonavabi

According to:

http://support.ntp.org/bin/view/Support/TroubleshootingNTP#Section_9.8.

ntpd requires full bidirectional access to the privileged UDP port 123

I assume that's what kpa was referring to, above. So, for those of you who do have NTP synchronizing time correctly without having the WAN interface selected (apparently, most everyone), do you have a firewall rule opening port 123 to UDP traffic on your WAN?

Also, according to:

http://support.ntp.org/bin/view/Support/TroubleshootingNTP#Section_9.9.

there's supposed to be an ntp.conf file at /etc. I don't see that on my SG-4860. Does pfSense use that file for NTP?

Gertjan

@beremonavabi said in NTP Not Working [SOLVED]:

According to:

http://support.ntp.org/bin/view/Support/TroubleshootingNTP#Section_9.8.

ntpd requires full bidirectional access to the privileged UDP port 123

Yep.
Part "9.8. Check the NTP port" also states :

Bla bla bla bla ...
If this is not possible, you may need to run ntpd on the firewall itself, so that it can have full unrestricted access to UDP port >123 in both directions, and then have it serve time to your internal clients. However, this may also be disallowed.

Of course, our ntpd is running on the firewall pfSense with 'root' privileges, so it can snap to this "123" port, so it can send out requests , and receives the replies.

I assume that's what kpa was referring to, above. So, for those of you who do have NTP synchronizing time correctly without having the WAN interface selected (apparently, most everyone), do you have a firewall rule opening port 123 to UDP traffic on your WAN?

No way !!!!
As seen above, the ntpd process can go outside anytime it wants to do so - connect itself as a "client" to a ntp "server". These connections are outbound.
Firewall rules on the (WAN) interface are for inbound connection only.
I have no (well, yes : one : an incoming VPN rule) firewall rules on WAN, so by default no initial connection comes in.
NTP servers like fr.pool.ntp.org do not connect to my pfSEnse. It's my pfSense that connects to fr.pool.ntp.org.

Also, according to:
http://support.ntp.org/bin/view/Support/TroubleshootingNTP#Section_9.9.
there's supposed to be an ntp.conf file at /etc. I don't see that on my SG-4860. Does pfSense use that file for NTP?

Yes, no ... this is FreeBSD.
See your post number 4 here ( you !) told us were we (you !) can find this file ^^
As you said yourself over there : the file is here : /var/etc/ntpd.conf

beremonavabi

One more attempt at figuring this out.

I saw a bug report that was similar (though it involved CARP):

https://redmine.pfsense.org/issues/5548

The response was:

You're breaking NTP connectivity on the backup by sending the traffic using a CARP IP. It won't, and can't, receive those replies - they go to the primary. When WAN isn't bound, it's probably hitting NAT to a CARP IP because it has a private source IP. NAT it to the WAN IP in that case.

The OP responded:

Added the following NAT rules at the top of the Outbound manual rules list:

Interfc Source Src Pt Dest Dest Pt NAT Addr NAT Pt Static Description 
WAN1 This Fw udp/* * udp/123 WAN1 addr * NO NTP to WAN1 INTFC IP

and that fixed it.

I was wondering if my restricting all outbound traffic to go through my VPN might be causing a similar problem here. So, I added the following outbound NAT rule (even though it ought to have been covered by my existing NAT rules and I've got "redirect-gateway def1; in my VPN client Custom Options to make sure the firewall, itself, can get out the default interface) which, I think, is an equivalent:

0_1528665541541_20180610 -- pfSense NTP Fix Question.PNG

It doesn't make any difference. NTP still won't start without the WAN being selected. Can anyone confirm that these situations might be equivalent and that the NAT rule I added makes some kind of sense?

stephenw10

Can we see your full outbound NAT page?

Steve