NTP Server not synchronizing if localhost selected

tjsober · Jan 29, 2022, 9:14 PM

I’m having a problem with NTP that seems related to NAT. If under Services - NTP, I either select all interfaces or none of the interfaces, NTP works fine. If I select a subset of interfaces that does not include WAN, NTP works fine as long as that subset doesn’t also include localhost. If localhost is included and WAN is not included, NTP breaks and will not synchronize.

I have read about how NTP "selects" the lowest numbered interface for synchronizing but that doesn't see to be the full issue.

I have read through a number of posts and reached the conclusion Outbound NAT is involved because if I use the instructions jimp posted in the link below and switch from Automatic to Hybrid Outbound NAT and add this rule, it starts to work regardless of which networks are selected in the NTP server settings.

https://forum.netgate.com/topic/131506/ntp-not-working-solved-totally/38

Firewall > NAT, Outbound tab. Add rule to top.
• Disabled: Unchecked
• Do not NAT: Unchecked
• Interface: WAN (make one of these rules for each WAN)
• Protocol: any
• Source: This Firewall (self)
• Destination: any
• Not: Unchecked
• Translation Address: Interface Address
• Port or Range: Blank
• Description: NAT anything out from the firewall itself

So what is puzzling me is I have another pfsense firewall where localhost is selected in the NTP server settings, WAN is not selected, and NTP works. Both firewalls are set for Automatic Outbound NAT rule generation and 127.0.0.0/8 ::1/128 is in both automatic rules (along with all of the other networks on the device). So the Automatic Rules “look” correct to me (at least identical) between the two firewalls, so I’m at a loss to explain why one firewall needs Outbound NAT to be configured differently. But I will admit I don’t fully understand the NAT rules. The Automatic rules for Outbound NAT have always worked on other devices.

I only have a single WAN and the gateway is set by DHCP. pfBlockerNG is running. Not running Snort or Suricata.

Obviously I have something misconfigured but it is to the point where I am not seeing it and I and by no means an expert…just a struggling hobbyist. Any help would be appreciated.

stephenw10 · Jan 30, 2022, 3:21 PM

Mmm, I would expect the auto outbound rules to work there.
One firewall has IPv6 maybe?

There must be some difference between them.

Steve

tjsober · Jan 30, 2022, 3:38 PM

@stephenw10 I agree and I'm probably overlooking something obvious. I just checked and on both firewalls - VP2410 pfSense 2.5.2 (NTP doesn't work "properly") and SG-5100 pfSense+ 25.05.2 (NTP works without added NAT rule or WAN selection) the System - Advanced - Networking - Allow IPV6 box is unchecked.

I would greatly appreciate any other suggestions. I have an SG-3100 and and SG-1100 in the basement that I'm going to set up and test this as soon as I have time (I have to keep the network up so my wife has work access, so maintenance opportunities vary). I (vaguely) recall this also happening on my son's SG-2220 several years ago but leaving WAN selected was no biggie as there were no firewall rules that would have allowed access to NTP from the WAN.

If I switch to manual NAT rules and delete them all and then switch back to Auto, will the Auto NAT Rules be regenerated? I'm asking because the configuration on the VP2410 came from the SG-3100 and it is running 2.4.5_p1. The configuration seemed to come over fine but I thought I would ask.

stephenw10 · Jan 30, 2022, 3:50 PM

@tjsober said in NTP Server not synchronizing if localhost selected:

f I switch to manual NAT rules and delete them all and then switch back to Auto, will the Auto NAT Rules be regenerated?

Yes. The auto rules will be regenerated based on the actual subnets on the box at that time.

johnpoz · Jan 30, 2022, 3:59 PM

The interfaces you select in ntp service gui are for which interface ntp server listens on - I don't see how that would really have anything to do with ntp syncing with the ntp servers you are wanting to sync with?

Interfaces without an IP address will not be shown.
Selecting no interfaces will listen on all interfaces with a wildcard.
Selecting all interfaces will explicitly listen on only the interfaces/IPs specified.

stephenw10 · Jan 30, 2022, 4:24 PM

The interface selected there is also what ntp binds to and uses for queries:

Select the interface(s) to use for NTP. The NTP daemon binds to all interfaces by default to receive replies properly. This may be minimized by selecting at least one interface to bind, but that interface will also be used to source the NTP queries sent out to remote servers, not only to serve clients.

johnpoz · Jan 30, 2022, 5:56 PM

@stephenw10 ah.. ok that makes sense.. But yeah even if did localhost as source, normally the outbound nat would adjust that.. Sure wouldn't work using loopback as source ;)

tjsober · Jan 30, 2022, 7:19 PM

@johnpoz And that's my suspicion, although there is still something odd that I clearly don't understand. For some reason it is binding to 127.0.0.0 because it is a lower number than my LAN's (169) or my WAN (<100). I recall reading on one of the forum posts that for outgoing query's NTP binds to the lowest numbered address (or was it interface? I wish I could find that post now). I'm going to try redoing the Outgoing NAT rules as soon as I can and see if that makes a difference. Thanks!

johnpoz · Jan 30, 2022, 7:22 PM

@tjsober said in NTP Server not synchronizing if localhost selected:

LAN's (169)

I sure hope your lan is not a 169.254 address.. that going to have all kinds of issues as well - that is a link local sort of address.. Without some changes I don't think pfsense would even route or nat that.. I think there is a checkbox somewhere to allow for that? But its not a good choice to use such addressing.

tjsober · Jan 30, 2022, 7:24 PM

@johnpoz God. Sorry. Trying to multitask and failing. Not sure I can even blame dyslexia...

All LAN's and VLAN's are on 192.168.x.x addresses.

Can't get help if I'm not accurate :( Apologies.

johnpoz · Jan 30, 2022, 7:28 PM

@tjsober hahah - no problem.. But good to hear not trying to use 169.254, that range is suppose to have like ttl of 1, and not route, etc. etc. There are some special use cases where you might "have" to use it because your forced by some nonsense - I think they did some stuff recently to allow for such craziness ;)

tjsober · Jan 31, 2022, 1:36 PM

@stephenw10 It was a long shot but I tried switching to manual NAT, deleting the rules, and then switching back to automatic. No change in the behavior. Very odd.

I guess I just leave localhost out of the NTP network selection list. Am I breaking something else internal to the FW if it isn't selected?

johnpoz · Jan 31, 2022, 1:41 PM

@tjsober what ntp server are you trying to talk too? If your ntp server is local to your network there would be no nat done talking to some internal IP via your lan side interfaces.

tjsober · Jan 31, 2022, 1:44 PM

@johnpoz Not local. Servers are below with only the "is a pool" checkbox marked.

0.us.pool.ntp.org
1.us.pool.ntp.org
2.us.pool.ntp.org
3.us.pool.ntp.org

Everything else on the NTP server page is default.

johnpoz · Jan 31, 2022, 1:58 PM

@tjsober so can you sniff on your wan - do you see this traffic going out? When you have localhost selected - and its going out with 127.0.0.1 as the source?

tjsober · Jan 31, 2022, 2:05 PM

@johnpoz That would be quite informative. However, I don't think I have that capability as I would need to sniff between the modem and the pfsense box and I do not think I can put another device there...hmmm. Maybe I can by temporarily using one of the other firewalls I have lying around. I'll have to make sure a switch doesn't hid the traffic from me. I might have an old hub I can use for that.

Thanks...there might be a way. It will take a bit as I have to keep the network up most of the time. Appreciate the suggestion.

johnpoz · Jan 31, 2022, 2:07 PM

@tjsober just sniff on pfsense using diagnostic packet capture.

tjsober · Jan 31, 2022, 2:09 PM

@johnpoz Obviously I didn't know I could do that :) Looking at that page now. Thanks.

tjsober · Jan 31, 2022, 2:37 PM

@johnpoz Hopefully I am doing this right. But in either case I don't see 127.0.0.0 coming out of the WAN.

If I have localhost selected in the NTP server settings and set up the packet capture on the external WAN port 123, the capture shows a very small number of queries (about every 5 seconds but it varies...some captures came up with fewer even though I waited a longer time) and the NTP status page only has the 4 pool placeholders (I obfuscated my external address).

08:20:11.131407 IP XXX.XXX.XXX.XXX.23207 > 91.189.91.157.123: UDP, length 48
08:20:11.188443 IP 91.189.91.157.123 > XXX.XXX.XXX.XXX.23207: UDP, length 48
08:20:16.429680 IP XXX.XXX.XXX.XXX.10587 > 216.239.35.4.123: UDP, length 48
08:20:16.488484 IP 216.239.35.4.123 > XXX.XXX.XXX.XXX.10587: UDP, length 48
08:20:16.631163 IP XXX.XXX.XXX.XXX.61541 > 216.239.35.0.123: UDP, length 48
08:20:16.662519 IP 216.239.35.0.123 > XXX.XXX.XXX.XXX.61541: UDP, length 48

If I unselect localhost on the server and repeat the same packet capture settings, I have a bunch of traffic with multiple queries per second to the hosts that are listed on the NTP status page and the status page look good (increasing reach).

08:15:02.385915 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:02.401901 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:02.405044 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:02.405793 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:02.410820 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:02.485883 IP 38.229.52.9.123 > XXX.XXX.XXX.XXX.12162: UDP, length 48
08:15:03.333342 IP XXX.XXX.XXX.XXX.39530 > 74.6.168.72.123: UDP, length 48
08:15:03.333352 IP XXX.XXX.XXX.XXX.2826 > 204.2.134.162.123: UDP, length 48
08:15:03.333364 IP XXX.XXX.XXX.XXX.6699 > 45.79.111.114.123: UDP, length 48
08:15:03.395559 IP 45.79.111.114.123 > XXX.XXX.XXX.XXX.6699: UDP, length 48
08:15:03.402028 IP 204.2.134.162.123 > XXX.XXX.XXX.XXX.2826: UDP, length 48
08:15:03.411953 IP 74.6.168.72.123 > XXX.XXX.XXX.XXX.39530: UDP, length 48
08:15:04.316106 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:04.316116 IP XXX.XXX.XXX.XXX.13967 > 192.48.105.15.123: UDP, length 48
08:15:04.316122 IP XXX.XXX.XXX.XXX.12162 > 38.229.52.9.123: UDP, length 48
08:15:04.316131 IP XXX.XXX.XXX.XXX.35551 > 107.194.210.155.123: UDP, length 48
08:15:04.316137 IP XXX.XXX.XXX.XXX.58278 > 38.229.56.9.123: UDP, length 48
08:15:04.316146 IP XXX.XXX.XXX.XXX.26863 > 64.79.100.196.123: UDP, length 48
08:15:04.316152 IP XXX.XXX.XXX.XXX.4376 > 142.147.88.111.123: UDP, length 48
08:15:04.354872 IP 38.229.56.9.123 > XXX.XXX.XXX.XXX.58278: UDP, length 48
08:15:04.355546 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:04.370874 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:04.374696 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:04.375746 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:04.380672 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:04.458872 IP 38.229.52.9.123 > XXX.XXX.XXX.XXX.12162: UDP, length 48
08:15:05.318165 IP XXX.XXX.XXX.XXX.39530 > 74.6.168.72.123: UDP, length 48
08:15:05.318175 IP XXX.XXX.XXX.XXX.2826 > 204.2.134.162.123: UDP, length 48
08:15:05.318184 IP XXX.XXX.XXX.XXX.6699 > 45.79.111.114.123: UDP, length 48
08:15:05.381470 IP 45.79.111.114.123 > XXX.XXX.XXX.XXX.6699: UDP, length 48
08:15:05.386845 IP 204.2.134.162.123 > XXX.XXX.XXX.XXX.2826: UDP, length 48
08:15:05.393770 IP 74.6.168.72.123 > XXX.XXX.XXX.XXX.39530: UDP, length 48
08:15:06.334311 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:06.334322 IP XXX.XXX.XXX.XXX.13967 > 192.48.105.15.123: UDP, length 48
08:15:06.334330 IP XXX.XXX.XXX.XXX.12162 > 38.229.52.9.123: UDP, length 48
08:15:06.334337 IP XXX.XXX.XXX.XXX.35551 > 107.194.210.155.123: UDP, length 48
08:15:06.334345 IP XXX.XXX.XXX.XXX.58278 > 38.229.56.9.123: UDP, length 48
08:15:06.334352 IP XXX.XXX.XXX.XXX.26863 > 64.79.100.196.123: UDP, length 48
08:15:06.334359 IP XXX.XXX.XXX.XXX.4376 > 142.147.88.111.123: UDP, length 48
08:15:06.371777 IP 38.229.56.9.123 > XXX.XXX.XXX.XXX.58278: UDP, length 48
08:15:06.373076 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:06.388750 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:06.392176 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:06.396649 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:06.402724 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:06.476775 IP 38.229.52.9.123 > XXX.XXX.XXX.XXX.12162: UDP, length 48
08:15:07.327668 IP XXX.XXX.XXX.XXX.39530 > 74.6.168.72.123: UDP, length 48
08:15:07.327680 IP XXX.XXX.XXX.XXX.2826 > 204.2.134.162.123: UDP, length 48
08:15:07.396804 IP 204.2.134.162.123 > XXX.XXX.XXX.XXX.2826: UDP, length 48
08:15:07.406730 IP 74.6.168.72.123 > XXX.XXX.XXX.XXX.39530: UDP, length 48
08:15:07.413128 IP XXX.XXX.XXX.XXX.16773 > 216.239.35.12.123: UDP, length 48
08:15:07.446754 IP 216.239.35.12.123 > XXX.XXX.XXX.XXX.16773: UDP, length 48
08:15:08.331466 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:08.331482 IP XXX.XXX.XXX.XXX.13967 > 192.48.105.15.123: UDP, length 48
08:15:08.331494 IP XXX.XXX.XXX.XXX.35551 > 107.194.210.155.123: UDP, length 48
08:15:08.331506 IP XXX.XXX.XXX.XXX.26863 > 64.79.100.196.123: UDP, length 48
08:15:08.331517 IP XXX.XXX.XXX.XXX.4376 > 142.147.88.111.123: UDP, length 48
08:15:08.370885 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:08.388662 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:08.389940 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:08.390686 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:08.395610 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:10.315957 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:10.379708 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48

I'm not sure how to interpret the difference in query rate.

tjsober · Jan 31, 2022, 2:43 PM

@johnpoz So I think I figured out that the small number of queries are coming from a couple of the linux boxes I have on my network and not from the firewall. They are not currently configured to get time from pfSense. (Edit: also probably a chromecast as google is in there)