NTP Server not synchronizing if localhost selected

stephenw10

@tjsober said in NTP Server not synchronizing if localhost selected:

f I switch to manual NAT rules and delete them all and then switch back to Auto, will the Auto NAT Rules be regenerated?

Yes. The auto rules will be regenerated based on the actual subnets on the box at that time.

johnpoz

The interfaces you select in ntp service gui are for which interface ntp server listens on - I don't see how that would really have anything to do with ntp syncing with the ntp servers you are wanting to sync with?

Interfaces without an IP address will not be shown.
Selecting no interfaces will listen on all interfaces with a wildcard.
Selecting all interfaces will explicitly listen on only the interfaces/IPs specified.

stephenw10

The interface selected there is also what ntp binds to and uses for queries:

Select the interface(s) to use for NTP. The NTP daemon binds to all interfaces by default to receive replies properly. This may be minimized by selecting at least one interface to bind, but that interface will also be used to source the NTP queries sent out to remote servers, not only to serve clients.

johnpoz

@stephenw10 ah.. ok that makes sense.. But yeah even if did localhost as source, normally the outbound nat would adjust that.. Sure wouldn't work using loopback as source ;)

tjsober

@johnpoz And that's my suspicion, although there is still something odd that I clearly don't understand. For some reason it is binding to 127.0.0.0 because it is a lower number than my LAN's (169) or my WAN (<100). I recall reading on one of the forum posts that for outgoing query's NTP binds to the lowest numbered address (or was it interface? I wish I could find that post now). I'm going to try redoing the Outgoing NAT rules as soon as I can and see if that makes a difference. Thanks!

johnpoz

@tjsober said in NTP Server not synchronizing if localhost selected:

LAN's (169)

I sure hope your lan is not a 169.254 address.. that going to have all kinds of issues as well - that is a link local sort of address.. Without some changes I don't think pfsense would even route or nat that.. I think there is a checkbox somewhere to allow for that? But its not a good choice to use such addressing.

tjsober

@johnpoz God. Sorry. Trying to multitask and failing. Not sure I can even blame dyslexia...

All LAN's and VLAN's are on 192.168.x.x addresses.

Can't get help if I'm not accurate :( Apologies.

johnpoz

@tjsober hahah - no problem.. But good to hear not trying to use 169.254, that range is suppose to have like ttl of 1, and not route, etc. etc. There are some special use cases where you might "have" to use it because your forced by some nonsense - I think they did some stuff recently to allow for such craziness ;)

tjsober

@stephenw10 It was a long shot but I tried switching to manual NAT, deleting the rules, and then switching back to automatic. No change in the behavior. Very odd.

I guess I just leave localhost out of the NTP network selection list. Am I breaking something else internal to the FW if it isn't selected?

johnpoz

@tjsober what ntp server are you trying to talk too? If your ntp server is local to your network there would be no nat done talking to some internal IP via your lan side interfaces.

tjsober

@johnpoz Not local. Servers are below with only the "is a pool" checkbox marked.

0.us.pool.ntp.org
1.us.pool.ntp.org
2.us.pool.ntp.org
3.us.pool.ntp.org

Everything else on the NTP server page is default.

johnpoz

@tjsober so can you sniff on your wan - do you see this traffic going out? When you have localhost selected - and its going out with 127.0.0.1 as the source?

tjsober

@johnpoz That would be quite informative. However, I don't think I have that capability as I would need to sniff between the modem and the pfsense box and I do not think I can put another device there...hmmm. Maybe I can by temporarily using one of the other firewalls I have lying around. I'll have to make sure a switch doesn't hid the traffic from me. I might have an old hub I can use for that.

Thanks...there might be a way. It will take a bit as I have to keep the network up most of the time. Appreciate the suggestion.

johnpoz

@tjsober just sniff on pfsense using diagnostic packet capture.

tjsober

@johnpoz Obviously I didn't know I could do that :) Looking at that page now. Thanks.

tjsober

@johnpoz Hopefully I am doing this right. But in either case I don't see 127.0.0.0 coming out of the WAN.

If I have localhost selected in the NTP server settings and set up the packet capture on the external WAN port 123, the capture shows a very small number of queries (about every 5 seconds but it varies...some captures came up with fewer even though I waited a longer time) and the NTP status page only has the 4 pool placeholders (I obfuscated my external address).

08:20:11.131407 IP XXX.XXX.XXX.XXX.23207 > 91.189.91.157.123: UDP, length 48
08:20:11.188443 IP 91.189.91.157.123 > XXX.XXX.XXX.XXX.23207: UDP, length 48
08:20:16.429680 IP XXX.XXX.XXX.XXX.10587 > 216.239.35.4.123: UDP, length 48
08:20:16.488484 IP 216.239.35.4.123 > XXX.XXX.XXX.XXX.10587: UDP, length 48
08:20:16.631163 IP XXX.XXX.XXX.XXX.61541 > 216.239.35.0.123: UDP, length 48
08:20:16.662519 IP 216.239.35.0.123 > XXX.XXX.XXX.XXX.61541: UDP, length 48

If I unselect localhost on the server and repeat the same packet capture settings, I have a bunch of traffic with multiple queries per second to the hosts that are listed on the NTP status page and the status page look good (increasing reach).

08:15:02.385915 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:02.401901 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:02.405044 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:02.405793 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:02.410820 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:02.485883 IP 38.229.52.9.123 > XXX.XXX.XXX.XXX.12162: UDP, length 48
08:15:03.333342 IP XXX.XXX.XXX.XXX.39530 > 74.6.168.72.123: UDP, length 48
08:15:03.333352 IP XXX.XXX.XXX.XXX.2826 > 204.2.134.162.123: UDP, length 48
08:15:03.333364 IP XXX.XXX.XXX.XXX.6699 > 45.79.111.114.123: UDP, length 48
08:15:03.395559 IP 45.79.111.114.123 > XXX.XXX.XXX.XXX.6699: UDP, length 48
08:15:03.402028 IP 204.2.134.162.123 > XXX.XXX.XXX.XXX.2826: UDP, length 48
08:15:03.411953 IP 74.6.168.72.123 > XXX.XXX.XXX.XXX.39530: UDP, length 48
08:15:04.316106 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:04.316116 IP XXX.XXX.XXX.XXX.13967 > 192.48.105.15.123: UDP, length 48
08:15:04.316122 IP XXX.XXX.XXX.XXX.12162 > 38.229.52.9.123: UDP, length 48
08:15:04.316131 IP XXX.XXX.XXX.XXX.35551 > 107.194.210.155.123: UDP, length 48
08:15:04.316137 IP XXX.XXX.XXX.XXX.58278 > 38.229.56.9.123: UDP, length 48
08:15:04.316146 IP XXX.XXX.XXX.XXX.26863 > 64.79.100.196.123: UDP, length 48
08:15:04.316152 IP XXX.XXX.XXX.XXX.4376 > 142.147.88.111.123: UDP, length 48
08:15:04.354872 IP 38.229.56.9.123 > XXX.XXX.XXX.XXX.58278: UDP, length 48
08:15:04.355546 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:04.370874 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:04.374696 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:04.375746 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:04.380672 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:04.458872 IP 38.229.52.9.123 > XXX.XXX.XXX.XXX.12162: UDP, length 48
08:15:05.318165 IP XXX.XXX.XXX.XXX.39530 > 74.6.168.72.123: UDP, length 48
08:15:05.318175 IP XXX.XXX.XXX.XXX.2826 > 204.2.134.162.123: UDP, length 48
08:15:05.318184 IP XXX.XXX.XXX.XXX.6699 > 45.79.111.114.123: UDP, length 48
08:15:05.381470 IP 45.79.111.114.123 > XXX.XXX.XXX.XXX.6699: UDP, length 48
08:15:05.386845 IP 204.2.134.162.123 > XXX.XXX.XXX.XXX.2826: UDP, length 48
08:15:05.393770 IP 74.6.168.72.123 > XXX.XXX.XXX.XXX.39530: UDP, length 48
08:15:06.334311 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:06.334322 IP XXX.XXX.XXX.XXX.13967 > 192.48.105.15.123: UDP, length 48
08:15:06.334330 IP XXX.XXX.XXX.XXX.12162 > 38.229.52.9.123: UDP, length 48
08:15:06.334337 IP XXX.XXX.XXX.XXX.35551 > 107.194.210.155.123: UDP, length 48
08:15:06.334345 IP XXX.XXX.XXX.XXX.58278 > 38.229.56.9.123: UDP, length 48
08:15:06.334352 IP XXX.XXX.XXX.XXX.26863 > 64.79.100.196.123: UDP, length 48
08:15:06.334359 IP XXX.XXX.XXX.XXX.4376 > 142.147.88.111.123: UDP, length 48
08:15:06.371777 IP 38.229.56.9.123 > XXX.XXX.XXX.XXX.58278: UDP, length 48
08:15:06.373076 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:06.388750 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:06.392176 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:06.396649 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:06.402724 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:06.476775 IP 38.229.52.9.123 > XXX.XXX.XXX.XXX.12162: UDP, length 48
08:15:07.327668 IP XXX.XXX.XXX.XXX.39530 > 74.6.168.72.123: UDP, length 48
08:15:07.327680 IP XXX.XXX.XXX.XXX.2826 > 204.2.134.162.123: UDP, length 48
08:15:07.396804 IP 204.2.134.162.123 > XXX.XXX.XXX.XXX.2826: UDP, length 48
08:15:07.406730 IP 74.6.168.72.123 > XXX.XXX.XXX.XXX.39530: UDP, length 48
08:15:07.413128 IP XXX.XXX.XXX.XXX.16773 > 216.239.35.12.123: UDP, length 48
08:15:07.446754 IP 216.239.35.12.123 > XXX.XXX.XXX.XXX.16773: UDP, length 48
08:15:08.331466 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:08.331482 IP XXX.XXX.XXX.XXX.13967 > 192.48.105.15.123: UDP, length 48
08:15:08.331494 IP XXX.XXX.XXX.XXX.35551 > 107.194.210.155.123: UDP, length 48
08:15:08.331506 IP XXX.XXX.XXX.XXX.26863 > 64.79.100.196.123: UDP, length 48
08:15:08.331517 IP XXX.XXX.XXX.XXX.4376 > 142.147.88.111.123: UDP, length 48
08:15:08.370885 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:08.388662 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:08.389940 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:08.390686 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:08.395610 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:10.315957 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:10.379708 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48

I'm not sure how to interpret the difference in query rate.

tjsober

@johnpoz So I think I figured out that the small number of queries are coming from a couple of the linux boxes I have on my network and not from the firewall. They are not currently configured to get time from pfSense. (Edit: also probably a chromecast as google is in there)

johnpoz

@tjsober yeah I was going to try and duplicate your problem. But I have loads of ntp traffic back and forth since I provide a ntp server to the ntp pool. I would have to block my ntp server from answering, etc. Or it would be hard to weed through all the ntp traffic.

But I don't see any 127.0.0.1 as source traffic there.. So doesn't seem like pfsense is not natting to yoru public IP even when localhost is selected?

edit: But what I did do is just select localhost, and then pointed it to pool.ntp.org and that seems to be broken.

edit2: If I select the local interfaces I want to provide time to, and localhost - its seems to be working

edit3: Why are you wanting to select localhost? I don't really see a reason to do that, other than you would think it should be fine to do..

tjsober

@johnpoz I figured something out. I disabled pfBlockerNG and restarted NTP Server with localhost selected in the NTP settings. And guess what? NTP syncs fine.

Then I turned pfblocker back on and NTP continued to sync. So I restarted the NTP server and it broke.

I found this because put in some rules to lock down NTP so the linux boxes couldn't reach out and repeated the packet capture. And the IP that showed up in the capture was 10.10.10.1... the pfBlocker VIP. That's not going to work!

Isn't it always what is left out of the problem description that is the issue...and I tried to be complete :)

I have to start looking at pfblocker to see how it is messing with localhost or why the NTP server is trying to sync using the pfblocker VIP. That's probably going to be a can of worms. Interestingly, it should be configured the same as the SG-5100 where I don't have this issue. But I'll have to look closer as there are some differences (currently no VLAN's on the SG5100 for example).

I still don't understand why the port forward rule in the post I originally linked fixes this. And I am back to wondering if anything matters if localhost isn't bound to the NTP server. The rabbit hole just keeps getting deeper.

Thank you so much for taking the time to teach me some things. Your patience is appreciated.

johnpoz

@tjsober said in NTP Server not synchronizing if localhost selected:

10.10.10.1... the pfBlocker VIP. That's not going to work!

nope that wouldn't work ;) hmmm I would have to change back and sniff to see if have same thing.. But I am not using pfblocker in that fashion, I have no vip setup with it. I just use it to manipulate some aliases that I use in my firewall.

Rabbit holes can be fun, if you ever get to the end.. Other times they can be real head scratchers ;)

edit: see I don't have that feature enabled in pfblocker

It shouldn't be creating that vip if I don't have that enabled - have to take a double check look see.

edit2: I just looked and there is no vip on my loopback interface.. I do see my other vip on my wan interface, etc. But no 10.10.10.1 anywhere on any interface..