NTP Server not synchronizing if localhost selected

tjsober · Jan 31, 2022, 2:37 PM

@johnpoz Hopefully I am doing this right. But in either case I don't see 127.0.0.0 coming out of the WAN.

If I have localhost selected in the NTP server settings and set up the packet capture on the external WAN port 123, the capture shows a very small number of queries (about every 5 seconds but it varies...some captures came up with fewer even though I waited a longer time) and the NTP status page only has the 4 pool placeholders (I obfuscated my external address).

08:20:11.131407 IP XXX.XXX.XXX.XXX.23207 > 91.189.91.157.123: UDP, length 48
08:20:11.188443 IP 91.189.91.157.123 > XXX.XXX.XXX.XXX.23207: UDP, length 48
08:20:16.429680 IP XXX.XXX.XXX.XXX.10587 > 216.239.35.4.123: UDP, length 48
08:20:16.488484 IP 216.239.35.4.123 > XXX.XXX.XXX.XXX.10587: UDP, length 48
08:20:16.631163 IP XXX.XXX.XXX.XXX.61541 > 216.239.35.0.123: UDP, length 48
08:20:16.662519 IP 216.239.35.0.123 > XXX.XXX.XXX.XXX.61541: UDP, length 48

If I unselect localhost on the server and repeat the same packet capture settings, I have a bunch of traffic with multiple queries per second to the hosts that are listed on the NTP status page and the status page look good (increasing reach).

08:15:02.385915 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:02.401901 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:02.405044 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:02.405793 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:02.410820 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:02.485883 IP 38.229.52.9.123 > XXX.XXX.XXX.XXX.12162: UDP, length 48
08:15:03.333342 IP XXX.XXX.XXX.XXX.39530 > 74.6.168.72.123: UDP, length 48
08:15:03.333352 IP XXX.XXX.XXX.XXX.2826 > 204.2.134.162.123: UDP, length 48
08:15:03.333364 IP XXX.XXX.XXX.XXX.6699 > 45.79.111.114.123: UDP, length 48
08:15:03.395559 IP 45.79.111.114.123 > XXX.XXX.XXX.XXX.6699: UDP, length 48
08:15:03.402028 IP 204.2.134.162.123 > XXX.XXX.XXX.XXX.2826: UDP, length 48
08:15:03.411953 IP 74.6.168.72.123 > XXX.XXX.XXX.XXX.39530: UDP, length 48
08:15:04.316106 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:04.316116 IP XXX.XXX.XXX.XXX.13967 > 192.48.105.15.123: UDP, length 48
08:15:04.316122 IP XXX.XXX.XXX.XXX.12162 > 38.229.52.9.123: UDP, length 48
08:15:04.316131 IP XXX.XXX.XXX.XXX.35551 > 107.194.210.155.123: UDP, length 48
08:15:04.316137 IP XXX.XXX.XXX.XXX.58278 > 38.229.56.9.123: UDP, length 48
08:15:04.316146 IP XXX.XXX.XXX.XXX.26863 > 64.79.100.196.123: UDP, length 48
08:15:04.316152 IP XXX.XXX.XXX.XXX.4376 > 142.147.88.111.123: UDP, length 48
08:15:04.354872 IP 38.229.56.9.123 > XXX.XXX.XXX.XXX.58278: UDP, length 48
08:15:04.355546 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:04.370874 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:04.374696 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:04.375746 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:04.380672 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:04.458872 IP 38.229.52.9.123 > XXX.XXX.XXX.XXX.12162: UDP, length 48
08:15:05.318165 IP XXX.XXX.XXX.XXX.39530 > 74.6.168.72.123: UDP, length 48
08:15:05.318175 IP XXX.XXX.XXX.XXX.2826 > 204.2.134.162.123: UDP, length 48
08:15:05.318184 IP XXX.XXX.XXX.XXX.6699 > 45.79.111.114.123: UDP, length 48
08:15:05.381470 IP 45.79.111.114.123 > XXX.XXX.XXX.XXX.6699: UDP, length 48
08:15:05.386845 IP 204.2.134.162.123 > XXX.XXX.XXX.XXX.2826: UDP, length 48
08:15:05.393770 IP 74.6.168.72.123 > XXX.XXX.XXX.XXX.39530: UDP, length 48
08:15:06.334311 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:06.334322 IP XXX.XXX.XXX.XXX.13967 > 192.48.105.15.123: UDP, length 48
08:15:06.334330 IP XXX.XXX.XXX.XXX.12162 > 38.229.52.9.123: UDP, length 48
08:15:06.334337 IP XXX.XXX.XXX.XXX.35551 > 107.194.210.155.123: UDP, length 48
08:15:06.334345 IP XXX.XXX.XXX.XXX.58278 > 38.229.56.9.123: UDP, length 48
08:15:06.334352 IP XXX.XXX.XXX.XXX.26863 > 64.79.100.196.123: UDP, length 48
08:15:06.334359 IP XXX.XXX.XXX.XXX.4376 > 142.147.88.111.123: UDP, length 48
08:15:06.371777 IP 38.229.56.9.123 > XXX.XXX.XXX.XXX.58278: UDP, length 48
08:15:06.373076 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:06.388750 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:06.392176 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:06.396649 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:06.402724 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:06.476775 IP 38.229.52.9.123 > XXX.XXX.XXX.XXX.12162: UDP, length 48
08:15:07.327668 IP XXX.XXX.XXX.XXX.39530 > 74.6.168.72.123: UDP, length 48
08:15:07.327680 IP XXX.XXX.XXX.XXX.2826 > 204.2.134.162.123: UDP, length 48
08:15:07.396804 IP 204.2.134.162.123 > XXX.XXX.XXX.XXX.2826: UDP, length 48
08:15:07.406730 IP 74.6.168.72.123 > XXX.XXX.XXX.XXX.39530: UDP, length 48
08:15:07.413128 IP XXX.XXX.XXX.XXX.16773 > 216.239.35.12.123: UDP, length 48
08:15:07.446754 IP 216.239.35.12.123 > XXX.XXX.XXX.XXX.16773: UDP, length 48
08:15:08.331466 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:08.331482 IP XXX.XXX.XXX.XXX.13967 > 192.48.105.15.123: UDP, length 48
08:15:08.331494 IP XXX.XXX.XXX.XXX.35551 > 107.194.210.155.123: UDP, length 48
08:15:08.331506 IP XXX.XXX.XXX.XXX.26863 > 64.79.100.196.123: UDP, length 48
08:15:08.331517 IP XXX.XXX.XXX.XXX.4376 > 142.147.88.111.123: UDP, length 48
08:15:08.370885 IP 64.79.100.196.123 > XXX.XXX.XXX.XXX.26863: UDP, length 48
08:15:08.388662 IP 107.194.210.155.123 > XXX.XXX.XXX.XXX.35551: UDP, length 48
08:15:08.389940 IP 142.147.88.111.123 > XXX.XXX.XXX.XXX.4376: UDP, length 48
08:15:08.390686 IP 192.48.105.15.123 > XXX.XXX.XXX.XXX.13967: UDP, length 48
08:15:08.395610 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48
08:15:10.315957 IP XXX.XXX.XXX.XXX.12090 > 50.205.57.38.123: UDP, length 48
08:15:10.379708 IP 50.205.57.38.123 > XXX.XXX.XXX.XXX.12090: UDP, length 48

I'm not sure how to interpret the difference in query rate.

tjsober · Jan 31, 2022, 2:43 PM

@johnpoz So I think I figured out that the small number of queries are coming from a couple of the linux boxes I have on my network and not from the firewall. They are not currently configured to get time from pfSense. (Edit: also probably a chromecast as google is in there)

johnpoz · Jan 31, 2022, 3:18 PM

@tjsober yeah I was going to try and duplicate your problem. But I have loads of ntp traffic back and forth since I provide a ntp server to the ntp pool. I would have to block my ntp server from answering, etc. Or it would be hard to weed through all the ntp traffic.

But I don't see any 127.0.0.1 as source traffic there.. So doesn't seem like pfsense is not natting to yoru public IP even when localhost is selected?

edit: But what I did do is just select localhost, and then pointed it to pool.ntp.org and that seems to be broken.

edit2: If I select the local interfaces I want to provide time to, and localhost - its seems to be working

edit3: Why are you wanting to select localhost? I don't really see a reason to do that, other than you would think it should be fine to do..

tjsober · Jan 31, 2022, 3:19 PM

@johnpoz I figured something out. I disabled pfBlockerNG and restarted NTP Server with localhost selected in the NTP settings. And guess what? NTP syncs fine.

Then I turned pfblocker back on and NTP continued to sync. So I restarted the NTP server and it broke.

I found this because put in some rules to lock down NTP so the linux boxes couldn't reach out and repeated the packet capture. And the IP that showed up in the capture was 10.10.10.1... the pfBlocker VIP. That's not going to work!

Isn't it always what is left out of the problem description that is the issue...and I tried to be complete :)

I have to start looking at pfblocker to see how it is messing with localhost or why the NTP server is trying to sync using the pfblocker VIP. That's probably going to be a can of worms. Interestingly, it should be configured the same as the SG-5100 where I don't have this issue. But I'll have to look closer as there are some differences (currently no VLAN's on the SG5100 for example).

I still don't understand why the port forward rule in the post I originally linked fixes this. And I am back to wondering if anything matters if localhost isn't bound to the NTP server. The rabbit hole just keeps getting deeper.

Thank you so much for taking the time to teach me some things. Your patience is appreciated.

johnpoz · Jan 31, 2022, 3:34 PM

@tjsober said in NTP Server not synchronizing if localhost selected:

10.10.10.1... the pfBlocker VIP. That's not going to work!

nope that wouldn't work ;) hmmm I would have to change back and sniff to see if have same thing.. But I am not using pfblocker in that fashion, I have no vip setup with it. I just use it to manipulate some aliases that I use in my firewall.

Rabbit holes can be fun, if you ever get to the end.. Other times they can be real head scratchers ;)

edit: see I don't have that feature enabled in pfblocker

It shouldn't be creating that vip if I don't have that enabled - have to take a double check look see.

edit2: I just looked and there is no vip on my loopback interface.. I do see my other vip on my wan interface, etc. But no 10.10.10.1 anywhere on any interface..

stephenw10 · Jan 31, 2022, 3:29 PM

@tjsober said in NTP Server not synchronizing if localhost selected:

I still don't understand why the port forward rule in the post I originally linked fixes this.

Probably because 'This firewall' is all IPs addresses on the firewall, including VIPs.

tjsober · Jan 31, 2022, 3:59 PM

@johnpoz Under pfBlockerNG - DNSBL if I change the Web Server Interface from Localhost to my LAN with high numbered unused ports, the problem is fixed. Likely because the LAN is handled by the outgoing NAT auto rules.

Oddly, the SG-5100 doesn't seem to care about that setting and the NTP server just works.

More homework to do...

EDIT: What I said doesn't make sense to me. It doesn't explain why NTP is picking localhost as the outgoing port? It only identifies some interaction with pfBlocker.

tjsober · Jan 31, 2022, 4:31 PM

@johnpoz @johnpoz The last comments in this post from 4 years ago seem to discuss this same problem.

https://www.reddit.com/r/PFSENSE/comments/7k00yz/dnsbl_blocks_ntp_server_from_syncing/

But I do not understand why NTP decides to bind to the pfblocker VIP. So more googling and I found a person talking about NTP

https://www.reddit.com/r/pfBlockerNG/comments/kfnujp/avahi_binding_to_the_virtual_ip_of_pfblockerng/

So I shut off Avahi and set pfblocker back to localhost and all is well. I re-enabled avahi, restarted NTP and it's broken again. More googling...but I guess this is progress.

johnpoz · Jan 31, 2022, 4:29 PM

@tjsober and down the rabbit hole you go ;)

stephenw10 · Jan 31, 2022, 4:49 PM

@tjsober said in NTP Server not synchronizing if localhost selected:

But I do not understand why NTP decides to bind to the pfblocker VIP.

Probably because it's the lowest available source IP as you said previously.

tjsober · Jan 31, 2022, 4:52 PM

@stephenw10 Does "lowest" mean IP? Because I tried changing the VIP from 10.10.10.1 to 192.168.250.250 (above any of the IP subnets) and the behavior is the same.

tjsober · Jan 31, 2022, 5:05 PM

@johnpoz So I think I can ignore avahi. I was watching as I did a force reload on pfblocker and even though I picked localhost for the web server interface, it still thinks it is on the lan high ports. So it was fooling me because the configuration wasn't changing to localhost. I'm going to uninstall pfblocker and start from scratch.

What a way to waste a day...

tjsober · Jan 31, 2022, 5:41 PM

OK...I'm stopping. Apparently the pfBlocker VIP and Web Server Interface ports were "sticking" and after several changes and Force Updates I got the DNSBL Webserver ports to change to 80 and 443 on localhost and set the VIP from 10.10.10.1 to 192.168.254.254. Everything seems to be working... NTP with localhost and not WAN selected syncs, Avahi is happy (that was a red herring), and pfBlocker is working.

It all seems to come down to, for some reason that I don't know, on this firewall NTP binds to the VIP because it is lower, where as on the SG-5100 NTP ignores the VIP even though it is also lower.

Thank you for your patience and all of the assistance.

Edit: I figured out why the SG-5100 works... It has a VPN server at 10.3.x.x, BELOW the DNSBL VIP of 10.10.10.1. Whereas the VP2410 VPN is at 10.42.x.x, ABOVE the DNSBL IP. This was the difference all along and it is the VPN IP that NTP is now syncing to on both systems.