[SOLVED] NTP not answering on 2-nd uplink WAN

Sergei_Shablovsky

Dear pfSense Gurus!

There are on bare metal server
2.7.2-RELEASE (amd64)
built on Wed Dec 6 22:10:00 EET 2023
FreeBSD 14.0-CURRENT

2(two) balanced WANs

And the problem are (and Packet Capture confirm that)
NTP daemon answering on incoming requests on one WAN, and NOT ANSWERING ON OTHER WAN.

Not touching any NTP settings.

This just happened AFTER REINSTALL FROM SCRATCH (booting from usb drive and PREVIOUS CONFIG picked up.)

Thank You so much to all!

stephenw10

I wouldn't expect NTP to respond to queries on any WAN by default so I assume you have added rules to allow that? Selected all interfaces in the ntpd settings?

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

I wouldn't expect NTP to respond to queries on any WAN by default so I assume you have added rules to allow that?

Definitely GROUP rule pass UDP on 123 port on BOTH uplinks WAN.
(As I told before, that happened after reinstalling, before both successfully working.)

Selected all interfaces in the ntpd settings?

YES. Triple checked;)

stephenw10

What version were you running before you reinstalled?

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

What version were you running before you reinstalled?

The SAME, 2.7.2-RELEASE

Sergei_Shablovsky

How to see on FreeBSD level how NTPD answering on requests?

Because looks like all others working

pcap show that income requests pass the firewall;
from NTP WebGUI all interfaces are selected;

stephenw10

Do you see it listening on WAN2 in sockstat -l4?

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

Do you see it listening on WAN2 in sockstat -l4?

Definitely YES. (If You mean “see” by pfSense’s WebGUI “Diagnostics/Sockets”)

stephenw10

Hmm, curious. So you see incoming requests in the pcap? And there is a state opened on WAN2? But no replies?

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

Hmm, curious.

EACH problem about I wrote on this forum ARE SERIOUS: always I try to resolving by myself and only after much efforts are unsuccessful, - write here on forum.
I try to respect time life of others professional members. ;)

So you see incoming requests in the pcap?

Exactly!

And there is a state opened on WAN2?

Screenshot from states filtering:

But no replies?

Yes sir! ;)

stephenw10

That's on all interfaces or just WAN2? Are those outbound connections showing with two way traffic?

I assume WAN2 is not the default route? Are replies leaving via the default gateway instead?

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

That's on all interfaces or just WAN2?

Exactly just WAN2.
(You ask about WAN2, I was answering about WAN2). ;)

Are those outbound connections showing with two way traffic?

Not, only incoming to WAN2 from outside, take a look on a screenshot above: the port are 123 in all rows.

I assume WAN2 is not the default route? Are replies leaving via the default gateway instead?

There are common GROUP, that consist of WAN2 and WAN2, both are Tier 1, and “Packet loss or High Latency”.

And “Default Gateway IPv4” (on System/Routing/Gateways) are set to this GROUP.

stephenw10

You can't use a load-balancing gateway group as the default gateway. Load-balancing like that is a pf function so can only be done via policy routing.

In the state table then there are 6 connections showing two way traffic. Are those not seeing ntp replies?

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

You can't use a load-balancing gateway group as the default gateway. Load-balancing like that is a pf function so can only be done via policy routing.

I am not “initial” sysadmin of this device. So, when I see that You describe, the first that I try was CHANGING the “Default Gateway IPv4” from this group to just WAN1 or WAN2.
But after “Normal Restart” pcap not even see incoming requests from outside to port 123.
After returning “Default Gateway IPv4” back TO GROUP, all return to previous state: incoming requests on WAN1 in port 123 pass, and ntpd answering pass back. (Look from pcap perspective).

In the state table then there are 6 connections showing two way traffic. Are those not seeing ntp replies?

I also see that and confused a little bit, because pcap NOT SHOWING THE ANSWERING from ntpd.

What I am missing!?

Thank you again for help and patience!

stephenw10

Ok that's why I asked about replies going via a different interface. The states show two way traffic but it's not in a pcap on WAN2. So is it leaving via WAN1?

The default gateway should be either a single gateway or a failover group. If it's set to a load-balance group like that the actual gateway is undetermined at best.

Setting the default to either WAN1 or WAN2 should have no effect on incoming traffic. Except maybe if those public IPs are being advertised via BGP (or some other dynamic protocol) and that's failing.

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

Ok that's why I asked about replies going via a different interface. The states show two way traffic but it's not in a pcap on WAN2. So is it leaving via WAN1?

Definitely not leaving, requests are coming (allow by pf rules) on both WANs, but answering are only on ONE of WANs.

The default gateway should be either a single gateway or a failover group. If it's set to a load-balance group like that the actual gateway is undetermined at best.

Setting the default to either WAN1 or WAN2 should have no effect on incoming traffic. Except maybe if those public IPs are being advertised via BGP (or some other dynamic protocol) and that's failing.

Ok, Thank You for explanation. Now I little better understanding pfSense’s behavior. ;)

So, how to make possible behavior when NTP client from outside would receive the answer from pfSense’s NTP server (ntpd in this case) FROM THE SAME IP, which client request for?

Creating some hard NAT rules?

stephenw10

So to be clear if you run a pcap on WAN1 whilst trying to connect to NTP on WAN2 do you see replies incorrectly leaving WAN1?

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

So to be clear if you run a pcap on WAN1 whilst trying to connect to NTP on WAN2 do you see replies incorrectly leaving WAN1?

Sorry for delaying in reply.

Requests from outside come to both WAN1 and WAN2 successfully because pf rule for WAN_GROUP (Tier 1 for each of both WANs) that allow from any on port 123. I see the requests by pcap.;

Both WAN1 and WAN2 have stable IP by DHCP. (DHCP is in WAN interface settings, do I need to change it to STATIC?)

In fw log I able to see live sessions to BOTH(?) WANs

(The results are filtered by real external WAN IP)

By pcap I cannot see the answers on port 123 on WAN2. But able to see answers on port 123 on WAN1.
pfSense itself are act as NTP server. (if this matter);

Where I am missing something?

Thank You for patience!

stephenw10

But specifically you do not see the replies to requests on WAN2 leaving via WAN1?

I suspect that is what's happening because of the recent reply-to changes in pf. Replies leave via the default route in some circumstances when reply-to does not apply correctly.

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

But specifically you do not see the replies to requests on WAN2 leaving via WAN1?

For definitely know this I need to pcap with filtering ‘123’ port simultaneously both WAN1 and WAN2 (mean two WebGUI windows), and than try to compare?

Or some more elegant way exist? ;)

I suspect that is what's happening because of the recent reply-to changes in pf. Replies leave via the default route in some circumstances when reply-to does not apply correctly.

Right now the pf rule to “allow from all to port 123 and ”Default” as a “Gaitway” as exist on WAN_GROUP Interfaces group

So may be creating TWO(2) SEPARATE RULES instead (and different “Gaitway”-s?) exactly on a each of WANs?