[SOLVED] NTP not answering on 2-nd uplink WAN
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
Hmm, curious.
EACH problem about I wrote on this forum ARE SERIOUS: always I try to resolving by myself and only after much efforts are unsuccessful, - write here on forum.
I try to respect time life of others professional members. ;)So you see incoming requests in the pcap?
Exactly!
And there is a state opened on WAN2?
Screenshot from states filtering:
But no replies?
Yes sir! ;)
-
That's on all interfaces or just WAN2? Are those outbound connections showing with two way traffic?
I assume WAN2 is not the default route? Are replies leaving via the default gateway instead?
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
That's on all interfaces or just WAN2?
Exactly just WAN2.
(You ask about WAN2, I was answering about WAN2). ;)Are those outbound connections showing with two way traffic?
Not, only incoming to WAN2 from outside, take a look on a screenshot above: the port are 123 in all rows.
I assume WAN2 is not the default route? Are replies leaving via the default gateway instead?
There are common GROUP, that consist of WAN2 and WAN2, both are Tier 1, and “Packet loss or High Latency”.
And “Default Gateway IPv4” (on System/Routing/Gateways) are set to this GROUP.
-
You can't use a load-balancing gateway group as the default gateway. Load-balancing like that is a pf function so can only be done via policy routing.
In the state table then there are 6 connections showing two way traffic. Are those not seeing ntp replies?
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
You can't use a load-balancing gateway group as the default gateway. Load-balancing like that is a pf function so can only be done via policy routing.
I am not “initial” sysadmin of this device. So, when I see that You describe, the first that I try was CHANGING the “Default Gateway IPv4” from this group to just WAN1 or WAN2.
But after “Normal Restart” pcap not even see incoming requests from outside to port 123.
After returning “Default Gateway IPv4” back TO GROUP, all return to previous state: incoming requests on WAN1 in port 123 pass, and ntpd answering pass back. (Look from pcap perspective).In the state table then there are 6 connections showing two way traffic. Are those not seeing ntp replies?
I also see that and confused a little bit, because pcap NOT SHOWING THE ANSWERING from ntpd.
What I am missing!?
Thank you again for help and patience!
-
Ok that's why I asked about replies going via a different interface. The states show two way traffic but it's not in a pcap on WAN2. So is it leaving via WAN1?
The default gateway should be either a single gateway or a failover group. If it's set to a load-balance group like that the actual gateway is undetermined at best.
Setting the default to either WAN1 or WAN2 should have no effect on incoming traffic. Except maybe if those public IPs are being advertised via BGP (or some other dynamic protocol) and that's failing.
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
Ok that's why I asked about replies going via a different interface. The states show two way traffic but it's not in a pcap on WAN2. So is it leaving via WAN1?
Definitely not leaving, requests are coming (allow by pf rules) on both WANs, but answering are only on ONE of WANs.
The default gateway should be either a single gateway or a failover group. If it's set to a load-balance group like that the actual gateway is undetermined at best.
Setting the default to either WAN1 or WAN2 should have no effect on incoming traffic. Except maybe if those public IPs are being advertised via BGP (or some other dynamic protocol) and that's failing.
Ok, Thank You for explanation. Now I little better understanding pfSense’s behavior. ;)
So, how to make possible behavior when NTP client from outside would receive the answer from pfSense’s NTP server (ntpd in this case) FROM THE SAME IP, which client request for?
Creating some hard NAT rules?
-
So to be clear if you run a pcap on WAN1 whilst trying to connect to NTP on WAN2 do you see replies incorrectly leaving WAN1?
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
So to be clear if you run a pcap on WAN1 whilst trying to connect to NTP on WAN2 do you see replies incorrectly leaving WAN1?
Sorry for delaying in reply.
- Requests from outside come to both WAN1 and WAN2 successfully because pf rule for WAN_GROUP (Tier 1 for each of both WANs) that allow from any on port 123. I see the requests by pcap.;
Both WAN1 and WAN2 have stable IP by DHCP. (DHCP is in WAN interface settings, do I need to change it to STATIC?)
- In fw log I able to see live sessions to BOTH(?) WANs
(The results are filtered by real external WAN IP)
-
By pcap I cannot see the answers on port 123 on WAN2. But able to see answers on port 123 on WAN1.
-
pfSense itself are act as NTP server. (if this matter);
Where I am missing something?
Thank You for patience!
-
But specifically you do not see the replies to requests on WAN2 leaving via WAN1?
I suspect that is what's happening because of the recent reply-to changes in pf. Replies leave via the default route in some circumstances when reply-to does not apply correctly.
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
But specifically you do not see the replies to requests on WAN2 leaving via WAN1?
For definitely know this I need to pcap with filtering ‘123’ port simultaneously both WAN1 and WAN2 (mean two WebGUI windows), and than try to compare?
Or some more elegant way exist? ;)
I suspect that is what's happening because of the recent reply-to changes in pf. Replies leave via the default route in some circumstances when reply-to does not apply correctly.
Right now the pf rule to “allow from all to port 123 and ”Default” as a “Gaitway” as exist on WAN_GROUP Interfaces group
So may be creating TWO(2) SEPARATE RULES instead (and different “Gaitway”-s?) exactly on a each of WANs?
-
I create TWO(2) SEPARATE RULES, allowing all on 123 port for each of WANs and with different “Gateway” in “Advanced Options”.
Result are SAME: by pcap I able to see answers on incoming NTP on WAN1 and no answers on incoming NTP on WAN2.
Help? ;)
-
I would send queries to WAN2 from some external IP addess. Then pcap on WAN1 and filter by the external address and see if any replies leave there.
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
I would send queries to WAN2 from some external IP addess. Then pcap on WAN1 and filter by the external address and see if any replies leave there.
I pcap both interfaces WAN1 and WAN2 then merge both files in WireShark, and CANNOT SEE the replies on WAN2.
So,
-
pf rules on each of WANs allow incoming NTP UDP requests on port 123. In rules Gaitway Options are exactly the same interface. DIFFERENT for each of WANs;
-
the ntpd service are running and in their settings pfSense page ALL interfaces are selected;
-
I see incoming requests outside coming and pass on both interfaces;
-
on WAN1 I see the answers to outside initiator IP, on WAN2 - no answers (and no answers thru WAN1 on requests on WAN2), only incoming;
-
-
If you set WAN2 as the default route does that allow it to work? And break the requests to WAN1?
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
If you set WAN2 as the default route does that allow it to work? And break the requests to WAN1?
As I wrote before, I create two pf rule: one per WAN interface, and difference between rules are only in interface on which rule exist (of course) and Gaitway (different for each interface).
So, the Gaitway directly fixed in pf rule. -
Right and those rules should apply reply-to tags to the incoming traffic such that replies to that go back out of the correct WAN. But that isn't happening.
So if reply-to doesn't work replies usually end up going via the default gateway instead. But that also doesn't appear to be happening.
However it could be failing to match the state for some other reason and dropping the replies on any non-default gateway. If that is happening then switching the default to WAN2 would move that behaviour confirming the cause.
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
Right and those rules should apply reply-to tags to the incoming traffic such that replies to that go back out of the correct WAN. But that isn't happening.
Agree.
So if reply-to doesn't work replies usually end up going via the default gateway instead. But that also doesn't appear to be happening.
Also agree.
However it could be failing to match the state for some other reason and dropping the replies on any non-default gateway. If that is happening then switching the default to WAN2 would move that behaviour confirming the cause.
Today case
resolvedAFTER UPGRADE
curl upgraded: 8.5.0 -> 8.6.0
unbound upgraded: 1.18.0_1 -> 1.19.1So, I thinking this was some combination of bug-behavior in unbound and my settings on this pfSense.
THANK YOU SO MUCH, Stephen for AMAZING PATIENCE and help!
-
Huh. Surprising. Nice result!
-
@stephenw10 said in [SOLVED] NTP not answering on 2-nd uplink WAN:
Huh. Surprising. Nice result!
Thank You, but after restart the issue come back again but opposite: WAN2 answering, but WAN1 - no!
On both WANs pcap show that incoming requests exist.
Have no idea how resolving…?