[SOLVED] NTP not answering on 2-nd uplink WAN
-
@stephenw10 said in [SOLVED] NTP not answering on 2-nd uplink WAN:
Huh. Surprising. Nice result!
Thank You, but after restart the issue come back again but opposite: WAN2 answering, but WAN1 - no!
On both WANs pcap show that incoming requests exist.
Have no idea how resolving…?
-
Is the default route now using WAN2?
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
Is the default route now using WAN2?
Yes
Ok. May be better I take Your concern, and You give me step-by-step plan like how You would be resolve this issue. :) Because for now I have heavy mashed mind about this issue...
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
Right and those rules should apply reply-to tags to the incoming traffic such that replies to that go back out of the correct WAN. But that isn't happening.
So if reply-to doesn't work …
How to ensure that automatic reply-to created by pfSense?
-
Well I'm not sure how to resolve it right now. First we need to confirm that the working connection follows the default route. So try setting the default route back to WAN1 and make sure that changes the working NTP responses back to that.
Then we should investigate what happened when you updated those pkgs that seemed to temporarily allow both WANs to work. See if you can replicate that by reinstalling those pkgs for example.This is probably something low level in pf though.
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
Well I'm not sure how to resolve it right now. First we need to confirm that the working connection follows the default route. So try setting the default route back to WAN1 and make sure that changes the working NTP responses back to that.
Then we should investigate what happened when you updated those pkgs that seemed to temporarily allow both WANs to work. See if you can replicate that by reinstalling those pkgs for example.This is probably something low level in pf though.
THANK YOU SO MUCH about patience and help!
I purpose to going step-by-step and You just correct me if I doing something wrong. :)
-
Right now:
- on both WLAN1 and WAN2 in firewall rules for NTP "States Details" in "State" column
most of all connections are in MULTIPLE:MULTIPLE and SINGLE:MULTIPLE. - ntpd are listening both WAN1 and WAN2 ("Diagnostics / Sockets" "IPv4 System Socket Information" table)
So is this mean that pf rules are working ok and NTP receive requests and answering ok ?
—
CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
Help Ukraine to resist, save civilians people’s lives !
(Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.) - on both WLAN1 and WAN2 in firewall rules for NTP "States Details" in "State" column
-
@Sergei_Shablovsky Thank you for reaching out via message...I read through the thread and Steve's diagnosing makes sense about the default WAN routing...I couldn't add anything more...
-
@Sergei_Shablovsky said in NTP not answering on 2-nd uplink WAN:
So is this mean that pf rules are working ok and NTP receive requests and answering ok ?
NTPd is fine. pf appears to be opening states correctly but what doesn't appear to be happening is the replies going back out via the correct gateway.
So did you confirm that moving the default gateway back to WAN1 switches the working WAN for NTP?
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
@Sergei_Shablovsky said in NTP not answering on 2-nd uplink WAN:
So is this mean that pf rules are working ok and NTP receive requests and answering ok ?
NTPd is fine. pf appears to be opening states correctly but what doesn't appear to be happening is the replies going back out via the correct gateway.
I have the same decision. But the gateway are directly set in “Advanced Option / Gateway” in pf rule.
In which case that’s may be not enough for ntpd (or any other service?) answers going out this “directly set Gateway”?
—
CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
Help Ukraine to resist, save civilians people’s lives !
(Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.) -
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
So did you confirm that moving the default gateway back to WAN1 switches the working WAN for NTP?
Previously I starting two(2) instance of pcap (different tty), one for WLAN1 + one for WLAN2, and than merge both .pcap in WireShark to see how answers happened.
Is this method correct?
-
@Sergei_Shablovsky said in NTP not answering on 2-nd uplink WAN:
I have the same decision. But the gateway are directly set in “Advanced Option / Gateway” in pf rule.
Ah wait you set the gateway on the inbound WAN pass rules for NTP queries? You should not set a gateway there. The fact traffic is passed on an interface with a gateway is sufficient to tag it reply-to for that gateway. Setting a gateway on an inbound rule is incorrect as it will try to force traffic that way.
-
@stephenw10 said in NTP not answering on 2-nd uplink WAN:
@Sergei_Shablovsky said in NTP not answering on 2-nd uplink WAN:
I have the same decision. But the gateway are directly set in “Advanced Option / Gateway” in pf rule.
Ah wait you set the gateway on the inbound WAN pass rules for NTP queries?
Exactly.
You should not set a gateway there.
At the start of topic I have a “Default” as Gateway in “Advanced Options”.
But issue still exist, so I decide to set directly.The fact traffic is passed on an interface with a gateway is sufficient to tag it reply-to for that gateway. Setting a gateway on an inbound rule is incorrect as it will try to force traffic that way.
Ok, set “Default” in “Advanced Options” back. ;)
And reboot.But issue still exist.
-
Ok make sure the states still appear the same for connections on both WANs without the gateway set on the rules.
Make sure the working WAN still follows the system default gateway. -
@stephenw10 said in [SOLVED] NTP not answering on 2-nd uplink WAN:
Ok make sure the states still appear the same for connections on both WANs without the gateway set on the rules.
Still appear.
Make sure the working WAN still follows the system default gateway.
Still follows.
But issue exist. ;)
-
@stephenw10 Thank You SO MUCH for patience and help!
After rewriting a lot of rules, and manually saving some scripts (f..k previous admin!), double checks all physical connections and all in System / Advanced: I just reinstall pfSense CE 2.7.2-RELEASE on this server with “Use previous config” option and the issue gone away
P.S.
Of course, I would be happy to help John to finding this bug, but due complexity of settings not possible to reproduce it… Sad. -
Hmm, weird. Hard to see why it wouldn't have been applying reply-to tags.
I guess good result in the end!
-
@stephenw10 said in [SOLVED] NTP not answering on 2-nd uplink WAN:
Hmm, weird. Hard to see why it wouldn't have been applying reply-to tags.
Yeah! I find how to reproduce the issue, that’s the scheme:
(there are 2 uplink ISP, the 2 connections from each)
WAN-ALL group
— WAN-ISP-A group
—— WAN-ISP-A-1 DHCP interface
—— WAN-ISP-A-2 DHCP interface
— WAN-ISP-B group
—— WAN-ISP-B-1 DHCP interface
—— WAN-ISP-B-2 DHCP interfaceExist 1 main WAN interface group, that consist of 2 groups, one for each ISP. The each ISP group consist of 2 uplinks/interfaces.
So, totally 4 uplinks (with 2 of them temporarily not activated, mean 100% packet loss, but 1Gb link established).
There are BALANCED interface group created (with Tear 1 for each, and “Packet loss or Hight Latency” as Trigger Level.
And in System / Routing / Gateways this BALANCED group set as “Default Gateway IPv4”
So, ordinary multi-WAN setup for IPv4.
(I know that some settings in System / Advanced may impact on a result, so if You ask, I send screenshots to You by PM)In Firewall aliases exist NTP_PORT alias with 123 and 1023 ports.
THE ISSUE!!!:
If “pass from all incoming UDP on This Firewall (self) on NTP_PORT” pf rule exist on WAN-ISP-B or WAN-ALL level ~> NTPD ANSWERS OUT ONLY FROM (default) INTERFACE !If the SAME rule moving to INTERFACE level (both WAN-ISP-B-1 DHCP interface & WAN-ISP-B-2 DHCP interface) ~> NTPD ANSWERS OUT FROM BOTH INTERFACES on which NTP requests come in.
I reinstall pfSense from scratch several times (with “using previous config” option) and triple confirm the issue!
I guess good result in the end!
With a lot of spending hours also… ;)
-
Ah you actually have an interface group for the WANs with the rule on it?
Yes, if you do that reply-to tags cannot works because the rule applies to multiple interfaces. It cannot know which interface (gateway) to reply to.
For reply-to tagging to work incoming traffic must be passed on the interface itself. It's the same reason that OpenVPN traffic must be passed on an assigned interface for repy-to to work. The group openvpn interface will not tags it.@Sergei_Shablovsky said in [SOLVED] NTP not answering on 2-nd uplink WAN:
And in System / Routing / Gateways this BALANCED group set as “Default Gateway IPv4”
That's still invalid. The system default gateway should only be a specific gateway or a failover group. You cannot load-balance traffic like that.
Steve
