[SOLVED] NTP not answering on 2-nd uplink WAN

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

So to be clear if you run a pcap on WAN1 whilst trying to connect to NTP on WAN2 do you see replies incorrectly leaving WAN1?

Sorry for delaying in reply.

Requests from outside come to both WAN1 and WAN2 successfully because pf rule for WAN_GROUP (Tier 1 for each of both WANs) that allow from any on port 123. I see the requests by pcap.;

Both WAN1 and WAN2 have stable IP by DHCP. (DHCP is in WAN interface settings, do I need to change it to STATIC?)

In fw log I able to see live sessions to BOTH(?) WANs

(The results are filtered by real external WAN IP)

By pcap I cannot see the answers on port 123 on WAN2. But able to see answers on port 123 on WAN1.
pfSense itself are act as NTP server. (if this matter);

Where I am missing something?

Thank You for patience!

stephenw10

But specifically you do not see the replies to requests on WAN2 leaving via WAN1?

I suspect that is what's happening because of the recent reply-to changes in pf. Replies leave via the default route in some circumstances when reply-to does not apply correctly.

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

But specifically you do not see the replies to requests on WAN2 leaving via WAN1?

For definitely know this I need to pcap with filtering ‘123’ port simultaneously both WAN1 and WAN2 (mean two WebGUI windows), and than try to compare?

Or some more elegant way exist? ;)

I suspect that is what's happening because of the recent reply-to changes in pf. Replies leave via the default route in some circumstances when reply-to does not apply correctly.

Right now the pf rule to “allow from all to port 123 and ”Default” as a “Gaitway” as exist on WAN_GROUP Interfaces group

So may be creating TWO(2) SEPARATE RULES instead (and different “Gaitway”-s?) exactly on a each of WANs?

Sergei_Shablovsky

I create TWO(2) SEPARATE RULES, allowing all on 123 port for each of WANs and with different “Gateway” in “Advanced Options”.

Result are SAME: by pcap I able to see answers on incoming NTP on WAN1 and no answers on incoming NTP on WAN2.

Help? ;)

stephenw10

I would send queries to WAN2 from some external IP addess. Then pcap on WAN1 and filter by the external address and see if any replies leave there.

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

I would send queries to WAN2 from some external IP addess. Then pcap on WAN1 and filter by the external address and see if any replies leave there.

I pcap both interfaces WAN1 and WAN2 then merge both files in WireShark, and CANNOT SEE the replies on WAN2.

So,

pf rules on each of WANs allow incoming NTP UDP requests on port 123. In rules Gaitway Options are exactly the same interface. DIFFERENT for each of WANs;
the ntpd service are running and in their settings pfSense page ALL interfaces are selected;
I see incoming requests outside coming and pass on both interfaces;
on WAN1 I see the answers to outside initiator IP, on WAN2 - no answers (and no answers thru WAN1 on requests on WAN2), only incoming;

stephenw10

If you set WAN2 as the default route does that allow it to work? And break the requests to WAN1?

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

If you set WAN2 as the default route does that allow it to work? And break the requests to WAN1?

As I wrote before, I create two pf rule: one per WAN interface, and difference between rules are only in interface on which rule exist (of course) and Gaitway (different for each interface).
So, the Gaitway directly fixed in pf rule.

stephenw10

Right and those rules should apply reply-to tags to the incoming traffic such that replies to that go back out of the correct WAN. But that isn't happening.

So if reply-to doesn't work replies usually end up going via the default gateway instead. But that also doesn't appear to be happening.

However it could be failing to match the state for some other reason and dropping the replies on any non-default gateway. If that is happening then switching the default to WAN2 would move that behaviour confirming the cause.

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

Right and those rules should apply reply-to tags to the incoming traffic such that replies to that go back out of the correct WAN. But that isn't happening.

Agree.

So if reply-to doesn't work replies usually end up going via the default gateway instead. But that also doesn't appear to be happening.

Also agree.

However it could be failing to match the state for some other reason and dropping the replies on any non-default gateway. If that is happening then switching the default to WAN2 would move that behaviour confirming the cause.

Today case ~~resolved~~ AFTER UPGRADE
curl upgraded: 8.5.0 -> 8.6.0
unbound upgraded: 1.18.0_1 -> 1.19.1

So, I thinking this was some combination of bug-behavior in unbound and my settings on this pfSense.

THANK YOU SO MUCH, Stephen for AMAZING PATIENCE and help!

stephenw10

Huh. Surprising. Nice result!

Sergei_Shablovsky

@stephenw10 said in [SOLVED] NTP not answering on 2-nd uplink WAN:

Huh. Surprising. Nice result!

Thank You, but after restart the issue come back again but opposite: WAN2 answering, but WAN1 - no!

On both WANs pcap show that incoming requests exist.

Have no idea how resolving…?

stephenw10

Is the default route now using WAN2?

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

Is the default route now using WAN2?

Yes

Ok. May be better I take Your concern, and You give me step-by-step plan like how You would be resolve this issue. :) Because for now I have heavy mashed mind about this issue...

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

Right and those rules should apply reply-to tags to the incoming traffic such that replies to that go back out of the correct WAN. But that isn't happening.

So if reply-to doesn't work …

How to ensure that automatic reply-to created by pfSense?

stephenw10

Well I'm not sure how to resolve it right now. First we need to confirm that the working connection follows the default route. So try setting the default route back to WAN1 and make sure that changes the working NTP responses back to that.
Then we should investigate what happened when you updated those pkgs that seemed to temporarily allow both WANs to work. See if you can replicate that by reinstalling those pkgs for example.

This is probably something low level in pf though.

Sergei_Shablovsky

@stephenw10 said in NTP not answering on 2-nd uplink WAN:

Well I'm not sure how to resolve it right now. First we need to confirm that the working connection follows the default route. So try setting the default route back to WAN1 and make sure that changes the working NTP responses back to that.
Then we should investigate what happened when you updated those pkgs that seemed to temporarily allow both WANs to work. See if you can replicate that by reinstalling those pkgs for example.

This is probably something low level in pf though.

THANK YOU SO MUCH about patience and help!

I purpose to going step-by-step and You just correct me if I doing something wrong. :)

Sergei_Shablovsky

Right now:

on both WLAN1 and WAN2 in firewall rules for NTP "States Details" in "State" column
most of all connections are in MULTIPLE:MULTIPLE and SINGLE:MULTIPLE.
ntpd are listening both WAN1 and WAN2 ("Diagnostics / Sockets" "IPv4 System Socket Information" table)

So is this mean that pf rules are working ok and NTP receive requests and answering ok ?

NollipfSense

@Sergei_Shablovsky Thank you for reaching out via message...I read through the thread and Steve's diagnosing makes sense about the default WAN routing...I couldn't add anything more...

stephenw10

@Sergei_Shablovsky said in NTP not answering on 2-nd uplink WAN:

So is this mean that pf rules are working ok and NTP receive requests and answering ok ?

NTPd is fine. pf appears to be opening states correctly but what doesn't appear to be happening is the replies going back out via the correct gateway.

So did you confirm that moving the default gateway back to WAN1 switches the working WAN for NTP?