Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    [SOLVED] NTP not answering on 2-nd uplink WAN

    Scheduled Pinned Locked Moved General pfSense Questions
    47 Posts 3 Posters 8.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Sergei_ShablovskyS
      Sergei_Shablovsky @stephenw10
      last edited by Sergei_Shablovsky

      @stephenw10 said in NTP not answering on 2-nd uplink WAN:

      I would send queries to WAN2 from some external IP addess. Then pcap on WAN1 and filter by the external address and see if any replies leave there.

      I pcap both interfaces WAN1 and WAN2 then merge both files in WireShark, and CANNOT SEE the replies on WAN2.

      So,

      • pf rules on each of WANs allow incoming NTP UDP requests on port 123. In rules Gaitway Options are exactly the same interface. DIFFERENT for each of WANs;

      • the ntpd service are running and in their settings pfSense page ALL interfaces are selected;

      • I see incoming requests outside coming and pass on both interfaces;

      • on WAN1 I see the answers to outside initiator IP, on WAN2 - no answers (and no answers thru WAN1 on requests on WAN2), only incoming;

      —
      CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
      Help Ukraine to resist, save civilians people’s lives !
      (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        If you set WAN2 as the default route does that allow it to work? And break the requests to WAN1?

        Sergei_ShablovskyS 1 Reply Last reply Reply Quote 0
        • Sergei_ShablovskyS
          Sergei_Shablovsky @stephenw10
          last edited by

          @stephenw10 said in NTP not answering on 2-nd uplink WAN:

          If you set WAN2 as the default route does that allow it to work? And break the requests to WAN1?

          As I wrote before, I create two pf rule: one per WAN interface, and difference between rules are only in interface on which rule exist (of course) and Gaitway (different for each interface).
          So, the Gaitway directly fixed in pf rule.

          —
          CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
          Help Ukraine to resist, save civilians people’s lives !
          (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Right and those rules should apply reply-to tags to the incoming traffic such that replies to that go back out of the correct WAN. But that isn't happening.

            So if reply-to doesn't work replies usually end up going via the default gateway instead. But that also doesn't appear to be happening.

            However it could be failing to match the state for some other reason and dropping the replies on any non-default gateway. If that is happening then switching the default to WAN2 would move that behaviour confirming the cause.

            Sergei_ShablovskyS 2 Replies Last reply Reply Quote 1
            • Sergei_ShablovskyS
              Sergei_Shablovsky @stephenw10
              last edited by Sergei_Shablovsky

              @stephenw10 said in NTP not answering on 2-nd uplink WAN:

              Right and those rules should apply reply-to tags to the incoming traffic such that replies to that go back out of the correct WAN. But that isn't happening.

              Agree.

              So if reply-to doesn't work replies usually end up going via the default gateway instead. But that also doesn't appear to be happening.

              Also agree.

              However it could be failing to match the state for some other reason and dropping the replies on any non-default gateway. If that is happening then switching the default to WAN2 would move that behaviour confirming the cause.

              Today case resolved AFTER UPGRADE
              curl upgraded: 8.5.0 -> 8.6.0
              unbound upgraded: 1.18.0_1 -> 1.19.1

              So, I thinking this was some combination of bug-behavior in unbound and my settings on this pfSense.

              THANK YOU SO MUCH, Stephen for AMAZING PATIENCE and help!

              —
              CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
              Help Ukraine to resist, save civilians people’s lives !
              (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                Huh. Surprising. Nice result!

                Sergei_ShablovskyS 1 Reply Last reply Reply Quote 1
                • Sergei_ShablovskyS
                  Sergei_Shablovsky @stephenw10
                  last edited by Sergei_Shablovsky

                  @stephenw10 said in [SOLVED] NTP not answering on 2-nd uplink WAN:

                  Huh. Surprising. Nice result!

                  Thank You, but after restart the issue come back again but opposite: WAN2 answering, but WAN1 - no!

                  On both WANs pcap show that incoming requests exist.

                  Have no idea how resolving…?

                  —
                  CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
                  Help Ukraine to resist, save civilians people’s lives !
                  (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Is the default route now using WAN2?

                    Sergei_ShablovskyS 1 Reply Last reply Reply Quote 0
                    • Sergei_ShablovskyS
                      Sergei_Shablovsky @stephenw10
                      last edited by Sergei_Shablovsky

                      @stephenw10 said in NTP not answering on 2-nd uplink WAN:

                      Is the default route now using WAN2?

                      Yes

                      Ok. May be better I take Your concern, and You give me step-by-step plan like how You would be resolve this issue. :) Because for now I have heavy mashed mind about this issue...

                      —
                      CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
                      Help Ukraine to resist, save civilians people’s lives !
                      (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

                      1 Reply Last reply Reply Quote 0
                      • Sergei_ShablovskyS
                        Sergei_Shablovsky @stephenw10
                        last edited by

                        @stephenw10 said in NTP not answering on 2-nd uplink WAN:

                        Right and those rules should apply reply-to tags to the incoming traffic such that replies to that go back out of the correct WAN. But that isn't happening.

                        So if reply-to doesn't work …

                        How to ensure that automatic reply-to created by pfSense?

                        —
                        CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
                        Help Ukraine to resist, save civilians people’s lives !
                        (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Well I'm not sure how to resolve it right now. First we need to confirm that the working connection follows the default route. So try setting the default route back to WAN1 and make sure that changes the working NTP responses back to that.
                          Then we should investigate what happened when you updated those pkgs that seemed to temporarily allow both WANs to work. See if you can replicate that by reinstalling those pkgs for example.

                          This is probably something low level in pf though.

                          Sergei_ShablovskyS 1 Reply Last reply Reply Quote 1
                          • Sergei_ShablovskyS
                            Sergei_Shablovsky @stephenw10
                            last edited by

                            @stephenw10 said in NTP not answering on 2-nd uplink WAN:

                            Well I'm not sure how to resolve it right now. First we need to confirm that the working connection follows the default route. So try setting the default route back to WAN1 and make sure that changes the working NTP responses back to that.
                            Then we should investigate what happened when you updated those pkgs that seemed to temporarily allow both WANs to work. See if you can replicate that by reinstalling those pkgs for example.

                            This is probably something low level in pf though.

                            THANK YOU SO MUCH about patience and help!

                            I purpose to going step-by-step and You just correct me if I doing something wrong. :)

                            —
                            CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
                            Help Ukraine to resist, save civilians people’s lives !
                            (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

                            1 Reply Last reply Reply Quote 0
                            • Sergei_ShablovskyS
                              Sergei_Shablovsky
                              last edited by Sergei_Shablovsky

                              Right now:

                              • on both WLAN1 and WAN2 in firewall rules for NTP "States Details" in "State" column
                                most of all connections are in MULTIPLE:MULTIPLE and SINGLE:MULTIPLE.
                              • ntpd are listening both WAN1 and WAN2 ("Diagnostics / Sockets" "IPv4 System Socket Information" table)

                              So is this mean that pf rules are working ok and NTP receive requests and answering ok ?

                              —
                              CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
                              Help Ukraine to resist, save civilians people’s lives !
                              (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

                              NollipfSenseN stephenw10S 2 Replies Last reply Reply Quote 0
                              • NollipfSenseN
                                NollipfSense @Sergei_Shablovsky
                                last edited by

                                @Sergei_Shablovsky Thank you for reaching out via message...I read through the thread and Steve's diagnosing makes sense about the default WAN routing...I couldn't add anything more...

                                pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
                                pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

                                1 Reply Last reply Reply Quote 1
                                • stephenw10S
                                  stephenw10 Netgate Administrator @Sergei_Shablovsky
                                  last edited by

                                  @Sergei_Shablovsky said in NTP not answering on 2-nd uplink WAN:

                                  So is this mean that pf rules are working ok and NTP receive requests and answering ok ?

                                  NTPd is fine. pf appears to be opening states correctly but what doesn't appear to be happening is the replies going back out via the correct gateway.

                                  So did you confirm that moving the default gateway back to WAN1 switches the working WAN for NTP?

                                  Sergei_ShablovskyS 2 Replies Last reply Reply Quote 0
                                  • Sergei_ShablovskyS
                                    Sergei_Shablovsky @stephenw10
                                    last edited by

                                    @stephenw10 said in NTP not answering on 2-nd uplink WAN:

                                    @Sergei_Shablovsky said in NTP not answering on 2-nd uplink WAN:

                                    So is this mean that pf rules are working ok and NTP receive requests and answering ok ?

                                    NTPd is fine. pf appears to be opening states correctly but what doesn't appear to be happening is the replies going back out via the correct gateway.

                                    I have the same decision. But the gateway are directly set in “Advanced Option / Gateway” in pf rule.

                                    In which case that’s may be not enough for ntpd (or any other service?) answers going out this “directly set Gateway”?

                                    —
                                    CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
                                    Help Ukraine to resist, save civilians people’s lives !
                                    (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

                                    stephenw10S 1 Reply Last reply Reply Quote 0
                                    • Sergei_ShablovskyS
                                      Sergei_Shablovsky @stephenw10
                                      last edited by

                                      @stephenw10 said in NTP not answering on 2-nd uplink WAN:

                                      So did you confirm that moving the default gateway back to WAN1 switches the working WAN for NTP?

                                      Previously I starting two(2) instance of pcap (different tty), one for WLAN1 + one for WLAN2, and than merge both .pcap in WireShark to see how answers happened.

                                      Is this method correct?

                                      —
                                      CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
                                      Help Ukraine to resist, save civilians people’s lives !
                                      (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator @Sergei_Shablovsky
                                        last edited by

                                        @Sergei_Shablovsky said in NTP not answering on 2-nd uplink WAN:

                                        I have the same decision. But the gateway are directly set in “Advanced Option / Gateway” in pf rule.

                                        Ah wait you set the gateway on the inbound WAN pass rules for NTP queries? You should not set a gateway there. The fact traffic is passed on an interface with a gateway is sufficient to tag it reply-to for that gateway. Setting a gateway on an inbound rule is incorrect as it will try to force traffic that way.

                                        Sergei_ShablovskyS 1 Reply Last reply Reply Quote 0
                                        • Sergei_ShablovskyS
                                          Sergei_Shablovsky @stephenw10
                                          last edited by

                                          @stephenw10 said in NTP not answering on 2-nd uplink WAN:

                                          @Sergei_Shablovsky said in NTP not answering on 2-nd uplink WAN:

                                          I have the same decision. But the gateway are directly set in “Advanced Option / Gateway” in pf rule.

                                          Ah wait you set the gateway on the inbound WAN pass rules for NTP queries?

                                          Exactly.

                                          You should not set a gateway there.

                                          At the start of topic I have a “Default” as Gateway in “Advanced Options”.
                                          But issue still exist, so I decide to set directly.

                                          The fact traffic is passed on an interface with a gateway is sufficient to tag it reply-to for that gateway. Setting a gateway on an inbound rule is incorrect as it will try to force traffic that way.

                                          Ok, set “Default” in “Advanced Options” back. ;)
                                          And reboot.

                                          But issue still exist.

                                          —
                                          CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
                                          Help Ukraine to resist, save civilians people’s lives !
                                          (Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.)

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10S
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            Ok make sure the states still appear the same for connections on both WANs without the gateway set on the rules.
                                            Make sure the working WAN still follows the system default gateway.

                                            Sergei_ShablovskyS 1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.