Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Dual WAN - Simultaneous packetloss/latency alarm

    Scheduled Pinned Locked Moved Routing and Multi WAN
    21 Posts 5 Posters 1.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      brewha12
      last edited by brewha12

      Hello,

      Dual WAN connection on a pfSense install on a 4-port quotum hardware with intel NICs.
      1 connection is Cable, 1 connection is ADSL.
      Usually I don't notice the interruption, but the gateway logs show alarm latency on both gateways simultaneous several times through the day. Each gateway has a different monitor IP.
      Testing ICMP from clients on the network, 1 out Cable, 1 out ADSL confirms the temporary packet drop.

      M 1 Reply Last reply Reply Quote 0
      • M
        michmoor LAYER 8 Rebel Alliance @brewha12
        last edited by

        @brewha12 does each gateway have the same monitor IP?

        Firewall: NetGate,Palo Alto-VM,Juniper SRX
        Routing: Juniper, Arista, Cisco
        Switching: Juniper, Arista, Cisco
        Wireless: Unifi, Aruba IAP
        JNCIP,CCNP Enterprise

        B 1 Reply Last reply Reply Quote 0
        • B
          brewha12 @michmoor
          last edited by brewha12

          @michmoor each have a different monitor IP. I think the alerts might be legitiment as well. 2 LAN clients also log spikes in my continus ping tests:

          From client on LAN at the same time pfSense logs latency alarms on each WAN:

          Tue 02/07/2023 14:22:07.18 Reply from 8.8.4.4: bytes=32 time=20ms TTL=114
          Tue 02/07/2023 14:22:08.21 Reply from 8.8.4.4: bytes=32 time=17ms TTL=114
          Tue 02/07/2023 14:22:12.13 Request timed out.
          Tue 02/07/2023 14:22:13.16 Reply from 8.8.4.4: bytes=32 time=20ms TTL=114

          M B 3 Replies Last reply Reply Quote 0
          • M
            michmoor LAYER 8 Rebel Alliance @brewha12
            last edited by

            @brewha12 IMO,
            Could be both LECs share the same path somewhere? Perhaps the trouble is within the path toward your monitor IPs [maybe they are in the same network]? Could be a few things tbh.

            Not sure you've done it but i would try to set up a monitor IP within your ISP's infrastructure like their DNS servers. This rules out a path issue outside the carrier's influence. If your clients are still seeing packet loss but not your monitor IPs [which are pointed to your ISPs dns server for example] then we know its an upstream path issue and you cant do anything about it anyway.

            Firewall: NetGate,Palo Alto-VM,Juniper SRX
            Routing: Juniper, Arista, Cisco
            Switching: Juniper, Arista, Cisco
            Wireless: Unifi, Aruba IAP
            JNCIP,CCNP Enterprise

            B 1 Reply Last reply Reply Quote 0
            • B
              brewha12 @brewha12
              last edited by

              @brewha12 Interesting info, thank you!
              So right now, I have all clients on the same LAN, and a few static rules to send 3 specific clients with static IPs out WAN#2. Every other connection defaults out WAN#1(Cable)
              I will switch the respective gateway monitor IPs to an IP on each seperate ISP...thanks for the idea.

              Dobby_D 1 Reply Last reply Reply Quote 0
              • Dobby_D
                Dobby_ @brewha12
                last edited by

                @brewha12

                May be it would be nice to set up load balancing and you will get fail over on top of it, as a site effect. You can try out;

                • session based load balancing
                • service based load balancing
                • policy based load balancing

                #~. @Dobby

                Turris Omnia - 4 Ports - 2 GB RAM / TurrisOS 7 Release (Btrfs)
                PC Engines APU4D4 - 4 Ports - 4 GB RAM / pfSense CE 2.7.2 Release (ZFS)
                PC Engines APU6B4 - 4 Ports - 4 GB RAM / pfSense+ (Plus) 24.03_1 Release (ZFS)

                B 1 Reply Last reply Reply Quote 0
                • B
                  brewha12 @Dobby_
                  last edited by

                  @dobby_ The primary WAN is 1Gbps Cable and secondary is 50Mbps DSL that easily can get saturated, so I was kind of hesitant on the load balancing. The 50Mbps is "mission critical" traffic.

                  1 Reply Last reply Reply Quote 0
                  • B
                    brewha12 @brewha12
                    last edited by

                    @brewha12 When I manually remove the monitor IP for each gateway, it seems to auto assign the gateway IP as the monitor IP...is this OK?

                    M S 2 Replies Last reply Reply Quote 0
                    • M
                      michmoor LAYER 8 Rebel Alliance @brewha12
                      last edited by

                      @brewha12 per documentation that is expected behavior.

                      Firewall: NetGate,Palo Alto-VM,Juniper SRX
                      Routing: Juniper, Arista, Cisco
                      Switching: Juniper, Arista, Cisco
                      Wireless: Unifi, Aruba IAP
                      JNCIP,CCNP Enterprise

                      1 Reply Last reply Reply Quote 0
                      • S
                        SteveITS Galactic Empire @brewha12
                        last edited by

                        @brewha12 said in Dual WAN - Simultaneous packetloss/latency alarm:

                        @brewha12 When I manually remove the monitor IP for each gateway, it seems to auto assign the gateway IP as the monitor IP...is this OK?

                        Depends...if the ISP modem is providing NAT then it's kind of useless because typically the local modem/router will be on even if the ISP has an outage. So you probably want an IP outside your office.

                        FYI you can actually control what types of traffic can fail over, see
                        https://docs.netgate.com/pfsense/en/latest/multiwan/policy-route.html

                        Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                        When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                        Upvote 👍 helpful posts!

                        M 1 Reply Last reply Reply Quote 0
                        • B
                          brewha12 @michmoor
                          last edited by brewha12

                          @michmoor I've inputted a DNS server for gateway monitor IP onWAN#2 DSL connection.. Both my Cable modem and DSL modem are in bypass/bridged mode so no NAT or traffic shaping AFAIK is happening.
                          I guess beyond this, if the issue persists, could it be the hardware I'm using to run pfSense.

                          1 Reply Last reply Reply Quote 0
                          • M
                            michmoor LAYER 8 Rebel Alliance @SteveITS
                            last edited by

                            @steveits yep! I use my service providers DNS service as a monitor.

                            Firewall: NetGate,Palo Alto-VM,Juniper SRX
                            Routing: Juniper, Arista, Cisco
                            Switching: Juniper, Arista, Cisco
                            Wireless: Unifi, Aruba IAP
                            JNCIP,CCNP Enterprise

                            R 1 Reply Last reply Reply Quote 0
                            • R
                              rcoleman-netgate Netgate @michmoor
                              last edited by

                              @michmoor The only downside of that is you don't know if they are having an upstream outage in that situation.

                              Ryan
                              Repeat, after me: MESH IS THE DEVIL! MESH IS THE DEVIL!
                              Requesting firmware for your Netgate device? https://go.netgate.com
                              Switching: Mikrotik, Netgear, Extreme
                              Wireless: Aruba, Ubiquiti

                              M 1 Reply Last reply Reply Quote 0
                              • M
                                michmoor LAYER 8 Rebel Alliance @rcoleman-netgate
                                last edited by

                                @rcoleman-netgate Agreed. Per the documentation, it does state to use the ISPs dns server. I tend to shy away from Google DNS or Cloudflare as they are not meant to be a source of ping(reachability).

                                Do you have a suggestion on what one should monitor?

                                Funny enough im looking into some outages i had around 2am today. Multiple monitoring endpoints just stopped responding. There was some packet loss on my WAN_DHCP gateway but i dont think that was the problem. More likely something upstream but cant really prove that out. If theres a better method im all ears.
                                Is there a way to monitor multiple IPs?

                                Firewall: NetGate,Palo Alto-VM,Juniper SRX
                                Routing: Juniper, Arista, Cisco
                                Switching: Juniper, Arista, Cisco
                                Wireless: Unifi, Aruba IAP
                                JNCIP,CCNP Enterprise

                                R 1 Reply Last reply Reply Quote 0
                                • R
                                  rcoleman-netgate Netgate @michmoor
                                  last edited by

                                  @michmoor said in Dual WAN - Simultaneous packetloss/latency alarm:

                                  Do you have a suggestion on what one should monitor?

                                  I use Google. ¯\_(ツ)_/¯

                                  Ryan
                                  Repeat, after me: MESH IS THE DEVIL! MESH IS THE DEVIL!
                                  Requesting firmware for your Netgate device? https://go.netgate.com
                                  Switching: Mikrotik, Netgear, Extreme
                                  Wireless: Aruba, Ubiquiti

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    brewha12
                                    last edited by brewha12

                                    To me it just doesn't make sense that both WAN connections, different physical modems, ISPs, and lines, experience simultaneous packet loss/drops. Is there any possibility of it being hardware, config, etc. on my pfSense miniPC?

                                    Feb 8 07:12:55 dpinger 4294 send_interval 2500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 2500ms latency_alarm 500ms loss_alarm 20% dest_addr 209.202.xx bind_addr 209.202.xx identifier "WAN_DHCP "
                                    Feb 8 07:12:55 dpinger 4524 send_interval 2000ms loss_interval 2500ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 2500ms latency_alarm 500ms loss_alarm 20% dest_addr 198.251.xx bind_addr 104.158.xx identifier "dsl_ig2 "
                                    Feb 8 07:12:47 dpinger 70704 WAN_DHCP 209.202.xx: sendto error: 50
                                    Feb 8 07:12:47 dpinger 22960 send_interval 2000ms loss_interval 2500ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 2500ms latency_alarm 500ms loss_alarm 20% dest_addr 198.251xx bind_addr 104.158.xx identifier "dsl_ig2 "
                                    Feb 8 05:01:37 dpinger 70704 send_interval 2500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 2500ms latency_alarm 500ms loss_alarm 20% dest_addr 209.202.xx bind_addr 209.202.xx identifier "WAN_DHCP "
                                    Feb 8 05:01:37 dpinger 71358 send_interval 2000ms loss_interval 2500ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 2500ms latency_alarm 500ms loss_alarm 20% dest_addr 198.251xx bind_addr 104.158.xx identifier "dsl_ig2 "

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      brewha12
                                      last edited by

                                      Some additional info:
                                      Primary WAN/default is Cable, Secondary WAN is DSL.
                                      When I unplug the network cable on the Cable-WAN from my router, my secondary WAN/DSL is briefly experiencing packetloss.
                                      2 clients timed out using a continuous ICMP via static rule out secondary-DSL when unplugging cable on primary-WAN.
                                      I don't get why DSL would be interrupted when it has it's own static rules.

                                      M B 2 Replies Last reply Reply Quote 0
                                      • M
                                        michmoor LAYER 8 Rebel Alliance @brewha12
                                        last edited by

                                        @brewha12 Hmm..The monitor IP isnt pointing to the other right? So cable modem isnt using the DSL Modem as the montior IP?
                                        I assume not as i can see that as the issue.
                                        Assuming it isnt.....im at a lost tbh.

                                        Firewall: NetGate,Palo Alto-VM,Juniper SRX
                                        Routing: Juniper, Arista, Cisco
                                        Switching: Juniper, Arista, Cisco
                                        Wireless: Unifi, Aruba IAP
                                        JNCIP,CCNP Enterprise

                                        1 Reply Last reply Reply Quote 0
                                        • B
                                          brewha12 @brewha12
                                          last edited by

                                          @brewha12 DSL is using DNS from ISP as GW monitor...thanks for your help.

                                          M B 2 Replies Last reply Reply Quote 0
                                          • M
                                            michmoor LAYER 8 Rebel Alliance @brewha12
                                            last edited by

                                            @brewha12 Do you have both WAN connections plugging into a switch OR do they go direclty into their modems

                                            Firewall: NetGate,Palo Alto-VM,Juniper SRX
                                            Routing: Juniper, Arista, Cisco
                                            Switching: Juniper, Arista, Cisco
                                            Wireless: Unifi, Aruba IAP
                                            JNCIP,CCNP Enterprise

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.