Dual WAN - Simultaneous packetloss/latency alarm

[brewha12]

Hello,

Dual WAN connection on a pfSense install on a 4-port quotum hardware with intel NICs.
1 connection is Cable, 1 connection is ADSL.
Usually I don't notice the interruption, but the gateway logs show alarm latency on both gateways simultaneous several times through the day. Each gateway has a different monitor IP.
Testing ICMP from clients on the network, 1 out Cable, 1 out ADSL confirms the temporary packet drop.

[michmoor]

@brewha12 does each gateway have the same monitor IP?

[brewha12]

@michmoor each have a different monitor IP. I think the alerts might be legitiment as well. 2 LAN clients also log spikes in my continus ping tests:

From client on LAN at the same time pfSense logs latency alarms on each WAN:

Tue 02/07/2023 14:22:07.18 Reply from 8.8.4.4: bytes=32 time=20ms TTL=114
Tue 02/07/2023 14:22:08.21 Reply from 8.8.4.4: bytes=32 time=17ms TTL=114
Tue 02/07/2023 14:22:12.13 Request timed out.
Tue 02/07/2023 14:22:13.16 Reply from 8.8.4.4: bytes=32 time=20ms TTL=114

[michmoor]

@brewha12 IMO,
Could be both LECs share the same path somewhere? Perhaps the trouble is within the path toward your monitor IPs [maybe they are in the same network]? Could be a few things tbh.

Not sure you've done it but i would try to set up a monitor IP within your ISP's infrastructure like their DNS servers. This rules out a path issue outside the carrier's influence. If your clients are still seeing packet loss but not your monitor IPs [which are pointed to your ISPs dns server for example] then we know its an upstream path issue and you cant do anything about it anyway.

[brewha12]

@brewha12 Interesting info, thank you!
So right now, I have all clients on the same LAN, and a few static rules to send 3 specific clients with static IPs out WAN#2. Every other connection defaults out WAN#1(Cable)
I will switch the respective gateway monitor IPs to an IP on each seperate ISP...thanks for the idea.

[Dobby_]

@brewha12

May be it would be nice to set up load balancing and you will get fail over on top of it, as a site effect. You can try out;

• session based load balancing
• service based load balancing
• policy based load balancing

[brewha12]

@dobby_ The primary WAN is 1Gbps Cable and secondary is 50Mbps DSL that easily can get saturated, so I was kind of hesitant on the load balancing. The 50Mbps is "mission critical" traffic.

[brewha12]

@brewha12 When I manually remove the monitor IP for each gateway, it seems to auto assign the gateway IP as the monitor IP...is this OK?

[michmoor]

@brewha12 per documentation that is expected behavior.

[SteveITS]

@brewha12 said in Dual WAN - Simultaneous packetloss/latency alarm:

@brewha12 When I manually remove the monitor IP for each gateway, it seems to auto assign the gateway IP as the monitor IP...is this OK?

Depends...if the ISP modem is providing NAT then it's kind of useless because typically the local modem/router will be on even if the ISP has an outage. So you probably want an IP outside your office.

FYI you can actually control what types of traffic can fail over, see
https://docs.netgate.com/pfsense/en/latest/multiwan/policy-route.html

[brewha12]

@michmoor I've inputted a DNS server for gateway monitor IP onWAN#2 DSL connection.. Both my Cable modem and DSL modem are in bypass/bridged mode so no NAT or traffic shaping AFAIK is happening.
I guess beyond this, if the issue persists, could it be the hardware I'm using to run pfSense.

[michmoor]

@steveits yep! I use my service providers DNS service as a monitor.

[rcoleman-netgate]

@michmoor The only downside of that is you don't know if they are having an upstream outage in that situation.

[michmoor]

@rcoleman-netgate Agreed. Per the documentation, it does state to use the ISPs dns server. I tend to shy away from Google DNS or Cloudflare as they are not meant to be a source of ping(reachability).

Do you have a suggestion on what one should monitor?

Funny enough im looking into some outages i had around 2am today. Multiple monitoring endpoints just stopped responding. There was some packet loss on my WAN_DHCP gateway but i dont think that was the problem. More likely something upstream but cant really prove that out. If theres a better method im all ears.
Is there a way to monitor multiple IPs?

[rcoleman-netgate]

@michmoor said in Dual WAN - Simultaneous packetloss/latency alarm:

Do you have a suggestion on what one should monitor?

I use Google. ¯\_(ツ)_/¯

[brewha12]

To me it just doesn't make sense that both WAN connections, different physical modems, ISPs, and lines, experience simultaneous packet loss/drops. Is there any possibility of it being hardware, config, etc. on my pfSense miniPC?

Feb 8 07:12:55 dpinger 4294 send_interval 2500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 2500ms latency_alarm 500ms loss_alarm 20% dest_addr 209.202.xx bind_addr 209.202.xx identifier "WAN_DHCP "
Feb 8 07:12:55 dpinger 4524 send_interval 2000ms loss_interval 2500ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 2500ms latency_alarm 500ms loss_alarm 20% dest_addr 198.251.xx bind_addr 104.158.xx identifier "dsl_ig2 "
Feb 8 07:12:47 dpinger 70704 WAN_DHCP 209.202.xx: sendto error: 50
Feb 8 07:12:47 dpinger 22960 send_interval 2000ms loss_interval 2500ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 2500ms latency_alarm 500ms loss_alarm 20% dest_addr 198.251xx bind_addr 104.158.xx identifier "dsl_ig2 "
Feb 8 05:01:37 dpinger 70704 send_interval 2500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 2500ms latency_alarm 500ms loss_alarm 20% dest_addr 209.202.xx bind_addr 209.202.xx identifier "WAN_DHCP "
Feb 8 05:01:37 dpinger 71358 send_interval 2000ms loss_interval 2500ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 2500ms latency_alarm 500ms loss_alarm 20% dest_addr 198.251xx bind_addr 104.158.xx identifier "dsl_ig2 "

[brewha12]

Some additional info:
Primary WAN/default is Cable, Secondary WAN is DSL.
When I unplug the network cable on the Cable-WAN from my router, my secondary WAN/DSL is briefly experiencing packetloss.
2 clients timed out using a continuous ICMP via static rule out secondary-DSL when unplugging cable on primary-WAN.
I don't get why DSL would be interrupted when it has it's own static rules.

[michmoor]

@brewha12 Hmm..The monitor IP isnt pointing to the other right? So cable modem isnt using the DSL Modem as the montior IP?
I assume not as i can see that as the issue.
Assuming it isnt.....im at a lost tbh.

[brewha12]

@brewha12 DSL is using DNS from ISP as GW monitor...thanks for your help.

[michmoor]

@brewha12 Do you have both WAN connections plugging into a switch OR do they go direclty into their modems