Failover gateway in a multi-WAN setup has odd packets

[mig]

I run 2.4.4-RELEASE-p3 with two WANs: an unlimited cable connection (cheap, "main-WAN") and a MiFi to a local mobile tower (expensive, "WAN-via-MiFi"). Failover works OK but each packet via WAN-via-MiFi costs me something so when the main-WAN connection is up I need the firewall to only (infrequently) send packets to the WAN-via-MiFi "Monitor IP" (it uses 1.1.1.1). I can dial up the probe interval to reduce this cost.

However, when I capture packets via "WAN-via-MiFi" gateway (192.168.8.101) I see quite a lot of unexpected traffic to IP=9.9.9.9 which is the DNS server for the main-WAN connection specified in the "System/General Setup":

13:00:40.687525 IP 192.168.8.101 > 1.1.1.1: ICMP echo request, id 39212, seq 10888, length 8
13:00:40.708176 IP 1.1.1.1 > 192.168.8.101: ICMP echo reply, id 39212, seq 10888, length 8
13:00:41.188981 IP 192.168.8.101 > 1.1.1.1: ICMP echo request, id 39212, seq 10889, length 8
13:00:41.207163 IP 1.1.1.1 > 192.168.8.101: ICMP echo reply, id 39212, seq 10889, length 8
13:00:41.359524 IP 192.168.8.101.50727 > 9.9.9.9.853: tcp 0
13:00:41.387279 IP 9.9.9.9.853 > 192.168.8.101.50727: tcp 0
13:00:41.387326 IP 192.168.8.101.50727 > 9.9.9.9.853: tcp 0
13:00:41.387354 IP 192.168.8.101.50727 > 9.9.9.9.853: tcp 311
13:00:41.420094 IP 9.9.9.9.853 > 192.168.8.101.50727: tcp 0
13:00:41.422391 IP 9.9.9.9.853 > 192.168.8.101.50727: tcp 1350
13:00:41.422438 IP 192.168.8.101.50727 > 9.9.9.9.853: tcp 0
13:00:41.422847 IP 9.9.9.9.853 > 192.168.8.101.50727: tcp 1350
13:00:41.422875 IP 192.168.8.101.50727 > 9.9.9.9.853: tcp 0
13:00:41.422878 IP 9.9.9.9.853 > 192.168.8.101.50727: tcp 174
13:00:41.422901 IP 192.168.8.101.50727 > 9.9.9.9.853: tcp 0

What puzzles me is that the WAN-via-MiFi has a separate DNS server set via "System/General Setup"(and it is NOT 9.9.9.9). FWIW I have "DNS Resolver" enabled (with "DNSSEC" and "Forwarding mode" both on) and "Outgoing Network Interfaces" include both main-WAN and WAN-via-MiFi.

I would appreciate all ideas about preventing this "9.9.9.9" traffic going into my costly failover link while main-WAN is OK (obviously the solution has to keep the DNS working)

Thanks in advance!

[dotdash]

That's doing TLS to Quad9. Do you have the resolver enabled with forwarding?

[mig]

Yes, I do have DNSSEC enabled in the resolver and Forwarding is also enabled. Are you saying disabling Forwarding should remove the unwanted traffic while maintaining DNSSEC working during failover? (Can't test it now as this would upset all the users!)

Come to think of it a good way to additionally reduce expensive traffic over WAN-via-MiFi would be by switching to plain DNS when main-WAN is down. Is it possible with pfSense? If so - how?

[dotdash]

I'm just saying the traffic you are seeing is coming from unbound. If you have 'all' selected for outgoing network interfaces, I think it uses both of your wans for queries. I'm not familiar enough with unbound to know if you can make it use the WAN only and fail over to the secondary.

[mig]

Thanks, I'll poke around DNS resolver to see if I will be able to fix it without breaking DNS. Need to schedule a maintenance...

Do you think it is worth posting a question in DNS section of the forum about switching from DNSSEC to plain DNS when failover occurs?

[dotdash]

Sure, can't hurt to ask. I think what you'd need would be unbound querying via the failover gateway instead of using all available interfaces. It's still going to send queries even if it's not using DNSSEC.

[mig]

Switching from DNSSEC to plain DNS when a failover occurs is apparently not possible:
[https://forum.netgate.com/topic/150106/switching-from-dnssec-to-plain-dns-in-multi-wan-configuration]

[mig]

I was able to find a solution to this problem with the help of Netgate admin expert (@stephenw10) advice, the trick was to force Unbound to use "localhost" for outgoing queries. Unbound, when it is configured to operate via localhost outbound gateway would use only the gateway currently chosen by the system (i.e. dictated by the failover rules). This allowed me to completely eliminate unwanted unbound DNS/DNSSEC traffic via an expensive SIM-based failover link.

See more details in the following thread:

https://forum.netgate.com/topic/150176/php-shell-has-gatewaystatus-but-why-does-it-report-all-gateways-as-status-none