New HA / DualWAN, NAT Outbound rule breaks internet connection.

bac0n8t0r

I have setup two exact same routers, after following HA Multi-WAN doc and watching the hangouts as well as Lawrence systems video on setting HA up. I have everything working expect when I add the outbound NAT rule to use the carp WAN and CARP wan 2.
** for any reference to my IPs I am just going to be using the once listed on the doc , any logs or such I will replace with those IPs **
I tried both hybrid and manual mode , each time if the NAT Address is set to the CARP IP. I loose internet connection I am unable to ping from client connected or pfsense itself or get a webpage .
The only part I am unsure if I have set correctly is this part of the docs..

Firewall Configuration
With Multi-WAN a firewall rule must be in place to pass traffic to local networks using the default gateway. Otherwise, when traffic attempts to reach the CARP address or from LAN to DMZ it will instead go out a WAN connection.

A rule must be added at the top of the firewall rules for all internal interfaces which will direct traffic for all local networks to the default gateway. The important part is the gateway needs to be default for this rule and not one of the failover or load balance gateway groups. The destination for this rule would be the local LAN network, or an alias containing any locally reachable networks.

its says all internal interfaces, does this mean my LAN or WANs? ( I ask cause it also says that the destination for this rule needs to be the local lan. TBH I tried both and neither seemed to resolve my problem.
it seems that it is using the CARP if im understanding this packet capture

Packet Capture - WAN 2 - Protocol CARP

08:20:51.214057 IP 203.0.113.11 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=0 authlen=7 counter=8919520798374983921
08:20:52.273064 IP 203.0.113.11 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=0 authlen=7 counter=8919520798374983921

Packet Capture - LAN - Protocol CARP

08:32:52.399001 IP 192.168.1.2 > 224.0.0.18: CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 authlen=7 counter=3842395158048380786
08:32:53.439754 IP 192.168.1.2 > 224.0.0.18: CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 authlen=7 counter=3842395158048380786

This is my current NAT I have added
NAT Outbound

If I can provide anything that will help determine how to fix this please let me know. I have been trying to solve this for the past couple days to no success.
Thank you for any help.

viragomann

@bac0n8t0r said in New HA / DualWAN, NAT Outbound rule breaks internet connection.:

I tried both hybrid and manual mode , each time if the NAT Address is set to the CARP IP

Is your real CARP a public IP?
Is it defined with the same subnet mask as your primary WAN IP, so that it is within the same subnet as the upstream gateway?

@bac0n8t0r said in New HA / DualWAN, NAT Outbound rule breaks internet connection.:

The only part I am unsure if I have set correctly is this part of the docs..

This relates only to internal traffic between your local networks.

@bac0n8t0r said in New HA / DualWAN, NAT Outbound rule breaks internet connection.:

it seems that it is using the CARP if im understanding this packet capture

These are only CARP advertisements, no payload.

bac0n8t0r

@viragomann said in New HA / DualWAN, NAT Outbound rule breaks internet connection.:

@bac0n8t0r said in New HA / DualWAN, NAT Outbound rule breaks internet connection.:

I tried both hybrid and manual mode , each time if the NAT Address is set to the CARP IP

Is your real CARP a public IP?
Yes it is real I have a /29 for both WANs
Is it defined with the same subnet mask as your primary WAN IP, so that it is within the same subnet as the upstream gateway?
Yes when I made the CARP VIP I made sure it had the same /29 subnet

@bac0n8t0r said in New HA / DualWAN, NAT Outbound rule breaks internet connection.:

The only part I am unsure if I have set correctly is this part of the docs..

This relates only to internal traffic between your local networks.
ok, thanks, that explains why it didn't resolve anything for me.

@bac0n8t0r said in New HA / DualWAN, NAT Outbound rule breaks internet connection.:

it seems that it is using the CARP if im understanding this packet capture

These are only CARP advertisements, no payload.
Ok, is there a way I need to run it or do something other? I was just running this as its shown in the google hangouts netgate on multi wan, and it seemed similiar.
I reran it with the protocol set to ANY and this is the result.

10:31:29.820523 IP 203.0.113.11 > 203.0.113.09 (WAN2 GW): ICMP echo request, id 58866, seq 9682, length 9
10:31:29.820899 IP 203.0.113.09 (WAN2 GW) > 203.0.113.11: ICMP echo reply, id 58866, seq 9682, length 9
10:31:29.932539 IP 185.180.13.82.9993 > 203.0.113.11.32999: UDP, length 181
10:31:29.932859 IP 203.0.113.11.32999 > 185.180.13.82.9993: UDP, length 67
10:31:30.352517 IP 203.0.113.11 > 203.0.113.09 (WAN2 GW): ICMP echo request, id 58866, seq 9683, length 9
10:31:30.352519 IP 203.0.113.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36
10:31:30.352896 IP 203.0.113.09 (WAN2 GW) > 203.0.113.11: ICMP echo reply, id 58866, seq 9683, length 9
10:31:30.590613 IP 167.172.147.227.443 > 203.0.113.11.34013: tcp 31
10:31:30.590999 IP 203.0.113.11.34013 > 167.172.147.227.443: tcp 35
10:31:30.648722 IP 167.172.147.227.443 > 203.0.113.11.34013: tcp 0
10:31:30.884772 IP 203.0.113.11 > 203.0.113.09 (WAN2 GW): ICMP echo request, id 58866, seq 9684, length 9
10:31:30.885414 IP 203.0.113.09 (WAN2 GW) > 203.0.113.11: ICMP echo reply, id 58866, seq 9684, length 9
10:31:31.356262 IP 203.0.113.11 > 224.0.0.18: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36
10:31:31.402418 IP 203.0.113.11 > 203.0.113.09 (WAN2 GW): ICMP echo request, id 58866, seq 9685, length 9
10:31:31.402891 IP 203.0.113.09 (WAN2 GW) > 203.0.113.11: ICMP echo reply, id 58866, seq 9685, length 9
10:31:31.419722 ARP, Request who-has 65.144.241.162 tell 203.0.113.09 (WAN2 GW), length 46

This is without the NAT Outbound rule , which obviously shows traffic off the 113.11 address which is the IP on the primary router not the CARP IP

This is with the NAT Outbound rule to use CARP IP

10:50:02.429191 IP 203.0.113.10(WAN 2 CARP).15229 > 155.70.55.91.443: tcp 0
10:50:02.476659 IP 203.0.113.10(WAN 2 CARP).17194 > 155.70.118.10.443: tcp 0
10:50:02.476829 IP 203.0.113.10(WAN 2 CARP).7297 > 155.70.118.10.443: tcp 0
10:50:02.500235 IP 203.0.113.10(WAN 2 CARP).63105 > 104.16.148.64.443: tcp 0
10:50:02.542179 IP 203.0.113.10(WAN 2 CARP).44391 > 23.55.249.2.443: tcp 0
10:50:02.574170 IP 203.0.113.10(WAN 2 CARP).60445 > 23.40.179.27.443: tcp 0
10:50:02.579300 IP 203.0.113.10(WAN 2 CARP).24709 > 216.58.192.110.443: tcp 0
10:50:02.602073 IP 203.0.113.10(WAN 2 CARP).32060 > 13.224.42.75.443: tcp 0
10:50:02.602090 IP 203.0.113.10(WAN 2 CARP).48487 > 104.126.206.201.443: tcp 0
10:50:02.664195 IP 203.0.113.10(WAN 2 CARP).47999 > 13.33.71.221.443: tcp 0
10:50:02.693796 IP 203.0.113.11(WAN 2 Primary PFsense) > 203.0.113.09(WAN 2 GW): ICMP echo request, id 58866, seq 11787, length 9
10:50:02.695217 IP 203.0.113.09(WAN 2 GW) > 203.0.113.11(WAN 2 Primary PFsense): ICMP echo reply, id 58866, seq 11787, length 9
10:50:02.706941 IP 194.147.140.62.41006 > 203.0.113.10(WAN 2 CARP).17787: tcp 0
10:50:02.708232 IP 203.0.113.10(WAN 2 CARP).60537 > 104.17.72.206.443: tcp 0
10:50:02.754657 ARP, Request who-has 203.0.113.10(WAN 2 CARP) tell 203.0.113.09(WAN 2 GW), length 46
10:50:02.754663 ARP, Request who-has 203.0.113.14(**this was the last IP in the /29, I do not have this set anywhere?!?) tell 203.0.113.09(WAN 2 GW), length 46
10:50:02.834262 IP 203.0.113.10(WAN 2 CARP).56591 > 216.58.192.110.443: tcp 0
10:50:02.849199 IP 203.0.113.10(WAN 2 CARP).65340 > 13.108.232.15.443: tcp 0
10:50:03.174182 IP 203.0.113.11(WAN 2 Primary PFsense) > 224.0.0.18: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36
10:50:03.226040 IP 203.0.113.11(WAN 2 Primary PFsense) > 203.0.113.09(WAN 2 GW): ICMP echo request, id 58866, seq 11788, length 9
10:50:03.226569 IP 203.0.113.09(WAN 2 GW) > 203.0.113.11(WAN 2 Primary PFsense): ICMP echo reply, id 58866, seq 11788, length 9
10:50:03.528613 IP 203.0.113.10(WAN 2 CARP).43066 > 34.196.144.134.443: tcp 0
10:50:03.693816 IP 185.180.13.82.9993 > 203.0.113.11(WAN 2 Primary PFsense).32999: UDP, length 102
10:50:03.694166 IP 203.0.113.11(WAN 2 Primary PFsense).32999 > 185.180.13.82.9993: UDP, length 67

The one odd thing is on the second ARP request the IP is the last in my /29 for WAN 2, I have never set this address to anything and went thru to double check all configured IPs and all were not set to that and set correctly.

viragomann

@bac0n8t0r said in New HA / DualWAN, NAT Outbound rule breaks internet connection.:

The one odd thing is on the second ARP request the IP is the last in my /29 for WAN 2, I have never set this address to anything and went thru to double check all configured IPs and all were not set to that and set correctly.

That request came from the gateway. So I reason, that IP was requested from the internet.

The first ARP line worries me quite more. The gateway is asking for the CARP VIP, which is owned by the device, which your are sniffing the packets on, but there is no response.

No idea why with the provided infos.
Is there al right with the CARP status?

bac0n8t0r

@viragomann said in New HA / DualWAN, NAT Outbound rule breaks internet connection.:

@bac0n8t0r said in New HA / DualWAN, NAT Outbound rule breaks internet connection.:

The one odd thing is on the second ARP request the IP is the last in my /29 for WAN 2, I have never set this address to anything and went thru to double check all configured IPs and all were not set to that and set correctly.

That request came from the gateway. So I reason, that IP was requested from the internet.

The first ARP line worries me quite more. The gateway is asking for the CARP VIP, which is owned by the device, which your are sniffing the packets on, but there is no response.
Yes I dont understand why I cant get a response when I use the CARP address. Also earlier I wanted to be certain that the carp IP i was using was fine and so I switched CARP and the one set on the Primary router, and I had an internet connection fine as long as the NAT outbound wasnt set to the CARP IP. Which I knew I had all the IPs but i wanted a sanity check.

No idea why with the provided infos.
Is there al right with the CARP status?

Yup under Status CARP everything on primary green with Master and Secondary says backup, this is what you meant correct?

viragomann

@bac0n8t0r
Yes, due to the capture it might be the case, that the secondary is master on WAN2 and responses to the ARP request. So you wouldn't see the response on the primary.

bac0n8t0r

@viragomann If I were to say turn off the backup machineor just unplug Wan2 from secondary, the primary should be the only thing that can be Master for wan2? But also I guess i need to get to the root of why the secondary would be claiming master but under carp status it shows primary as master.

bac0n8t0r

Heres a picture of the primary pfsense
alt text

bac0n8t0r

I Just wanted to update, came in today and just reset both machines to factory and started again, all seems to be working, fine. So I must have done something wrong or out of order. But thanks to all who commented.