Backup CARP router unable to access internet

samelmore · Aug 26, 2017, 1:15 AM

I currently have an HA setup between two pfSense 2.3.4 routers. CARP failover works just fine. However, the backup CARP router cannot access the internet. I have advanced/manual outbound NAT rules setup for each of my subnets using the external CARP address, so all traffic is coming from the VIP's from those networks (going out the WAN). For the 127.0.0.0/8 network outbound NAT (WAN interface), I have tried NO NAT, I have tried the VIP WAN IP, as well as the Interface Address. None seem to allow the backup CARP router to access the internet. This is a problem because, as one example, I cannot check for updates from the backup router without switching it to master first.

To troubleshoot, I have verified that when pinging from the backup CARP router to the internet, the packet goes out using the WAN VIP. Therefore, if I packet capture on the master CARP router, it sees the ICMP packets returned to it, so they're ultimately not getting back to the backup router.

What outbound NAT rule do I need to enable this to work (to be able to ping/access from backup router to internet addresses)? Are the outbound rules even the problem in this case?

Thanks,
Sam

Derelict · Aug 26, 2017, 1:28 AM

Do you have a public IP address for the primary interface, secondary interface, and the CARP VIP on WAN?

I have verified that when pinging from the backup CARP router to the internet, the packet goes out using the WAN VIP.

Traffic from the firewall itself should go out sourced from the interface address, not the CARP VIP.

If the secondary is sending traffic from the firewall out WAN and translating the source address to the CARP VIP, your outbound NAT rules are wrong.

Source any there is not what you want. Neither is source WAN net or WAN address.

samelmore · Aug 26, 2017, 1:39 AM

Yes, I understand I don't want packets from the pfSense to go out as the WAN VIP. I have tried using the outbound rule as follows, which still fails:

Interface: WAN
Source: 127.0.0.0/8
Destination: any
NAT Address: Interface Address

This still results in ping packets originating from the backup pfSense router (using Diagnostics -> Ping) coming from the WAN VIP. I'm unsure how to troubleshoot further, as the above outbound NAT rule seems like it should work correctly.

Thanks,
Sam

Derelict · Aug 26, 2017, 2:19 AM

Yeah. Post your outbound NAT rules on the primary and secondary.

samelmore · Aug 26, 2017, 3:27 AM

Outbound NAT rules:

Iface WAN - src 127.0.0.0/8 - dst * - NAT: WAN Address
Iface WAN - src 192.168.187.0/24 - dst * - NAT: WAN VIP
Iface WAN - src 192.168.179.0/24 - dst * - NAT: WAN VIP
Iface WAN - src 192.168.21.0/24 - dst * - NAT: WAN VIP

Derelict · Aug 26, 2017, 4:38 AM

Outbound connections from the secondary are not sourced from 127.0.0.1. They are sourced from the interface address. There is no outbound NAT. In fact, it is normal to change those rules to use the CARP VIP if sourced from 127.0.0.0/8, not WAN address. Again, what you do not want to NAT is sourcing from the WAN subnet.

I cannot think of anything off the top of my head to tell you to check.

I'd need to see a lot more information about your firewall's config. The normal CARP config does not do what you are saying it is doing.

You might need to get over not providing WAN address information so we can actually see what is going on there.

pcaps on WAN interfaces for icmp on my lab HA pair:

Primary:
04:36:17.542757 IP 172.25.228.18 > 172.25.228.1: ICMP echo request, id 58300, seq 6383, length 8
04:36:17.542897 IP 172.25.228.1 > 172.25.228.18: ICMP echo reply, id 58300, seq 6383, length 8
04:36:18.070377 IP 172.25.228.18 > 172.25.228.1: ICMP echo request, id 58300, seq 6384, length 8
04:36:18.070492 IP 172.25.228.1 > 172.25.228.18: ICMP echo reply, id 58300, seq 6384, length 8
04:36:18.602750 IP 172.25.228.18 > 172.25.228.1: ICMP echo request, id 58300, seq 6385, length 8
04:36:18.602878 IP 172.25.228.1 > 172.25.228.18: ICMP echo reply, id 58300, seq 6385, length 8

Secondary:
04:34:26.972616 IP 172.25.228.19 > 172.25.228.1: ICMP echo request, id 35112, seq 17494, length 8
04:34:26.972759 IP 172.25.228.1 > 172.25.228.19: ICMP echo reply, id 35112, seq 17494, length 8
04:34:27.502601 IP 172.25.228.19 > 172.25.228.1: ICMP echo request, id 35112, seq 17495, length 8
04:34:27.502752 IP 172.25.228.1 > 172.25.228.19: ICMP echo reply, id 35112, seq 17495, length 8
04:34:28.032678 IP 172.25.228.19 > 172.25.228.1: ICMP echo request, id 35112, seq 17496, length 8
04:34:28.032828 IP 172.25.228.1 > 172.25.228.19: ICMP echo reply, id 35112, seq 17496, length 8

The CARP VIP is 172.26.228.17

Outbound NAT rules for WAN are attached.

![Screen Shot 2017-08-25 at 9.37.51 PM.png](/public/imported_attachments/1/Screen Shot 2017-08-25 at 9.37.51 PM.png)
![Screen Shot 2017-08-25 at 9.37.51 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2017-08-25 at 9.37.51 PM.png_thumb)

EdCazini · Nov 29, 2017, 3:43 AM

Hi, in my case (v 2.4.2-RELEASE (amd64) ), I solved it with the following settings:

Outbound NAT Mode = Hybrid Outbound NAT rule generation
Putting below as the 1st rule in Firewall/NAT/Outbound (in primary, will replicate to backup), see pic below

Iface=WAN - src=This Firewall - dst=* - NAT=WAN Address

P.S.

been using pfSense since 2006
but this is my 1st post. edit: oops my 2nd post actually, the 1st was way back 2015 :D
so I beg you tolerance if I woke up a sleeping thread or broke some forum rules
just wanted to help

Ed