CARP with multi-wan [SOLVED]

chris4916 · Nov 9, 2017, 9:28 PM

I'm trying to configure CARP with multi-wan and can't figure out what I'm making wrong but, basically, it doesn't work as expected.

Modem Modem 1 Modem 2
mode PPPoE PPPoE
public IP xxx.xxx.xxx.xxx yyy.yyy.yyy.yyy
IP (internal side) 192.168.253.14/29 192.168.253.6/29
forward to 192.168.253.9/29 192.168.253.1/29

pfSense igb0 igb1 igb2 igb3 igb5
interface WAN1 WAN2 LAN DMZ SYNC
WAN master 192.168.253.10/29 192.168.253.2/29 192.168.0.252/24 192.168.200.252 192.168.253.17/29
WAN slave 192.168.253.11/29 192.168.253.3/29 192.168.0.253/24 192.168.200.253 192.168.253.18/29
VIP 192.168.253.9/29 192.168.253.1/29 192.168.0.254/24 192.168.200.254/24 none
WHID 2 1 3 4 none
dft GW 192.168.253.14 192.168.253.6 none none none

WAN1 & WAN2 are both member of "WAN" group, both tier-1, none is designed as "default gateway"
WAN is configured as gateway in all rules at LAN interface level when flow is reaching internet. This works fine.

CARP itself is almost OK (some problems with DHCP because some clients receive DHCP offer from each DHCP server with different IP) but the real problem I'm facing is with in-coming flow.

If I try to reach internal mail server from internet using modem 1 public IP, this works.
When I try to reach same server using modem 2 public IP, I can see packets reaching WAN2, even going out from LAN to mail server but it looks like packets are back using route through VIP with WHID 2 instead of WHID 1.

I did read this https://doc.pfsense.org/index.php/Bypassing_Policy_Routing and applied it without any success. (and I don't understand how this applies to non-local IPs)
I also have multiple IPsec tunnels, for the time being hardcoded to connect to either WAN1 or WAN2, depending on remote network and this works fine (yes I'm facing the issue with IPSec status preventing to display is correctly).

I, of course, changed outbound NAT rules, in manual mode, to target WAN VIPs

Well, I'm quite confused.
I feel this is related to routing, perhaps bypass policy but I've to admit that, for the time being, I'm lost with poor understanding (and not real capability to closely investigate because, to make it worst, I'm configuring everything remotely through OpenVPN access ;D ;D ;D :o

Any idea?

jimp · Nov 9, 2017, 7:44 PM

DHCP is probably giving you trouble because it sounds like you didn't configure the failover addresses for each DHCP interface. DHCP should be coordinating leases between the two units provided that failover is setup correctly in the DHCP settings (also, make sure to set a CARP VIP as the gateway and DNS server in the DHCP settings)

As for the reply traffic, it sounds like maybe your routing or gateways are not correct. Check that:

1. On WAN and WAN2 you have the gateway for the interface selected on Interfaces > WAN and Interfaces > WAN2
2. On your firewall rules, make sure you DO NOT use interface groups/rules for inbound WAN traffic. Traffic must use rules on the WAN interface tabs or else it cannot be properly tagged with reply-to so the firewall can remember the proper return path for traffic
3. Make sure that on your WAN interface rules, you DO NOT set a gateway on any rules on WAN tabs, this will definitely break things.

Somewhat related but not to that, make sure you do have at least one gateway marked default under System > Routing, and you do not need to manually set a gateway on every LAN rule if it's taking the default path. Only set a gateway group or alternate gateway if you need it to take a non-default path.

chris4916 · Nov 9, 2017, 9:23 PM

Thank you Jimp for the anwser ;)

1 - Regarding DHCP, I configured failover peer IP address, setting here physical IP address of slave pfSense. BTW, on slave pfSense, DHCP has been automatically configured with master physical IP as failover peer IP. I also defined, in DHCP server, CARP VIP, on the LAN side, as defaut gateway and DNS. This works. I mean that CARP work well. If I stop master pfSense, everything switches smoothly to slave server

And DHCP works, but not always. Here is what I collect in my rsyslog server:

Nov 9 06:51:54 192.168.0.253 dhcpd: DHCPREQUEST for 192.168.0.3 from 7e:d8:99:8f:32:a4 via igb2
Nov 9 06:51:54 192.168.0.253 dhcpd: DHCPACK on 192.168.0.3 to 7e:d8:99:8f:32:a4 via igb2
Nov 9 06:51:54 192.168.0.253 dhcpd: Unable to add forward map from webserver.rmo.ci to 192.168.0.3: REFUSED
Nov 9 06:52:30 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2: load balance to peer dhcp_opt3
Nov 9 06:52:28 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:29 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:34 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2: load balance to peer dhcp_opt3
Nov 9 06:52:32 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:32 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:35 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:38 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:35 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:39 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:43 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:46 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:43 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:46 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:02 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:59 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:02 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:52:59 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:34 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2: load balance to peer dhcp_opt3
Nov 9 06:53:32 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:32 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:38 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:38 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:36 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:36 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:44 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:46 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:44 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:53:46 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:54:00 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:54:03 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:54:00 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
Nov 9 06:54:03 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2

This computer (Anne) requests IP and get offers from the 2 DHCP servers, with different IP. I'm just wondering how this work ;)
Notice that some devices are receiving same IP from each DHCP server.

2 - About FW rules:
I don't have any single gateway defined for WAN, WAN1, WAN2 or floating rules. everything uses "default".
Also, I just checked twice: both WAN1 & WAN2 have their own interface defined (modem to which interface is attached)
Well… everything looks clean thus I read your answer again and again... tada ;D ;D

Problem was with FW rules for incoming flow on the WAN "group" interface.
Having removed these rules and replaced with rules on each WAN1 & WAN2 interface fixed this issue with incoming flow.
Thanks a lot.

I'll focus now on DHCP ;)

jimp · Nov 9, 2017, 9:27 PM

@chris4916:

This computer (Anne) requests IP and get offers from the 2 DHCP servers, with different IP. I'm just wondering how this work ;)
Notice that some devices are receiving same IP from each DHCP server.

That is normal. They will both offer since they are both active, but whichever lease the client accepts will be shared between the two systems.

@chris4916:

Problem was with FW rules for incoming flow on the WAN "group" interface.
Having removed these rules and replaced with rules on each WAN1 & WAN2 interface fixed this issue with incoming flow.

Great!