CARP with multi-wan [SOLVED]



  • I'm trying to configure CARP with multi-wan and can't figure out what I'm making wrong but, basically, it doesn't work as expected.


    Modem                            Modem 1                              Modem 2           
    mode                              PPPoE                                  PPPoE           
    public IP                      xxx.xxx.xxx.xxx                  yyy.yyy.yyy.yyy           
    IP (internal side)        192.168.253.14/29          192.168.253.6/29           
    forward to                  192.168.253.9/29            192.168.253.1/29


    pfSense                            igb0                                  igb1                                  igb2                          igb3                            igb5
    interface                            WAN1                                WAN2                              LAN                            DMZ                            SYNC
    WAN master                  192.168.253.10/29          192.168.253.2/29        192.168.0.252/24        192.168.200.252        192.168.253.17/29
    WAN slave                    192.168.253.11/29          192.168.253.3/29        192.168.0.253/24        192.168.200.253        192.168.253.18/29
    VIP                                192.168.253.9/29            192.168.253.1/29      192.168.0.254/24        192.168.200.254/24            none
    WHID                                  2                                          1                                      3                                4                              none
    dft GW                          192.168.253.14                192.168.253.6                      none                          none                            none


    WAN1 & WAN2 are both member of "WAN" group, both tier-1, none is designed as "default gateway"
    WAN is configured as gateway in all rules at LAN interface level when flow is reaching internet. This works fine.

    CARP itself is almost OK (some problems with DHCP because some clients receive DHCP offer from each DHCP server with different IP) but the real problem I'm facing is with in-coming flow.

    If I try to reach internal mail server from internet using modem 1 public IP, this works.
    When I try to reach same server using modem 2 public IP, I can see packets reaching WAN2, even going out from LAN to mail server but it looks like packets are back using route through VIP with WHID 2 instead of WHID 1.

    I did read this https://doc.pfsense.org/index.php/Bypassing_Policy_Routing and applied it without any success. (and I don't understand how this applies to non-local IPs)
    I also have multiple IPsec tunnels, for the time being hardcoded to connect to either WAN1 or WAN2, depending on remote network and this works fine (yes I'm facing the issue with IPSec status preventing to display is correctly).

    I, of course, changed outbound NAT rules, in manual mode, to target WAN VIPs

    Well, I'm quite confused.
    I feel this is related to routing, perhaps bypass policy but I've to admit that, for the time being, I'm lost with poor understanding (and not real capability to closely investigate because, to make it worst, I'm configuring everything remotely through OpenVPN access  ;D ;D ;D :o

    Any idea?


  • Rebel Alliance Developer Netgate

    DHCP is probably giving you trouble because it sounds like you didn't configure the failover addresses for each DHCP interface. DHCP should be coordinating leases between the two units provided that failover is setup correctly in the DHCP settings (also, make sure to set a CARP VIP as the gateway and DNS server in the DHCP settings)

    As for the reply traffic, it sounds like maybe your routing or gateways are not correct. Check that:

    1. On WAN and WAN2 you have the gateway for the interface selected on Interfaces > WAN and Interfaces > WAN2
    2. On your firewall rules, make sure you DO NOT use interface groups/rules for inbound WAN traffic. Traffic must use rules on the WAN interface tabs or else it cannot be properly tagged with reply-to so the firewall can remember the proper return path for traffic
    3. Make sure that on your WAN interface rules, you DO NOT set a gateway on any rules on WAN tabs, this will definitely break things.

    Somewhat related but not to that, make sure you do have at least one gateway marked default under System > Routing, and you do not need to manually set a gateway on every LAN rule if it's taking the default path. Only set a gateway group or alternate gateway if you need it to take a non-default path.



  • Thank you Jimp for the anwser  ;)

    1 - Regarding DHCP, I configured failover peer IP address, setting here physical IP address of slave pfSense. BTW, on slave pfSense, DHCP has been automatically configured with master physical IP as failover peer IP. I also defined, in DHCP server, CARP VIP, on the LAN side, as defaut gateway and DNS. This works. I mean that CARP work well. If I stop master pfSense, everything switches smoothly to slave server

    And DHCP works, but not always. Here is what I collect in my rsyslog server:


    Nov  9 06:51:54 192.168.0.253 dhcpd: DHCPREQUEST for 192.168.0.3 from 7e:d8:99:8f:32:a4 via igb2
    Nov  9 06:51:54 192.168.0.253 dhcpd: DHCPACK on 192.168.0.3 to 7e:d8:99:8f:32:a4 via igb2
    Nov  9 06:51:54 192.168.0.253 dhcpd: Unable to add forward map from webserver.rmo.ci to 192.168.0.3: REFUSED
    Nov  9 06:52:30 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2: load balance to peer dhcp_opt3
    Nov  9 06:52:28 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:29 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:34 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2: load balance to peer dhcp_opt3
    Nov  9 06:52:32 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:32 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:35 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:38 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:35 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:39 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:43 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:46 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:43 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:46 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:02 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:59 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:02 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:52:59 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:34 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2: load balance to peer dhcp_opt3
    Nov  9 06:53:32 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:32 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:38 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:38 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:36 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:36 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:44 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:46 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:44 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:53:46 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:54:00 192.168.0.252 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:54:03 192.168.0.253 dhcpd: DHCPDISCOVER from 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:54:00 192.168.0.252 dhcpd: DHCPOFFER on 192.168.0.146 to 12:f0:2f:3e:d1:8c (Anne) via igb2
    Nov  9 06:54:03 192.168.0.253 dhcpd: DHCPOFFER on 192.168.0.165 to 12:f0:2f:3e:d1:8c (Anne) via igb2


    This computer (Anne) requests IP and get offers from the 2 DHCP servers, with different IP. I'm just wondering how this work  ;)
    Notice that some devices are receiving same IP from each DHCP server.

    2 - About FW rules:
    I don't have any single gateway defined for WAN, WAN1, WAN2 or floating rules. everything uses "default".
    Also, I just checked twice: both WAN1 & WAN2 have their own interface defined (modem to which interface is attached)
    Well… everything looks clean thus I read your answer again and again... tada  ;D ;D

    Problem was with FW rules for incoming flow on the WAN "group" interface.
    Having removed these rules and replaced with rules on each WAN1 & WAN2 interface fixed this issue with incoming flow.
    Thanks a lot.

    I'll focus now on DHCP  ;)


  • Rebel Alliance Developer Netgate

    @chris4916:

    This computer (Anne) requests IP and get offers from the 2 DHCP servers, with different IP. I'm just wondering how this work  ;)
    Notice that some devices are receiving same IP from each DHCP server.

    That is normal. They will both offer since they are both active, but whichever lease the client accepts will be shared between the two systems.

    @chris4916:

    Problem was with FW rules for incoming flow on the WAN "group" interface.
    Having removed these rules and replaced with rules on each WAN1 & WAN2 interface fixed this issue with incoming flow.

    Great!


Log in to reply