Port forwarding with CARP and gateway group



  • Hi,

    I'm struggling to get port forwarding working after setting up HA with CARP, it seems that the packets are not returning threw the firewall.

    My setup
    Interfaces:

    1. WAN, dummy IP so I can setup CARP, my main connection is via PPPoE on this physical port.
      main 10.99.99.2/24 & backup 10.99.99.3/24
      CARP set with my with public IP 109.x.x.x/32

    2. WANPPP (pppoe for my main link)
      PPP added manually
      pppoe0 with interface set to my public IP 109.x.x.x
      gw (189.x.x.x) - gateway group tier 1

    3. LTE (link to modem gateway)
      192.168.5.2/24 & 192.168.5.3/24 (so I can access the modem interface)
      CARP for public IP 31.x.x.x/32
      gw 31.x.x.x - gateway group tier 2

    4. LAN - CARP 192.168.1.1

    Default gateway for ipv4 set to GW_grp

    The dummy IP on WAN is required so only one PPPoE link is established.

    Outbound NAT set to manual .

    Routing, internet and fail-over are all working, I also have a S-2-S & access OpenVPN server setup and working.

    Opening ports to services on pfSense is working but the issue is with port forwarding.

    I setup NAT rules per wan interface with firewall rules. I tried with and without the gateway set on the rule.
    I tested with destination on the NAT set to any, WANPPP address, and my public IP.

    0_1549804898620_51dec07c-0ef5-4117-bfa3-fec5cd7de58c-image.png

    WANPPP rules:
    0_1549804955902_769de42a-88e5-4e22-8228-56bcd42dd3de-image.png

    The reply-to option is enabled globally and on the rule (disable is not selected).

    In the firewall log I see the traffic is passing, but when doing a packet capture, on the LAN interface I see the request and response, but on the WANPPP interface I only see the requests but not response.

    I also checked the states table:
    0_1549805714651_1703d842-d685-448d-a157-a324834944bb-image.png .

    Before setting up CARP for the PPPoE interface port forwarding was working.

    What else can be preventing the responses from passing the firewall?