Port forwarding with CARP and gateway group

  • Hi,

    I'm struggling to get port forwarding working after setting up HA with CARP, it seems that the packets are not returning threw the firewall.

    My setup

    1. WAN, dummy IP so I can setup CARP, my main connection is via PPPoE on this physical port.
      main & backup
      CARP set with my with public IP 109.x.x.x/32

    2. WANPPP (pppoe for my main link)
      PPP added manually
      pppoe0 with interface set to my public IP 109.x.x.x
      gw (189.x.x.x) - gateway group tier 1

    3. LTE (link to modem gateway) & (so I can access the modem interface)
      CARP for public IP 31.x.x.x/32
      gw 31.x.x.x - gateway group tier 2

    4. LAN - CARP

    Default gateway for ipv4 set to GW_grp

    The dummy IP on WAN is required so only one PPPoE link is established.

    Outbound NAT set to manual .

    Routing, internet and fail-over are all working, I also have a S-2-S & access OpenVPN server setup and working.

    Opening ports to services on pfSense is working but the issue is with port forwarding.

    I setup NAT rules per wan interface with firewall rules. I tried with and without the gateway set on the rule.
    I tested with destination on the NAT set to any, WANPPP address, and my public IP.


    WANPPP rules:

    The reply-to option is enabled globally and on the rule (disable is not selected).

    In the firewall log I see the traffic is passing, but when doing a packet capture, on the LAN interface I see the request and response, but on the WANPPP interface I only see the requests but not response.

    I also checked the states table:
    0_1549805714651_1703d842-d685-448d-a157-a324834944bb-image.png .

    Before setting up CARP for the PPPoE interface port forwarding was working.

    What else can be preventing the responses from passing the firewall?