Problem using CARP and accessing server in same subnet at WAN-side



  • I'm experiencing some problems. Even after many tries, I can't find the solution and I'm not sure what is happening, so it's even possible that I do something stupid.

    • I've got 2 PFSense servers in CARP, they use 1 floating IP inside and 1 floating IP outside
    • Connections from "the internet" to internal servers (using NAT on the floating external IP) work well
    • Connections from the internal servers to "the internet" work well

    When I try to connect from an internal server to an server that's on "the internet" but in the same subnet as the PFSense server, there's something wrong. I can ping, but when trying something like http/ssh I get connection timeouts at the client (I can't connect besides the ping)

    • On the receiving server (the external one, on same subnet as PFSense), I can see the incoming connection with netstat as "SYN_RECV"
    • On the PFSense I see states like:
      LAN tcp 192.168.1.67:33136 -> 1.2.3.5:80 (SYN_SENT:ESTABLISHED)
      WAN tcp 1.2.3.4:27970 (192.168.1.67:33136) -> 1.2.3.5:80 (ESTABLISHED:SYN_SENT)
      (in this example: 192.168.1.67 = internal server, 1.2.3.5 = external server (fake), 1.2.3.4 = floating pfsense ip (fake))

    Something else I see (I don't know whether it's relevant): tracepath <external-server> shows the IP of the master-PFSense, not the floating IP. the "route" command shows the floating IP.

    Some of the things I tried:

    • On the sending server (the internal one) I've tried both using the floating IP-gateway as the master-gateway-IP
    • Trying some things with outbound NAT rules (without any luck)
    • Reading some things about NAT reflection (but I don't know whether this applies), it's disabled on PFSense right now, however I think I understand that I don't need this in this situation (?)


  • Is the subnet mask correct on the WAN interface and also on the server?

    Is a firewall running on the server which may block the access?



  • Hi @viragomann

    • subnetmask WAN-interface seems allright: (x.x.x.199/26, other server is x.x.x.226)
    • firewall doesn't seem to be the problem (I even tried without & full logging)

    HTTP requests to the (external) server from PFSense cli works allright. So I guess it has to be a NAT or routing issue from the internal machine that goes through the PFSense. When initiated by PFSense through cli, there's nothing wrong.



  • Do use NAT? If yes, what do your outbound NAT rules look like?



  • Thank you for your quick response, @viragomann

    I do use outbound NAT:

    • "Manual Outbound NAT rule generation. (AON - Advanced Outbound NAT)"
    • 2 lines for each internal subnet (i.e. 192.168.0.0/24 and 192.168.30.0)

    See attachment for the 'general' line that's created for each subnet. There's also one for each subnet that concerns destination port 500 based on a static port ("Auto created rule for ISAKMP - LAN to WAN"), but I guess thats irrelevant

    0_1540650444419_screen-1.png



  • It seems to be a kind of an asymmetric routing issue, so that responses from the destination server do not be directed back to the external CARP-IP.
    However, since the outbound NAT rules use that address for translation, I don't know why.

    You may use the packet capture tool of pfSense to check that.
    Also a tcpdump on the external server could help to clarify where the packets go to.



  • See attachment.
    0_1540740465263_screen-2.png

    Some things that aren't clear to me:

    • tracepath shows the primary CARP member, even though the default gateway on this server is the floating ip. This also happens with the "VRRPv2 Advertisement" every few seconds. Is this a problem or normal?

    • tracepath doesn't finish (ends in repeating 'no reply') / pings do get response however

    • It seems there's no answer from the external server (?) I can't explain this.

    • In case I SSH directly from the pfsense to the external server, there's no problem and I can connect using SSH. See attachment for the tcpdump on the external server in case I connect directly. The main difference I see is the external ip of the primary CARP member instead of the floating ip.

    0_1540741254894_screen-3.png



  • @vmswtje said in Problem using CARP and accessing server in same subnet at WAN-side:

    tracepath shows the primary CARP member, even though the default gateway on this server is the floating ip. This also happens with the "VRRPv2 Advertisement" every few seconds. Is this a problem or normal?

    Yes, that's normal.

    The second tcpdump is taken on the external server?

    @vmswtje said in Problem using CARP and accessing server in same subnet at WAN-side:

    tracepath doesn't finish (ends in repeating 'no reply') / pings do get response however

    So take a tcpdump from pings. Pings don't use a stateful protocol like SSH or HTTP.
    However, that doesn't explain why the server doesn't respond to the SSH when the internal firewalls rules allow it.


  • Netgate

    Why do you have 1.2.3.4 identified as floating ip external in one capture and 1.2.3.6 identified as primary CARP IP in the other. What is the difference? What is what?

    I would check that MAC addresses on the ping traffic are sourcing from where they should be and not being ping-ponged around out there. I would double-check all netmasks of all the hosts/interfaces out on WAN.

    Do this:

    Diagnostics > Test Port

    Select the destination server, TCP, and a port the destination server is listening on.

    Do the test sourcing from the WAN interface address and the WAN CARP VIP. Does it work for both?