Problem using CARP and accessing server in same subnet at WAN-side

vmswtje · Oct 27, 2018, 11:30 AM

I'm experiencing some problems. Even after many tries, I can't find the solution and I'm not sure what is happening, so it's even possible that I do something stupid.

I've got 2 PFSense servers in CARP, they use 1 floating IP inside and 1 floating IP outside
Connections from "the internet" to internal servers (using NAT on the floating external IP) work well
Connections from the internal servers to "the internet" work well

When I try to connect from an internal server to an server that's on "the internet" but in the same subnet as the PFSense server, there's something wrong. I can ping, but when trying something like http/ssh I get connection timeouts at the client (I can't connect besides the ping)

On the receiving server (the external one, on same subnet as PFSense), I can see the incoming connection with netstat as "SYN_RECV"
On the PFSense I see states like:
LAN tcp 192.168.1.67:33136 -> 1.2.3.5:80 (SYN_SENT:ESTABLISHED)
WAN tcp 1.2.3.4:27970 (192.168.1.67:33136) -> 1.2.3.5:80 (ESTABLISHED:SYN_SENT)
(in this example: 192.168.1.67 = internal server, 1.2.3.5 = external server (fake), 1.2.3.4 = floating pfsense ip (fake))

Something else I see (I don't know whether it's relevant): tracepath <external-server> shows the IP of the master-PFSense, not the floating IP. the "route" command shows the floating IP.

Some of the things I tried:

On the sending server (the internal one) I've tried both using the floating IP-gateway as the master-gateway-IP
Trying some things with outbound NAT rules (without any luck)
Reading some things about NAT reflection (but I don't know whether this applies), it's disabled on PFSense right now, however I think I understand that I don't need this in this situation (?)

viragomann · Oct 27, 2018, 1:21 PM

Is the subnet mask correct on the WAN interface and also on the server?

Is a firewall running on the server which may block the access?

vmswtje · Oct 27, 2018, 2:04 PM

Hi @viragomann

subnetmask WAN-interface seems allright: (x.x.x.199/26, other server is x.x.x.226)
firewall doesn't seem to be the problem (I even tried without & full logging)

HTTP requests to the (external) server from PFSense cli works allright. So I guess it has to be a NAT or routing issue from the internal machine that goes through the PFSense. When initiated by PFSense through cli, there's nothing wrong.

viragomann · Oct 27, 2018, 2:20 PM

Do use NAT? If yes, what do your outbound NAT rules look like?

vmswtje · Oct 27, 2018, 2:27 PM

Thank you for your quick response, @viragomann

I do use outbound NAT:

"Manual Outbound NAT rule generation. (AON - Advanced Outbound NAT)"
2 lines for each internal subnet (i.e. 192.168.0.0/24 and 192.168.30.0)

See attachment for the 'general' line that's created for each subnet. There's also one for each subnet that concerns destination port 500 based on a static port ("Auto created rule for ISAKMP - LAN to WAN"), but I guess thats irrelevant

viragomann · Oct 27, 2018, 3:22 PM

It seems to be a kind of an asymmetric routing issue, so that responses from the destination server do not be directed back to the external CARP-IP.
However, since the outbound NAT rules use that address for translation, I don't know why.

You may use the packet capture tool of pfSense to check that.
Also a tcpdump on the external server could help to clarify where the packets go to.

vmswtje · Oct 28, 2018, 3:45 PM

See attachment.

Some things that aren't clear to me:

tracepath shows the primary CARP member, even though the default gateway on this server is the floating ip. This also happens with the "VRRPv2 Advertisement" every few seconds. Is this a problem or normal?
tracepath doesn't finish (ends in repeating 'no reply') / pings do get response however
It seems there's no answer from the external server (?) I can't explain this.
In case I SSH directly from the pfsense to the external server, there's no problem and I can connect using SSH. See attachment for the tcpdump on the external server in case I connect directly. The main difference I see is the external ip of the primary CARP member instead of the floating ip.

viragomann · Oct 28, 2018, 4:19 PM

@vmswtje said in Problem using CARP and accessing server in same subnet at WAN-side:

tracepath shows the primary CARP member, even though the default gateway on this server is the floating ip. This also happens with the "VRRPv2 Advertisement" every few seconds. Is this a problem or normal?

Yes, that's normal.

The second tcpdump is taken on the external server?

@vmswtje said in Problem using CARP and accessing server in same subnet at WAN-side:

tracepath doesn't finish (ends in repeating 'no reply') / pings do get response however

So take a tcpdump from pings. Pings don't use a stateful protocol like SSH or HTTP.
However, that doesn't explain why the server doesn't respond to the SSH when the internal firewalls rules allow it.

Derelict · Oct 28, 2018, 6:08 PM

Why do you have 1.2.3.4 identified as floating ip external in one capture and 1.2.3.6 identified as primary CARP IP in the other. What is the difference? What is what?

I would check that MAC addresses on the ping traffic are sourcing from where they should be and not being ping-ponged around out there. I would double-check all netmasks of all the hosts/interfaces out on WAN.

Do this:

Diagnostics > Test Port

Select the destination server, TCP, and a port the destination server is listening on.

Do the test sourcing from the WAN interface address and the WAN CARP VIP. Does it work for both?