Problem using CARP and accessing server in same subnet at WAN-side
-
I'm experiencing some problems. Even after many tries, I can't find the solution and I'm not sure what is happening, so it's even possible that I do something stupid.
- I've got 2 PFSense servers in CARP, they use 1 floating IP inside and 1 floating IP outside
- Connections from "the internet" to internal servers (using NAT on the floating external IP) work well
- Connections from the internal servers to "the internet" work well
When I try to connect from an internal server to an server that's on "the internet" but in the same subnet as the PFSense server, there's something wrong. I can ping, but when trying something like http/ssh I get connection timeouts at the client (I can't connect besides the ping)
- On the receiving server (the external one, on same subnet as PFSense), I can see the incoming connection with netstat as "SYN_RECV"
- On the PFSense I see states like:
LAN tcp 192.168.1.67:33136 -> 1.2.3.5:80 (SYN_SENT:ESTABLISHED)
WAN tcp 1.2.3.4:27970 (192.168.1.67:33136) -> 1.2.3.5:80 (ESTABLISHED:SYN_SENT)
(in this example: 192.168.1.67 = internal server, 1.2.3.5 = external server (fake), 1.2.3.4 = floating pfsense ip (fake))
Something else I see (I don't know whether it's relevant): tracepath <external-server> shows the IP of the master-PFSense, not the floating IP. the "route" command shows the floating IP.
Some of the things I tried:
- On the sending server (the internal one) I've tried both using the floating IP-gateway as the master-gateway-IP
- Trying some things with outbound NAT rules (without any luck)
- Reading some things about NAT reflection (but I don't know whether this applies), it's disabled on PFSense right now, however I think I understand that I don't need this in this situation (?)
-
Is the subnet mask correct on the WAN interface and also on the server?
Is a firewall running on the server which may block the access?
-
Hi @viragomann
- subnetmask WAN-interface seems allright: (x.x.x.199/26, other server is x.x.x.226)
- firewall doesn't seem to be the problem (I even tried without & full logging)
HTTP requests to the (external) server from PFSense cli works allright. So I guess it has to be a NAT or routing issue from the internal machine that goes through the PFSense. When initiated by PFSense through cli, there's nothing wrong.
-
Do use NAT? If yes, what do your outbound NAT rules look like?
-
Thank you for your quick response, @viragomann
I do use outbound NAT:
- "Manual Outbound NAT rule generation. (AON - Advanced Outbound NAT)"
- 2 lines for each internal subnet (i.e. 192.168.0.0/24 and 192.168.30.0)
See attachment for the 'general' line that's created for each subnet. There's also one for each subnet that concerns destination port 500 based on a static port ("Auto created rule for ISAKMP - LAN to WAN"), but I guess thats irrelevant
-
It seems to be a kind of an asymmetric routing issue, so that responses from the destination server do not be directed back to the external CARP-IP.
However, since the outbound NAT rules use that address for translation, I don't know why.You may use the packet capture tool of pfSense to check that.
Also a tcpdump on the external server could help to clarify where the packets go to. -
See attachment.
Some things that aren't clear to me:
-
tracepath shows the primary CARP member, even though the default gateway on this server is the floating ip. This also happens with the "VRRPv2 Advertisement" every few seconds. Is this a problem or normal?
-
tracepath doesn't finish (ends in repeating 'no reply') / pings do get response however
-
It seems there's no answer from the external server (?) I can't explain this.
-
In case I SSH directly from the pfsense to the external server, there's no problem and I can connect using SSH. See attachment for the tcpdump on the external server in case I connect directly. The main difference I see is the external ip of the primary CARP member instead of the floating ip.
-
-
@vmswtje said in Problem using CARP and accessing server in same subnet at WAN-side:
tracepath shows the primary CARP member, even though the default gateway on this server is the floating ip. This also happens with the "VRRPv2 Advertisement" every few seconds. Is this a problem or normal?
Yes, that's normal.
The second tcpdump is taken on the external server?
@vmswtje said in Problem using CARP and accessing server in same subnet at WAN-side:
tracepath doesn't finish (ends in repeating 'no reply') / pings do get response however
So take a tcpdump from pings. Pings don't use a stateful protocol like SSH or HTTP.
However, that doesn't explain why the server doesn't respond to the SSH when the internal firewalls rules allow it. -
Why do you have 1.2.3.4 identified as floating ip external in one capture and 1.2.3.6 identified as primary CARP IP in the other. What is the difference? What is what?
I would check that MAC addresses on the ping traffic are sourcing from where they should be and not being ping-ponged around out there. I would double-check all netmasks of all the hosts/interfaces out on WAN.
Do this:
Diagnostics > Test Port
Select the destination server, TCP, and a port the destination server is listening on.
Do the test sourcing from the WAN interface address and the WAN CARP VIP. Does it work for both?
