2 Separate pfSense sharing same WAN subnet
-
I have 2 separate pfsense boxes that are connected to the same ISP circuit. The circuit has 5 static IPs so each pfsense has it's own static public IP.
pfsense1
WAN = 1.2.3.90/29
LAN = 192.168.1.0/24pfsense2
WAN = 1.2.3.92/29
LAN = 192.168.10.0/24As you can see, they are on the same WAN subnet but have different LAN subnets.
The problem is that from pfsense1 I am unable to reach a NAT redirected server that is behind pfsense2. For example, say the LAN behind pfsense2 is hosting an exchange server with a public NAT redirect for ports 443 and 80. From anywhere else in the world that server is accessible EXCEPT from behind pfsense1. And vice versa. Any public services on pfsense1 are accessible across the globe EXCEPT from behind pfsense2.
Is this a route issue on the cable modem or is this a route issue in pfsense?
-
Well sniff on pfsense2 wan then you try and access its wan IP from pfsense1 - do you see that traffic get there?
-
@johnpoz I forgot to mention that when I ping pfsense1 from the LAN side of pfsense2 I do get a reply. This works vice versa as well.
-
This is a packet capture from pfsense1 while it is trying to reach pfsense2 in a browser https://1.2.3.92:10000 (10000 the port I use for remote management).
11:53:47.206735 IP 1.2.3.92.10000 > 1.2.3.90.9922: tcp 0
11:53:47.206825 IP 1.2.3.90.46824 > 1.2.3.92.10000: tcp 0
11:53:47.206949 IP 1.2.3.90.9922 > 1.2.3.92.10000: tcp 0
11:53:47.427922 IP 1.2.3.90.9922 > 1.2.3.92.10000: tcp 517
11:53:47.427954 IP 1.2.3.90.46824 > 1.2.3.92.10000: tcp 517
11:53:48.706584 IP 1.2.3.90.46824 > 1.2.3.92.10000: tcp 517
11:53:48.709737 IP 1.2.3.90.9922 > 1.2.3.92.10000: tcp 517
11:53:49.412252 IP 1.2.3.92.10000 > 1.2.3.90.9922: tcp 0
11:53:49.412294 IP 1.2.3.92.10000 > 1.2.3.90.46824: tcp 0
11:53:49.412411 IP 1.2.3.90.46824 > 1.2.3.92.10000: tcp 0
11:53:49.412414 IP 1.2.3.90.9922 > 1.2.3.92.10000: tcp 0
11:53:51.270594 IP 1.2.3.90.9922 > 1.2.3.92.10000: tcp 517
11:53:51.272548 IP 1.2.3.90.46824 > 1.2.3.92.10000: tcp 517
11:53:53.623438 IP 1.2.3.92.10000 > 1.2.3.90.46824: tcp 0
11:53:53.623480 IP 1.2.3.92.10000 > 1.2.3.90.9922: tcp 0
11:53:53.623748 IP 1.2.3.90.46824 > 1.2.3.92.10000: tcp 0
11:53:53.623751 IP 1.2.3.90.9922 > 1.2.3.92.10000: tcp 0
11:53:56.388068 IP 1.2.3.90.46824 > 1.2.3.92.10000: tcp 517
11:53:56.390001 IP 1.2.3.90.9922 > 1.2.3.92.10000: tcp 517Here is the packet capture from the pfsense2 side during the same process
11:54:53.485205 IP 1.2.3.90.18387 > 1.2.3.92.10000: tcp 0
11:54:53.485249 IP 1.2.3.92.10000 > 1.2.3.90.18387: tcp 0
11:54:53.486255 IP 1.2.3.90.45498 > 1.2.3.92.10000: tcp 0
11:54:53.486289 IP 1.2.3.92.10000 > 1.2.3.90.45498: tcp 0
11:54:54.487870 IP 1.2.3.92.10000 > 1.2.3.90.45498: tcp 0
11:54:54.487879 IP 1.2.3.92.10000 > 1.2.3.90.18387: tcp 0
11:54:56.704054 IP 1.2.3.92.10000 > 1.2.3.90.18387: tcp 0
11:54:56.704062 IP 1.2.3.92.10000 > 1.2.3.90.45498: tcp 0
11:55:00.951058 IP 1.2.3.92.10000 > 1.2.3.90.45498: tcp 0
11:55:00.951066 IP 1.2.3.92.10000 > 1.2.3.90.18387: tcp 0(obviously I've edited these logs with the fake IPs!)
When I run the process the other way, trying to access the web interface of pfsense1 on port 10000 from behind pfsense2, the results are the same just reversed.
-
well that doesn't make any sense then. Pfsense doesn't care if the source IP is on its wan network or some other public IP, etc.
The process would be exactly the same. So your saying it works from IP 1.2.3.4 but not 4.5.6.7 which if you have connectivity pfsense wouldn't care. What the source IP is.
Your sure its not a resolution issue?
These are different boxes - and not 2 vms say on the same host using the same nic for their wan?
-
@johnpoz these are 2 separate physical boxes. The only connection between them is that they are connected to the same cable modem.
I'm not sure what you mean by this: " So your saying it works from IP 1.2.3.4 but not 4.5.6.7" ?
I can access the web interface and other NAT'd services that are available on either pfsense1 or pfsense2 if I am coming to them from ANY other network just not when I hit pfsense1 from pfsense2 or 2 from 1.
When you say resolution, are you referring to DNS? I'm using the IPs when I try to make these connections so DNS shouldn't be an issue.
-
@itsystemsllc said in 2 Separate pfSense sharing same WAN subnet:
they are connected to the same cable modem.
Cable "modems" don't have multiple lan ports that I am aware of.. What is the make and model of this device? Some sort of gateway device..
is it doing some sort of odd passthru to give pfsense their public wan IPs?
-
@johnpoz cable modems for business class connections with more than 1 static IP have a 4 port switch in them.
-
And those are not "modems" those are gateways.. I have a few of those in a few different offices..
And they can do all kinds of odd stuff where the public IP is passed through it via a mac, etc. And some ports can be set to get a public IP, and others in the switch ports can be natted, etc.
-
@johnpoz - A "business gateway" is also a modem. It has a coax connection and 4 port switch. Modem means modulate/demodulate which is the process of converting from coax to ethernet and vice versa. So, yes, it is a modem LOL
I am not using any of the router or firewall functions that are sometimes included in a modem/gateway. Generally speaking these devices will automatically bridge the traffic when a static public IP address is assigned to customer equipment. Thus rendering any router or firewall on the ISP device moot. I don't have issues with access to private NAT'd services from the Internet into the aforementioned networks so I don't believe that the modem is interfering with any of that because it is bridging the connection.
-
Didn't say it didn't have a "modem" in it.. But that is a horrible term to use for something that is more than a modem.. Its not a "modem" its a gateway..
Sniff on pfsense2 wan when you try to access one of your forwarded ports from pfsense1 - do you see the traffic? If so then it would work. Just like any other port forward.
