OpenVPN site to site - Can't reach client LAN
I've setup quite a few site to site setups with pfsense and this one i'm banging my head against the wall because it's partially working. The difference with this one is the pfsense on the client side is not the default gateway for the client computers on that side. It seemed like this is possible, but now i'm not sure. It's setup like this because it's a datacenter that we are given servers with two interfaces, one with a public ip and one with a private ip. The public IP on the servers have the default gateway and the private is just so they can communicate locally.
I put a pfsense box there to create a VPN tie between our office and the datacenter. We'll only have a few servers here so I plan on adding the static routes on the servers themselves to reach the office lan.
Outbound nat has been setup on the datacenter pfsense side on the LAN interface with any traffic going to the LAN network to be the LAN IP address.
With this setup, I can ping between both pfsense boxes. The office LAN can ping servers on the datacenter's LAN just fine. The problem is the datacenter's servers can't ping anything on the office lan. I've done packet captures on the datacenter's pfsense LAN interface and when I ping from a datacenter server to the office lan, the packet makes it to the office lan and I can see the reply on the capture, but from there doesn't make it back to the datacenter server. I'm assuming I need another outbound NAT rule somewhere, but I can't figure it out. Anyone else have a setup like this that has it working?
I've done packet captures on the datacenter's pfsense LAN interface and when I ping from a datacenter server to the office lan, the packet makes it to the office lan and I can see the reply on the capture, but from there doesn't make it back to the datacenter server.
If you see the echo request come from the data center server and the echo reply in the capture on the datacenter LAN, then the reply was sent out that interface. You will need to look outward from there.
The next device it should reach should be the datacenter server that is pinging. One other note I failed to mention is when I ping from the Office LAN and do a packet capture on the datacenter pfsense LAN interface, the source address is the datacenter pfense LAN ip address because of the outbound NAT. But on the packet capture when I ping from a datacenter server the source address on the reply is the VPN tunnel endpoint, so it's like the outbound NAT isn't happening here and maybe why the reply never reaches the server.
I'll need a picture. I'm slow. See below for an example of the type of information required.
Here's a rough diagram of how it looks right now.
OK thanks. So what are you pinging (source/dest), where are you capturing, and what are you seeing in the capture?
Scenario 1. (Successful)
Packet Capture on LAN interface on Datacenter PFsense (Outbound NAT makes the source the LAN IP)
20:32:12.480265 IP 172.20.5.180 > 172.20.5.182: ICMP echo request, id 50637, seq 2, length 64
20:32:12.480549 IP 172.20.5.182 > 172.20.5.180: ICMP echo reply, id 50637, seq 2, length 64
Scenario 2. (Fails)
Packet Capture on LAN interface on Datacenter PFsense
20:35:36.055371 IP 172.20.5.182 > 10.0.0.5: ICMP echo request, id 1, seq 489, length 40
20:35:36.086793 IP 10.0.0.5 > 172.20.5.182: ICMP echo reply, id 1, seq 489, length 40
Running wireshark on 172.20.5.182, the reply never comes.
In the second capture are the MAC addresses associated with 172.20.5.182 different on the request and reply?
Is there a gateway set on the LAN interface in the datacenter? Or is it DHCP?
There is outbound NAT on the LAN interface there. The datacenter pfSense thinks LAN is a WAN (an interface with a gateway set or DHCP).
The source and destination mac addresses are the same, just flipped on the reply.
There's no gateway on the LAN interface. The server is a vm that is given a public and private interface. The LAN interface is set to the private interface's IP address.
I added the nat outbound rule on the LAN interface for Scenario 1 to work. Without the rule, the source comes from the VPN endpoint 10.60.40.2 and it fails like scenario 2.
Something in your datacenter, man. If the frames are going out to the correct MAC address and and the packet is addressed to the correct IP address and you don't see the traffic arrive on that interface…
Yeah, I hope I was going to figure it out without looking more at the datacenter. The only thing I can notice that is different between the scenarios is in the first one, you can see the NAT working by translating the src of the Office LAN to the Datacenter PFsense LAN IP. But on scenario 2, you don't see the NAT working and the src IP is the office lan IP. If I remove the NAT for scenario 1, it looks just like scenario 2 and doesn't work so my thought was maybe it's a nat issue.
Thanks for you help.
Maybe they have something like AWS's source/dest check or some other ACLs in play. idk. But pcaps don't lie in general.
You were right. After dealing with the datacenter's support, we found I could enable IP spoofing on the LAN interface for the pfSense VM and after allowing that, it works fine without NAT.