(Solved) pfsense IPSEC behind another pfsense WAN
-
I have an issue which I don't understand. IPSEC doesn't seem to work properly behind another pfsense unit.
I've 2 sites: A and B. Site A is connected to the ISP through a Unifi USG (PPPoE). Behind it I have 2 pfsense VMs (pfsCARP) in CARP failover which serve LAN and other VLANs. So everything on the network is supposed to go LAN -> pfsenseCARP -> Unifi USG -> Internet. PfsCARP connects to site B (pfsenseB, SG-2220) via IPSEC. I've forwarded ports from the USG to pfsCARP VIP. Site B is very simple with a switch and a few devices behind.
Everything works fine in the above setup.
I've switched the USG on site A with a netgate SG-2440 (pfsenseA). I've forwarded IPSEC ports (4500, 500, ESP, AH) and web ports (443 and 80 for haproxy) from pfsenseA to pfsCARP VIP. PfsenseA is a fresh install and there's barely anything else configured on it.
Now I have several issues:
- IPSEC is connected, but I can only access "some" things from site B:
- access freenas via SSH
- access pfsenseB webUI
- ping devices in site B
- no access to freenas via webUI
- no access to proxmox via webUI
- on pfsenseA I see outgoing traffic from pfsCARP VIP directly to IP from site A VLAN. So it looks like there's no outbound NAT;
- pfsenseA is blocking packets on port 4500 even though there's port forward and an associated (allow) firewall rule;
- the port forward rules are showing 0 traffic for IPSEC ports, but there's traffic on http/https ports.
When I switch pfsenseA with the USG, IPSEC starts working properly again.
All pfsenses are on 2.4.4-p3
Edit: I can see traffic coming out on site B. So the IPSEC tunnel works.
- IPSEC is connected, but I can only access "some" things from site B:
-
It seems the issue goes both ways. I can access pfsense web ui from either side, but I can't access anything else on the P2 subnets.
-
So, I've installed OPNsense and it seems to work properly there with a basic NAT port forward rule. I'm gonna do a couple more tests with different hardware on pfsense, but I suspect there're some issues with port forwarding rules on pfsense regardin IPSEC.
Edit: I've reset pfsenseA to factory defaults and set it up again. It was basically the same as the previous install but something went different this time. Now it seems to work properly.