Multi-LAN issue, VPN server behind pfSense, TCP:SA drops


  • I've been trying to set up a Wireguard VPN server on Unraid or a Pi 4 behind pfSense. I'm testing it in my PRIVNET network and I'm trying to route all traffic through the tunnel (AllowedIPS 0.0.0.0/24) and have the clients be able to access hosts on my local network, including some on other subnets and VLANs. This has worked fine for me using pfSense and IPSEC for some time. Here is a diagram:

    wireguard_vpn_diagram.png

    I believe I have the firewall rules set up okay. When I connect the remote client I can access Server 1 at 192.168.30.10 just fine. When I try to access Server 2 at 192.168.60.10 I can't get the web page up. When I look at the logs I don't see the traffic to Server 2 for some reason, even though I think I'm logging everything, though I do see the replies from Server 2. They're being dropped, however, and they show TCP:SA. I imaging it's a asymmetric routing issue, but I can't figure it out. I've tried the "Bypass firewall rules for traffic on the same interface" setting but that hasn't helped.

    Any assistance is appreciated.


  • If the VPN server isn't the default gateway, you should take the server out from the LAN and put into a transit network between it and the default gateway.
    Then add a static route for the VPN network to the gateway pointing to the server.

    Another solution is to do masquerading on the VPN server on outgoing packets to the LAN. But that makes it impossible to distiguish the client on the destination device.


  • @viragomann Shoot. I was starting to believe this is what it would take. I've not had to use a transit network before. I simplified it for my diagram, but both my Unraid servers have 2-4 ethernet ports and connections on multiple networks and VLANs depending upon what the Docker containers or VMs are providing and who needs access. I'm not sure even where to begin with the transit network in this setup...

    I guess NAT/masquerading would make it easier, and that's why OpenVPN and IPSEC are working, I take it. This will be my first "always on VPN" though and I may need to block some services for some clients on occasion so that's my hesitation to use that.


  • @burntoc said in Multi-LAN issue, VPN server behind pfSense, TCP:SA drops:

    I'm not sure even where to begin with the transit network in this setup...

    Yes, you only need a separated network between pfSense and the VPN server like VLAN30 in the updated diagram and the static route on pfSense to direct response packets back to the VPN server.
    Should be fine.


  • @viragomann That physical interface, eth2, has multiple VLANs on it though. I believe I have one free ethernet port so I'll have to see if I can break it out.


  • @burntoc
    A VLAN is fine for that purpose.


  • I'm afraid I'm still lost. I think this is how the communications are going:

    1. VPN Client (10.20.30.2) wants to talk to Server 2 (192.168.60.1).
    2. It sends it to its default GW, which I take it is the Unraid Server 1 host interface (192.168.30.10)
    3. Unraid server 1 rewrites the source header and sends it to pfSense@192.168.30.1.
      pfSense.
    4. pfSense forwards the request to Server 2@192.168.60.10.

    I’d expect Server 2 to see a request from 192.168.30.10 and replies going to that request to complete setup of the session, which would forward back to Server 1 and then the VPN Client. Instead I see replies to the VPN Client address directly at 10.20.30.2, which pfSense drops because it’s gone some requests from Server 1 and some replies from Server 2 but the IP addresses don’t match and it doesn’t connect them. Is this the correct understanding?
    If so, I don’t understand why Server 2 isn’t replying to Server 1@192.168.30.10, which I take it would go through.
    I’m sorry I am so dense about this but even after your previous replies I still don’t understand how I can correct this. I don't want to put another host or router between Server 1 and pfSense if I can avoid it, and even if I did it I'd have to move VLAN60 over to another shared interface, I think. Any chance someone can mock up a rough diagram or steps of what I need to do to fix this?


  • @burntoc said in Multi-LAN issue, VPN server behind pfSense, TCP:SA drops:

    Unraid server 1 rewrites the source header and sends it to pfSense@192.168.30.1.

    No. That would be the masquerading (NAT) solution. For that you don't need to separate the VPN server, cause packets from VPN client going out the LAN get the servers interface IP as source. Since any device in the network has a route to that server, responses will go directly back to it without adding a static route.
    That has to be set up on the unraid. I can't help there. However, whith masquerading you get the drawback that you can't see the real clients source IP on the destination server. Also you cannot filter the traffic on client basis on pfSense, cause its not passing it.

    I'm not familiar with Wireguard VPN, but all over it should work this way:

    • Server 1 is in a separate subnet.
    • The VPN server pushes the default route to the clients.
    • If the client requests Server 2 (192.168.60.10), it sends the packets to Server 1
    • Server 1 passes the packets to pfSense, cause this the default gateway
    • pfSense forward the packet to Server 2 on the VLAN60 interface.
      You have a firewall rule on VLAN30 interface to allow the access from VPN clients to Server 2.
    • Server 2 sends the response packet to pfSense. It is destined to the VPN clients IP 10.20.30.2, but it sends it to pfSense, cause its the default gateway.
    • On pfSense you have added a static route for 10.20.30.0/24 pointing to Server 1 192.168.30.10. So it forwards the respond packet to Server 1.
    • Server 1 sends it to the VPN client.

    You can use Diagnostic > Packet Capture on pfSense on VLAN30 and VLAN60 to investigate what the packets do and if they have the correct source and destination IPs.


  • @viragomann Of course your right regarding the replies to 10.20.30.2. I had my public routing thinking cap on and thought this would have to be encapsulated b/c it is RFC 1918, but of course this is all staying in my private LAN and RFC 1918 address spaces. Afraid I've worked my mind into a pretzel and I need to step back.

    I had begun to suspect addressing it on the Unraid server was the answer and you've confirmed that is where I should look. Wireguard is so simplistic -that's a double-edged sword- that this is really hard to address, but I'll definitely look at your proposed approach. Man I just with pfSense had an official plugin for WG. I know it's being worked on, but I don't know I can wait that long...


  • @burntoc said in Multi-LAN issue, VPN server behind pfSense, TCP:SA drops:

    Wireguard is so simplistic -that's a double-edged sword-

    🤣

    I'm wondering why you don't realize that with OpenVPN or IPSec on pfSense, since it's the edge router.


  • That's exactly why I think having this on the pfSense edge router would make this 10x easier. The others have been fine for me but they consume far too much battery and/or have too many issues transitioning from outside to inside on my wifi for me to use them as always on across my entire family, as I intend to do here.


  • @viragomann So as best as I can tell I've gone through all of your steps and that's exactly how I have it set up, but of course it isn't working. When I try to access the web GUI on Server 2 and run a capture I can see 10.20.30.2 talking to all sorts of stuff outside my firewall (dang chatty phone apps) but nothing going to Server 2. Nothing there. I can see Server 2 trying to respond on it's interface, and the TCP:SA drops. FWIW, I can ping Server 2 via the Wireguard tunnel, and a traceroute shows 2 hops - the WG tunnel endtpoint address of 10.20.30.1 and then Server 2 192.168.60.10.
    If it is not some weirdness with pfSense allowing crosstalk between VLANs on that same physical interface without logging it then it must be something weird about the way Unraid and Wireguard are handling it....


  • The only reason for this behavior I can think of is that the request packets to server 2 do not pass pfSense.
    Either server 1 is somehow within the same subnet as server 2 or the VLAN is misconfigured either on a switch, on Unraid or on pfSense.


  • @viragomann I think you're right.

    So my diagram skills remain challenged, but I drew up some more of the connections because I believe the issue is that traffic from the client to the server is bypassing the firewall as the VPN server (Server 1) and Server 2 are plugged into the same switch. I'd set each interface with it's own VLAN and they get GW info for the pfsense interfaces (also dedicated per VLAN and subnet) specifically because I wanted to force the traffic to pfSense so the rules could act on it. It appears that's not happening.

    I wonder if I need more routes on the Unraid servers and if so, what they need.

    Here's my attempt to clarify the situation:

    https://1drv.ms/u/s!AlWtSxcIfbcPhtNp5pu1XXLuzctz0A?e=YT3q4V

    2523f2d2-b210-4f95-96a3-77a0941caf82-image.png

    One last thing. Even though I'm starting to get whatI think is a better picture of the situation, I also still don't understand for sure if Wireguard is effectively bridging wg0 to the br0 interface on eth0 or not. I understood it was, which is why I thought it would force the traffic out that VLAN30 interface to the firewall irrespective of its connection to the same switch, then the firewall would route it back to Server2, but it doesn't seem that is in fact what happens. I know you may not know Wireguard, but if it acts like other VPNs and adapters wouldn't this normally be the case?


  • So I finally had a chance to relax this weekend and take a step back. Seems obvious now my mistake is that I have the two servers on the same vlan and same switch so they're communicating directly - at least one-way they are.
    I believe my options area:

    1. Change the VLAN for one of them
    2. Change the subnets, such as putting one in the lower half of a /24 with a /25 and the other in the other half
    3. Get a swich that can do Private VLANs, port-isolation, or protected ports to prevent them from talking without sending the traffic up to the FW first
      I'm leaning toward 3, and maybe doing 1 in the interim while I wait for the switch to arrive.