Site to Site routing with pfSense and remote Edgerouter not working



  • I set up an OpenVPN server at my house in pfSense and imported the VPN config to a remote Ubiquiti Edgerouter X. Tunnel is up but I'm having routing difficulty. Both pfSense and the ERX are the default gateway on their LANs. I want devices on either LAN to reach the opposite LAN, Site to Site. Worked with this guide https://doc.pfsense.org/index.php/OpenVPN_Site_To_Site

    From what I understand those last two things should set up the route/iroute OpenVPN stuff that's necessary. Unfortunately I have no idea where to see the raw OpenVPN server config file or CCD files to see if this is true. I've done this stuff with Windows as the OpenVPN server and the ERX before, just can't figure out the pfSense GUI. I haven't messed with entering any static routes myself, IMO OpenVPN and these two devices already being the default gateways should be enough?!?

    • 10.0.1.0/24 pfSense local LAN

    • 10.9.0.0/24 OpenVPN virtual network

    • 10.9.0.10 is ERX VPN IP, static assignment via client config

    • 10.0.3.0/24 Remote ERX LAN

    Symptoms

    • Nothing on 10.0.1.x including the pfSense ping utility can reach anything on 10.0.3.x

    • pfSense ping utility can reach 10.9.0.10, the remote ERX VPN IP

    • The remote ERX can ping anything it wants on 10.0.1.x entire LAN, and my phone using the same pfSense OpenVPN server can reach anything it wants 10.0.1.0/24

    • Other things on the 10.0.3.x LAN can not reach 10.0.1.x IPs

    • The remote ERX can ping 10.9.0.1, the OpenVPN server IP.

    • Other things on the 10.0.3.x LAN can not reach 10.9.0.1

    • In the ERX I can see an automatically created route to 10.0.1.0/24 with next hop 10.9.0.1

    pfSense OpenVPN server configuration

    • Remote access TUN / UDP

    • Tunnel network 10.9.0.0/24

    • Redirect gateway off

    • IPv4 local networks has 10.0.1.0/24 in it

    • Provide DNS server is available with 10.0.1.1 listed FWIW

    Client specific Overrides, I'm thinking of this as the OpenVPN CCD file, right?

    • has my openvpn server selected

    • has an entry of the connecting ERX common name

    • Tunnel: 10.9.0.10/24

    • IPv4 local networks: 10.0.1.0/24

    • IPv4 remote networks 10.0.3.0/24


  • LAYER 8 Netgate

    pfSense OpenVPN server configuration

    Remote access TUN / UDP
        Tunnel network 10.9.0.0/24
        Redirect gateway off
        IPv4 local networks has 10.0.1.0/24 in it
        Provide DNS server is available with 10.0.1.1 listed FWIW

    First off, 10.0.3.0/24 needs to be in Remote Networks there. Yes, there and the CCD.



  • @Derelict:

    First off, 10.0.3.0/24 needs to be in Remote Networks there. Yes, there and the CCD.

    I think that's key. Thing is there was no entry on that page for remote networks. Likely because I selected remote access? I know how it needs to be both in the server config and CCD in plain config files. I changed it to Site-To-Site shared key and that field appeared. Redid everything for that new configuration and now having much less success, endpoints can't even ping the opposite VPN IP let alone anything on the LAN.

    Time to back up and reassess this situation, I just spent a day getting something working and failing that I plainly know how to do, but already I'm thinking this would be easier if either I had pfSense on both ends so I could follow guides OR if I just skipped it in pfSense all together and ran OpenVPN on my ubuntu server with the same configuration we use at work in text files. I'm a little shocked at that last part, I am a big fan of GUI but in this one case I took the time to get things working in plain config files so I oddly prefer that now in this one case.

    Is there no way to just upload an OpenVPN server config and CCD file in pfSense?


  • LAYER 8 Netgate

    No. No upload.

    Yes, Remote Networks generally only makes sense in Site-to-Site vs Remote Access. There is really no difference in the server modes there other than the fields presented and the presumption that you are not talking to another router but a user workstation (OpenVPN Connect, Viscosity, etc.)

    This is not that difficult.

    Set a tunnel network.

    Set the Local Networks to push routes to the client.

    Set the Remote Networks to route traffic into OpenVPN.

    Set the Remote Networks in the Client-Specific override(s) to iroute the network to the correct client.

    Be sure OpenVPN firewall rules pass the INBOUND connections you require.

    Optionally assign an interface, Outbound NAT, etc.



  • Following up; I started over with following this guide; https://doc.pfsense.org/index.php/OpenVPN_Site_To_Site

    That guide is so simple and leaves out anything client specific overrides, no iroute CCD entires etc. Far as the firewall rules go there was already something in there from a wizard I tried before which still looked suitable.

    Loaded the client export to the ERX and it failed, logs show –pull is inappropriate for UDP. Without much hope I commented that out in the config file and retried, tunnel came up. But I couldn't ping back and forth. Then found this in the system logs for openvpn in pfSense:

    ERROR: FreeBSD route add command failed: external program exited with error status: 1

    Googling that it's a routing issue. So I checked over that guide again, couldn't find anything wrong and said heck with it, rebooted pfSense. When it came back up that error was gone, tunnel was up. Local LAN devices could reach remote LAN and vice versa.

    So yay?


  • LAYER 8 Netgate

    That's because iroutes are not required in a shared key network (that always uses a /30 for the tunnel no matter what you specify) or an SSL/TLS network with a /30 tunnel network specified.

    In both of those cases the tunnel is considered to be a site-to-site with only one possible remote endpoint so all traffic routed to openvpn is sent to the other side.

    In an SSL/TLS network with a tunnel network larger than /30, there can be multiple endpoints so iroutes are necessary for OpenVPN itself to make routing decisions.

    The route add command failed error means the route already existed in the routing table when you tried to start the server. Examining the routing table would have been a better troubleshooting option than rebooting. Whatever the case, when you rebooted that route did not exist in the routing table when the OpenVPN server process was started on reboot so it was able to add the necessary route.



  • I was troubleshooting my site-to-site setup and came across this thread.
    Can someone clarify on relations between the Tunnel network set in the server configuration and tunnel subnet set in the client's override?

    Here we see:
    Tunnel network 10.9.0.0/24 (server)
    and
    Tunnel: 10.9.0.10/24 (override)

    They are on the same subnet or in other words - override subnet is a subset of the tunnel network. Is this a strong requirement?

    According to the example found in pfSense how-to, we can use completely different networks there.
    In my case everything works only if I'm assigning override /30s from server's /24, i.e. like shown in this thread.
    With a separate network used in overrides (say, 10.9.8.0) I'm missing those /30s in pfSense routing table. On the client side everything looks good in this case.


  • LAYER 8 Netgate

    If you are trying to set a specific address in the tunnel network on a client using the CSOs I would not do it that way.

    I believe there is a way to configure the tunnel network to reserve a portion of the pool for static assignments but pfSense GUI does not have a knob for it.  Yeah there would need o be  way to specity the nopool flag to the server command then manually specify a pool in custom options.

    I would put another route in the Custom options of the server (or use a /25 netmask in the server and put the other half in custom options)

    Then you have to use ifconfig-push to set the IP address of the client.

    I am unsure if you need to place the tunnel network in the CSO or just the ifconfig-push.

    So in your case I would:

    Server:
    Tunnel Network: 10.9.0.0/25 (That will assign .1 to the server and a pool for .4-124 for clients I think)
    Custom options:
    route 10.9.0.128 255.255.255.128;

    CSO:
    Advanced option:
    ifconfig-push 10.9.0.131 255.255.255.128

    That seems to be working for me.



  • @Derelict:

    I would put another route in the Custom options of the server (or use a /25 netmask in the server and put the other half in custom options)

    Actually, this is what I had initially (almost) - it was 4x /26 splitted between 3 servers for different purposes, 2 of them were supposed to be used exactly as you suggest.

    Custom options:
    route 10.9.0.128 255.255.255.128;

    Added, that was the key.

    CSO:
    Advanced option:
    ifconfig-push 10.9.0.131 255.255.255.128

    That was not required - it's enough to place the tunnel network in the CSO. I put 10.9.0.132/30 there and corresponding ifconfig-push (ifconfig-push 10.9.0.134 10.9.0.133) was added to configuration file automatically.

    That seems to be working for me as well ;)
    Many thanks, Derelict!


  • LAYER 8 Netgate

    OK I have not looked at what that tunnel network puts in the actual CCD file. It is certainly better to use the gui fields if they do the right thing.

    I use subnet not net30.

    Glad it's working.



  • Tried "subnet":

    MULTI ERROR: primary virtual IP for username/X.X.X.X:40031 (10.9.0.134) violates tunnel network/netmask constraint (10.9.0.0/255.255.255.128)
    

    Any idea how to fix that? I'm concerned about the following statement in override config in GUI:

    With subnet topology, enter the client IP address and the subnet mask must match the IPv4 Tunnel Network on the server.

    Looks like openvpn itself has all the necessary configuration parameters but we're a bit limited with what is available through the GUI.


  • LAYER 8 Netgate

    Yeah looks like it is, again, trying to enforce what is defined in the tunnel network on to what is defined in the CCD.

    Using the extra config will probably bypass all that sanity checking.

    MULTI ERROR: primary virtual IP for username/X.X.X.X:40031 (10.9.0.134) violates tunnel network/netmask constraint (10.9.0.0/255.255.255.128)

    (10.9.0.134 does not lie in 10.9.0.0/255.255.255.128. It lies in 10.9.0.128/255.255.255.128)

    Is that a message from the pfSense GUI or OpenVPN?



  • that was an OpenVPN [server] message, System Logs / OpenVPN


  • LAYER 8 Netgate

    Then it must not be doing the same thing I did with the route in the server and the ifconfig-push in the CCD because I did that manually and it worked fine.



  • @Derelict:

    I did that manually and it worked fine.

    Could you please compare my configuration with yours? Here a the relevant part of the server config and override file:

    --- /var/etc/openvpn/server4.conf ---
    
    server 10.9.0.0 255.255.255.128
    ifconfig 10.9.0.1 10.9.0.2
    route 192.168.1.0 255.255.255.0
    topology subnet
    route 10.9.0.128 255.255.255.128
    
    --- /var/etc/openvpn-csc/server4/username ---
    push "route 10.0.1.0 255.255.255.0"
    iroute 192.168.1.0 255.255.255.0
    ifconfig-push 10.9.0.134 255.255.255.128
    
    

    With this configuration - client IP is not on "server" subnet- I get the error message I mentioned earlier.
    As I'm reading OpenVPN documentation I realize that server configuration page needs some modification in order to support pool configuration with topology subnet.


Log in to reply