[Solved]: Failing to create routes on boot. Must "pretend edit" a route



  • I have a pfsense tap vpn server on 10.0.0.1/24.

    I have several subnets connected to it via pfsense routers, i.e. 10.5.0.1/24, 10.10.0.1/24, 10.20.0.1/24, etc.

    The tap interface on the servers assigns them address at 10.0.0.100, .101, .102, respectively.

    I then add routes to those subnets on a case by case basis by creating a gateway on the interface, pointing it to the tap IP address (e.g. 10.0.0.102), and then adding the 10.20.0.0/24 subnet as a static route via that gateway.

    One day I noticed I could not reach the other client subnets or vice versa from 10.5.0.1.  I then noticed the reason was that the routes were not created.

    Eventually, I decided to pretend to edit one of the gateways, changed nothing, saved, and applied.  Then all the routes were created, and everything was working again.

    (I'm not sure if this was related, but I ran into a bug where suhosin.so was failing to load, and I worked around it by adding "extension=suhosin" to /usr/etc/php/extensions.ini.  It was only after that point that I attempted the successful "pretend edit", so I can't be sure if fixing that was necessary at all.)

    All clients and the server are qemu VMs.  Not sure why only one was affected.  It's a minor annoyance, but I'm curious to know why it happened, if anyone wants to venture to guess.



  • UPDATE:

    The error after the update caused other issues (can't recall what now), but it was not the cause of this issue.

    This is getting more and more annoying.

    Not only does it happen reliably on boot to multiple servers, it also happens spontaneously.

    The best temporary fix I've found is to log into the remote unresponsive server via ddns, edit any gateway, save it without making any changes, then apply the changes I did not make, and voila, it works.

    It would be preferable if I could make it stop happening, but I'm not exactly sure what is going on at the time it happens because I don't find out until hours later, and at that point I need to get it started ASAP.

    I'm going to try and be more proactive and see if I can catch it happening.



  • I'm still having this problem with all pfsense VPN clients.

    The tap server is also running on pfsense.  I was wondering if the fact that all the clients are using ddns may have something to do with it?

    Is there a way to have it refresh the connection automatically via the GUI?

    I suppose I could write a script to do it automatically, but that's inelegant.



  • creating gateways is more then likely not the way todo this.
    generally you don't add gateways/routes for directly attached networks, pfsense already knows about them

    draw up a schematic of your system (with subnets included) & also post some screenshots of the ovpns config



  • After writing this all down, it seems to me like the problem must be that the routes fail to be created before the VPN is up…

    @heper:

    creating gateways is more then likely not the way todo this.
    generally you don't add gateways/routes for directly attached networks, pfsense already knows about them

    draw up a schematic of your system (with subnets included) & also post some screenshots of the ovpns config

    Here is the general idea:

    –------------10.0.0.100 (TAP iface)---------------------- Client 1 - 10.5.0.0/24 (LAN iface)
                                  |
    (PUBLIC IP) Server - 10.0.0.1 (OPT iface)                             
                                  |
                                  ---------------10.0.0.101 (TAP iface)---------------------- Client 2 - 10.10.0.0/24 (LAN iface)

    My goal is to be able to resolve Client 1's subnet from Client 2, as well as the Server's subnet from either.

    
    : cat server1.conf
    dev ovpns1
    verb 1
    dev-type tap
    dev-node /dev/tap1
    writepid /var/run/openvpn_server1.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp
    cipher AES-128-CBC
    auth SHA1
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    local 122.187.103.110
    tls-server
    server-bridge 10.0.0.1 255.0.0.0 10.0.0.100 10.0.0.110
    client-config-dir /var/etc/openvpn-csc/server1
    tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'ads-dps-pfsense-1' 1"
    lport 1194
    management /var/etc/openvpn/server1.sock unix
    client-to-client
    ca /var/etc/openvpn/server1.ca
    cert /var/etc/openvpn/server1.cert
    key /var/etc/openvpn/server1.key
    dh /etc/dh-parameters.1024
    crl-verify /var/etc/openvpn/server1.crl-verify
    tls-auth /var/etc/openvpn/server1.tls-auth 0
    push "route 10.0.0.0 255.255.255.0"
    
    

    Then I use client specific over-rides to ensure that the client's always pull the same DHCP address on the bridge iface:

    
    openvpn-csc/server1/ads-120mainst-pfsense-1
    ifconfig-push 10.0.0.100 255.255.255.0
    
    

    On the client side:

    
    cat /var/etc/openvpn/client1.conf
    dev ovpnc1
    verb 1
    dev-type tap
    dev-node /dev/tap1
    writepid /var/run/openvpn_client1.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp
    cipher AES-128-CBC
    auth SHA1
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    local 76.102.33.103
    tls-client
    client
    lport 0
    management /var/etc/openvpn/client1.sock unix
    remote 122.187.103.110 1194
    ca /var/etc/openvpn/client1.ca
    cert /var/etc/openvpn/client1.cert
    key /var/etc/openvpn/client1.key
    tls-auth /var/etc/openvpn/client1.tls-auth 1
    resolv-retry infinite
    
    

    To get the routing working between them, on Client 1 I then set up gateway's pointing to the 10.0.0.100 and 10.0.0.101 addresses, and then corresponding static routes to the correct subnets.  I then set up Client 2 and the server similarly.

    There are then at least one or maybe a couple of routing rules and outbound rules (created automatically) that need to be in place on server and clients.

    It's certainly not the easiest way to set it up, and, again, I'm not married to it.  If I remember correctly the reasons I set it up this way are:

    • The ultimate goal is to be able to start a VM with  an ip from the client subnet on the server side of the VPN, and have clients on either side be none the wiser.  The way I was able get that working without the routing rules, IIRC, the netmasks had to be set up in an unacceptable fashion.

    • As there are a few more client subnets, this gives me control as to which subnets are resolvable to each other, and in what directorn  A rouge actor could set up routes and gateways to open up their subnet, but not reach others'.

    • I can still block or allow traffic at the Layer 3 level from the individual client pfsenses via routing rules (though not centrally, from the server, IIRC)

    As I mentioned at the open, I think this has to do with the timing of when pfsense attempts to create the route on boot, vs when pfsense brings up the VPN connection so that the route can be created.  Obviously, the former must happen last, and I must be forcing it to happen at a viable time by "pretend editing" the route after boot.

    Maybe there's a better way to do it?



  • Bump.  Any help, pointers, or suggestions are appreciated.



  • As a result of this problem, the VPNs are too unreliable to use for any sort of production purposes.  E.g. when I do backups to a remote server, I just send them across the WAN.

    Any suggestions would be appreciated.

    Bump.


  • Rebel Alliance Developer Netgate

    Never, ever, ever make static routes that point to OpenVPN. It fails in exactly this way.

    OpenVPN manages routes internally. Depending on your setup, you need to set them in the local/remote networks on the clients and servers and possibly in client-specific override entries on the server.

    If you can describe the setup of your VPN more in-depth that would help. For example, which VPN mode you're using (static key or SSL/TLS), the tunnel network you have set, etc.



  • I got this working.  I'm posting my setup for posterity, since there's a shortage of docs for this stuff.  The goal is to set up a TAP VPN in a hub-and-spoke-format:

    @jimp:

    Never, ever, ever make static routes that point to OpenVPN. It fails in exactly this way.

    OpenVPN manages routes internally. Depending on your setup, you need to set them in the local/remote networks on the clients and servers and possibly in client-specific override entries on the server.

    If you can describe the setup of your VPN more in-depth that would help. For example, which VPN mode you're using (static key or SSL/TLS), the tunnel network you have set, etc.

    I saw what you meant.  I should just be able to push the routes I need from the server.

    So I ripped everything out, except for the certs and my client specific overrides (which is just used to specify the bridge iface IP).  I deleted every route and gateway that I had manually made, and removed every reference to remote or local LANs in both the client and server setting.  I added just two directives to the advanced section of the client specific settings.

    To set the bridge interface ip address for the client on SITE A:

    ifconfig-push 10.0.0.100 255.255.255.0;

    I always had that, but to properly set the route, mask, and gw for client on SITE B's subnet, all I needed to do was:

    push "route 10.10.0.0 255.255.255.0 10.0.0.101";

    Therefore the client on SITE B must have it's address assigned as follows:

    ifconfig-push 10.0.0.101 255.255.255.0;

    … and it can resolve SITE A through SITE A's client's bridge interface address which we just set above ...

    push "route 10.5.0.0 255.255.255.0 10.0.0.100";

    The last thing you need to do is allow/block traffic on the bridge interface (Firewall -> Rules -> OpenVPN.)

    • Block 67-68 (DHCP) from any source to any destination

    • Allow from * to * (or on on a per subnet basis)

    That's it.  No need for anything else.

    Thanks everyone for all the help..