[Solved]: Failing to create routes on boot. Must "pretend edit" a route
-
I have a pfsense tap vpn server on 10.0.0.1/24.
I have several subnets connected to it via pfsense routers, i.e. 10.5.0.1/24, 10.10.0.1/24, 10.20.0.1/24, etc.
The tap interface on the servers assigns them address at 10.0.0.100, .101, .102, respectively.
I then add routes to those subnets on a case by case basis by creating a gateway on the interface, pointing it to the tap IP address (e.g. 10.0.0.102), and then adding the 10.20.0.0/24 subnet as a static route via that gateway.
One day I noticed I could not reach the other client subnets or vice versa from 10.5.0.1. I then noticed the reason was that the routes were not created.
Eventually, I decided to pretend to edit one of the gateways, changed nothing, saved, and applied. Then all the routes were created, and everything was working again.
(I'm not sure if this was related, but I ran into a bug where suhosin.so was failing to load, and I worked around it by adding "extension=suhosin" to /usr/etc/php/extensions.ini. It was only after that point that I attempted the successful "pretend edit", so I can't be sure if fixing that was necessary at all.)
All clients and the server are qemu VMs. Not sure why only one was affected. It's a minor annoyance, but I'm curious to know why it happened, if anyone wants to venture to guess.
-
UPDATE:
The error after the update caused other issues (can't recall what now), but it was not the cause of this issue.
This is getting more and more annoying.
Not only does it happen reliably on boot to multiple servers, it also happens spontaneously.
The best temporary fix I've found is to log into the remote unresponsive server via ddns, edit any gateway, save it without making any changes, then apply the changes I did not make, and voila, it works.
It would be preferable if I could make it stop happening, but I'm not exactly sure what is going on at the time it happens because I don't find out until hours later, and at that point I need to get it started ASAP.
I'm going to try and be more proactive and see if I can catch it happening.
-
I'm still having this problem with all pfsense VPN clients.
The tap server is also running on pfsense. I was wondering if the fact that all the clients are using ddns may have something to do with it?
Is there a way to have it refresh the connection automatically via the GUI?
I suppose I could write a script to do it automatically, but that's inelegant.
-
creating gateways is more then likely not the way todo this.
generally you don't add gateways/routes for directly attached networks, pfsense already knows about themdraw up a schematic of your system (with subnets included) & also post some screenshots of the ovpns config
-
After writing this all down, it seems to me like the problem must be that the routes fail to be created before the VPN is up…
creating gateways is more then likely not the way todo this.
generally you don't add gateways/routes for directly attached networks, pfsense already knows about themdraw up a schematic of your system (with subnets included) & also post some screenshots of the ovpns config
Here is the general idea:
–------------10.0.0.100 (TAP iface)---------------------- Client 1 - 10.5.0.0/24 (LAN iface)
|
(PUBLIC IP) Server - 10.0.0.1 (OPT iface)
|
---------------10.0.0.101 (TAP iface)---------------------- Client 2 - 10.10.0.0/24 (LAN iface)My goal is to be able to resolve Client 1's subnet from Client 2, as well as the Server's subnet from either.
: cat server1.conf dev ovpns1 verb 1 dev-type tap dev-node /dev/tap1 writepid /var/run/openvpn_server1.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp cipher AES-128-CBC auth SHA1 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 122.187.103.110 tls-server server-bridge 10.0.0.1 255.0.0.0 10.0.0.100 10.0.0.110 client-config-dir /var/etc/openvpn-csc/server1 tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'ads-dps-pfsense-1' 1" lport 1194 management /var/etc/openvpn/server1.sock unix client-to-client ca /var/etc/openvpn/server1.ca cert /var/etc/openvpn/server1.cert key /var/etc/openvpn/server1.key dh /etc/dh-parameters.1024 crl-verify /var/etc/openvpn/server1.crl-verify tls-auth /var/etc/openvpn/server1.tls-auth 0 push "route 10.0.0.0 255.255.255.0"
Then I use client specific over-rides to ensure that the client's always pull the same DHCP address on the bridge iface:
openvpn-csc/server1/ads-120mainst-pfsense-1 ifconfig-push 10.0.0.100 255.255.255.0
On the client side:
cat /var/etc/openvpn/client1.conf dev ovpnc1 verb 1 dev-type tap dev-node /dev/tap1 writepid /var/run/openvpn_client1.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp cipher AES-128-CBC auth SHA1 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 76.102.33.103 tls-client client lport 0 management /var/etc/openvpn/client1.sock unix remote 122.187.103.110 1194 ca /var/etc/openvpn/client1.ca cert /var/etc/openvpn/client1.cert key /var/etc/openvpn/client1.key tls-auth /var/etc/openvpn/client1.tls-auth 1 resolv-retry infinite
To get the routing working between them, on Client 1 I then set up gateway's pointing to the 10.0.0.100 and 10.0.0.101 addresses, and then corresponding static routes to the correct subnets. I then set up Client 2 and the server similarly.
There are then at least one or maybe a couple of routing rules and outbound rules (created automatically) that need to be in place on server and clients.
It's certainly not the easiest way to set it up, and, again, I'm not married to it. If I remember correctly the reasons I set it up this way are:
-
The ultimate goal is to be able to start a VM with an ip from the client subnet on the server side of the VPN, and have clients on either side be none the wiser. The way I was able get that working without the routing rules, IIRC, the netmasks had to be set up in an unacceptable fashion.
-
As there are a few more client subnets, this gives me control as to which subnets are resolvable to each other, and in what directorn A rouge actor could set up routes and gateways to open up their subnet, but not reach others'.
-
I can still block or allow traffic at the Layer 3 level from the individual client pfsenses via routing rules (though not centrally, from the server, IIRC)
As I mentioned at the open, I think this has to do with the timing of when pfsense attempts to create the route on boot, vs when pfsense brings up the VPN connection so that the route can be created. Obviously, the former must happen last, and I must be forcing it to happen at a viable time by "pretend editing" the route after boot.
Maybe there's a better way to do it?
-
-
Bump. Any help, pointers, or suggestions are appreciated.
-
As a result of this problem, the VPNs are too unreliable to use for any sort of production purposes. E.g. when I do backups to a remote server, I just send them across the WAN.
Any suggestions would be appreciated.
Bump.
-
Never, ever, ever make static routes that point to OpenVPN. It fails in exactly this way.
OpenVPN manages routes internally. Depending on your setup, you need to set them in the local/remote networks on the clients and servers and possibly in client-specific override entries on the server.
If you can describe the setup of your VPN more in-depth that would help. For example, which VPN mode you're using (static key or SSL/TLS), the tunnel network you have set, etc.
-
I got this working. I'm posting my setup for posterity, since there's a shortage of docs for this stuff. The goal is to set up a TAP VPN in a hub-and-spoke-format:
Never, ever, ever make static routes that point to OpenVPN. It fails in exactly this way.
OpenVPN manages routes internally. Depending on your setup, you need to set them in the local/remote networks on the clients and servers and possibly in client-specific override entries on the server.
If you can describe the setup of your VPN more in-depth that would help. For example, which VPN mode you're using (static key or SSL/TLS), the tunnel network you have set, etc.
I saw what you meant. I should just be able to push the routes I need from the server.
So I ripped everything out, except for the certs and my client specific overrides (which is just used to specify the bridge iface IP). I deleted every route and gateway that I had manually made, and removed every reference to remote or local LANs in both the client and server setting. I added just two directives to the advanced section of the client specific settings.
To set the bridge interface ip address for the client on SITE A:
ifconfig-push 10.0.0.100 255.255.255.0;
I always had that, but to properly set the route, mask, and gw for client on SITE B's subnet, all I needed to do was:
push "route 10.10.0.0 255.255.255.0 10.0.0.101";
Therefore the client on SITE B must have it's address assigned as follows:
ifconfig-push 10.0.0.101 255.255.255.0;
… and it can resolve SITE A through SITE A's client's bridge interface address which we just set above ...
push "route 10.5.0.0 255.255.255.0 10.0.0.100";
The last thing you need to do is allow/block traffic on the bridge interface (Firewall -> Rules -> OpenVPN.)
-
Block 67-68 (DHCP) from any source to any destination
-
Allow from * to * (or on on a per subnet basis)
That's it. No need for anything else.
Thanks everyone for all the help..
-