OpenVPN fails to create route after upgrade to rc1

[beyer]

Hi folks,

after an upgrade from Beta-5 (early February I believe), to RC-1, I have a problem with the openvpn client connections.
The connection itself works just fine, but when pfsense tries to add a route to the vpn network, it logs this error:

openvpn[766]: ERROR: FreeBSD route add command failed: external program exited with error status: 1

Naturally, without the route I cannot access the vpn network on the other side, so this is really a problem atm.
Does anyone have any suggestions on how to fix or debug this further?

Thanks,
Marcus

[eri--]

What is the command failing?

[beyer]

The actual command is not shown in the logs, just the error message posted above. Is there a way to up the verbosity of the openvpn client?

[dave99]

I was about to post the same thing. I just upgraded my main office from 1.2.3 to 2.0rc1, everything worked previously.

I'm using pre-shared key, site to site openvpn, main office is the server, remote office is client. My remote site (also 2.0 rc1) is fine, it can access everything in the main office without issue. However from the main office going to the remote site, I've got nothing. This error is in the openvpn log on the main office side:

openvpn[43883]: ERROR: FreeBSD route add command failed: external program exited with error status: 1

Now if I ssh into pfsense, I can ping the remote office fine, so somewhere the routing is correct. From any of my vlans though it fails, it appears it's routing straight out to the internet.

on the pfsense box, netstat -nr has a routing entry:

192.168.10.0/24    10.91.129.2        UGS         0     2424 ovpns2

Background info:
Local network is 10.90.0.0/19 (with a bunch of vlans)
remote site is 192.168.10.0/24

In the openvpn config, I have the info above, tunnel network is 10.91.129.0/24 on both sides.
On the server side, I have:

route 192.168.10.0 255.255.255.0;push "route 10.90.0.0 255.255.224.0";

in the advanced options.

[jimp]

The usual reason for FreeBSD to fail adding a route is that you already have a route to that subnet.

It's not 100% clear what exactly is where in your setup. Can you specify it a bit more in depth? At least distinguish exactly what subnets are at the main office and which are on the client side. Also, show the openvpn config and route table on both sides.

[dave99]

Ok, a little more testing and I was incorrect earlier, I can access the remote site from the other vlans, except 10.90.0.0/24 (vlan 100 - SAA). It figures this is the one I need though.
Let me know if you need more screenshots or info.

Main office: 10.90.0.0/19 - OpenVPN server - Dual WAN (WAN_T1 & WAN_DSL, openvpn tied to WAN_TA)
subnets can be seen below in the routing table, but generally they are
10.90.0.0/24 VLAN 100
10.90.1.0/24 VLAN 101
etc

Remote office: 192.168.10.0/24 - OpenVPN client

Routing tables -Main Office

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            67.130.244.161     UGS         0   931024    em1
4.2.2.2            216.161.116.210    UGHS        0      784 pppoe0
4.2.2.4            67.130.244.161     UGHS        0      784    em1
10.90.0.0/24       link#9             U           0  1588250 em0_vl
10.90.0.1          link#9             UHS         0        0    lo0
10.90.1.0/24       link#10            U           0    30961 em0_vl
10.90.1.1          link#10            UHS         0        0    lo0
10.90.2.0/24       link#11            U           0    25691 em0_vl
10.90.2.1          link#11            UHS         0        0    lo0
10.90.3.0/24       link#12            U           0     5922 em0_vl
10.90.3.1          link#12            UHS         0        0    lo0
10.90.4.0/24       link#13            U           0        0 em0_vl
10.90.4.1          link#13            UHS         0        0    lo0
10.90.5.0/24       link#14            U           0        0 em0_vl
10.90.5.1          link#14            UHS         0        0    lo0
10.90.6.0/24       link#15            U           0        0 em0_vl
10.90.6.1          link#15            UHS         0        0    lo0
10.90.9.0/24       link#16            U           0   251035 em0_vl
10.90.9.1          link#16            UHS         0        0    lo0
10.90.10.0/24      link#17            U           0    35227 em0_vl
10.90.10.1         link#17            UHS         0        0    lo0
10.90.11.0/24      link#18            U           0        0 em0_vl
10.90.11.1         link#18            UHS         0        0    lo0
10.90.15.0/24      link#24            U           0        0 em0_vl
10.90.15.1         link#24            UHS         0        0    lo0
10.90.28.8/29      link#19            U           0    36129 em0_vl
10.90.28.9         link#19            UHS         0        0    lo0
10.90.28.16/29     link#20            U           0    31750 em0_vl
10.90.28.17        link#20            UHS         0        0    lo0
10.90.28.24/29     link#21            U           0    74373 em0_vl
10.90.28.25        link#21            UHS         0        0    lo0
10.90.28.32/29     link#22            U           0    19915 em0_vl
10.90.28.33        link#22            UHS         0        0    lo0
10.90.30.0/24      link#23            U           0        0 em0_vl
10.90.30.1         link#23            UHS         0        0    lo0
10.90.128.0/24     10.90.128.2        UGS         0      157 ovpns1
10.90.128.1        link#26            UHS         0        0    lo0
10.90.128.2        link#26            UH          0        0 ovpns1
10.91.129.1        link#27            UHS         0        0    lo0
10.91.129.2        link#27            UH          0        0 ovpns2
63.227.79.178      link#25            UHS         0        0    lo0
67.130.244.160/28  link#2             U           0      234    em1
67.130.244.165     link#2             UHS         0        0    lo0
127.0.0.1          link#7             UH          0      194    lo0
192.168.10.0/24    10.91.129.2        UGS         0   202444 ovpns2
205.171.2.65       216.161.116.210    UGHS        0     4468 pppoe0
205.171.3.65       216.161.116.210    UGHS        0     4498 pppoe0
208.67.220.220     67.130.244.161     UGHS        0        0    em1
208.67.222.222     216.161.116.210    UGHS        0        0 pppoe0
216.161.116.210    link#25            UH          0        0 pppoe0

Routing tables- Remote Office

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            216.161.116.209    UGS         0 42180042 pppoe0
10.90.0.0/19       10.91.129.1        UGS         0   222473 ovpnc2
10.91.129.1        link#10            UH          0      788 ovpnc2
10.91.129.2        link#10            UHS         0        0    lo0
63.227.69.58       link#8             UHS         0      555    lo0
127.0.0.1          link#4             UH          0       97    lo0
192.168.0.0/24     192.168.0.2        US          0        0    em1
192.168.0.2        link#2             UHS         0  1808284    lo0
192.168.10.0/24    link#1             U           0 37595518    em0
192.168.10.1       link#1             UHS         0        0    lo0
192.168.190.0/24   192.168.190.2      UGS         0    23784 ovpns1
192.168.190.1      link#9             UHS         0        0    lo0
192.168.190.2      link#9             UH          0        0 ovpns1
216.161.116.209    link#8             UH          0   942669 pppoe0

[jimp]

Do you see anything in the firewall logs on either side?

Have you done packet captures on both routers to see if the traffic you expect to see is actually going into the VPN tunnel interface?

[dave99]

And doing some further testing, I'm not sure this is an openvpn thing. I also have a few ipsec tunnels to some other lesser used sites. The ipsec tunnels are created using the 10.90.0.0/19 subnet also, and again all vlans except for 100 can access the remote sites.

I'd guess there is a more basic network config problem, but I'm just not seeing it. All the main subnets/vlans are created with the same /24 scheme, with similar rules.

Nothing in the firewall logs on either side.

[dave99]

Ok, I've found the cause. My last rule in that vlans sends all non-local traffic (using an inverse) to my the 'failover' gateway group, vlan 100 is the only one that uses 'failover' as it's gateway, all the other vlans use the default. Is the gateway group bypassing the vpn tunnels when routing traffic?

Any thoughts on the best solution? (I can create a rule just before the inverse block allow the vpn traffic, didn't know if that is the cleanest solution).

[jimp]

That is the cleanest solution, make a rule above that that has a gateway of 'default'.