OpenVPN Issue with 2.4 upgrade

unclebacon · Nov 18, 2017, 2:44 AM

I am also having the same issue.

netstat -rn

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            WAN1_GW            UGS        bge1
8.8.4.4            WAN2_GW            UGHS       bge2
8.8.8.8            WAN1_GW            UGHS       bge1
9.9.9.9            WAN2_GW            UGHS       bge2
10.4.0.1           10.4.28.44         UGHS        lo0
127.0.0.1          link#4             UH          lo0
WAN1_RANGE/24      link#2             U          bge1
192.168.0.0/22     link#1             U          bge0
WAN1_IP            link#2             UHS         lo0
192.168.1.1        link#1             UHS         lo0
WAN2_RANGE/27      link#3             U          bge2
WAN2_IP            link#3             UHS         lo0
208.67.220.220     WAN1_GW            UGHS       bge1
208.67.222.222     WAN2_GW            UGHS       bge2

ifconfig -a

bge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=c009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso,linkstate>ether xx:xx:xx:xx:xx:xx
        hwaddr xx:xx:xx:xx:xx:xx
        inet6 xxxx::xxx:xxxx:xxxx:xxxx%bge0 prefixlen 64 scopeid 0x1
        inet 192.168.1.1 netmask 0xfffffc00 broadcast 192.168.3.255
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bge1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether xx:xx:xx:xx:xx:xx
        hwaddr xx:xx:xx:xx:xx:xx
        inet6 xxxx::xxx:xxxx:xxxx:xxxx%bge1 prefixlen 64 scopeid 0x2
        inet WAN1_IP netmask 0xffffffe0 broadcast 255.255.255.255
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
bge2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether xx:xx:xx:xx:xx:xx
        hwaddr xx:xx:xx:xx:xx:xx
        inet6 xxxx::xxx:xxxx:xxxx:xxxx%bge2 prefixlen 64 scopeid 0x3
        inet WAN2_IP netmask 0xffffffe0 broadcast 255.255.255.255
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21 <performnud,auto_linklocal>groups: lo
enc0: flags=0<> metric 0 mtu 1536
        nd6 options=21 <performnud,auto_linklocal>groups: enc
pflog0: flags=100 <promisc>metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
        syncpeer: 224.0.0.240 maxupd: 128 defer: on
        syncok: 1
ovpnc1: flags=8010 <pointopoint,multicast>metric 0 mtu 1500
        options=80000 <linkstate>nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn</performnud,auto_linklocal></linkstate></pointopoint,multicast></promisc></performnud,auto_linklocal></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></full-duplex,master></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,simplex,multicast></full-duplex,master></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast>

ps uxaww | grep openvpn

root 24877   0.0  0.0  14728  2432  0  S+   21:21      0:00.00 grep openvpn

1. Is the VPN interface assigned/enabled under the Interfaces menu? Yes.
2. Does the VPN gateway have an alternate monitoring IP address? No.
3. Is there a DNS server set to use the VPN gateway? No.
4. Are there any manually-defined static routes set to the use VPN gateway? No.
5. Any dynamic routing protocols using the VPN? No.

I currently have 1 AirVPN client configured and no OpenVPN servers configured. Hope this helps.

jarrad · Nov 18, 2017, 3:24 AM

An update from me.

I would say that the fault lays here:

Any OpenVPN gateways under System -> Routing that were present with an alternate monitoring IP prior to 2.4.1 are the cause of the issue.

I went through and deleted the following:
OpenVPN Server, Interface, CA, Certs, Gateways

I am now able to

Stop and start the OpenVPN server correctly without having to reboot
Have alternate IP monitoring respond correctly via dpinger.

RHLinux · Nov 18, 2017, 12:23 PM

@jarrad:

An update from me.

I would say that the fault lays here:

Any OpenVPN gateways under System -> Routing that were present with an alternate monitoring IP prior to 2.4.1 are the cause of the issue.

I went through and deleted the following:
OpenVPN Server, Interface, CA, Certs, Gateways

I am now able to

Stop and start the OpenVPN server correctly without having to reboot

Have alternate IP monitoring respond correctly via dpinger.

That is good to know. I also cleared all the above but still had some issues. I also had an alternative monitoring IP address (monitoring the other end of the link) and had this issue only when upgrading. I ended up changing the IP address of the link and rebooted the system. This seemed to fix the problem.

RHLinux

amires · Nov 21, 2017, 5:15 PM

I have clean installed pfSense 2.4.1 and made all the necessary configs all by hand and still have this issue. I am going to disable ip monitoring
for OpenVPN and see if that helps.

I found out that is not just a WAN IP change that triggers this, there are other circumstances involved. I manually forced WAN IP change many
times and OpenVPN client recovered successfully every time. So it is not just a WAN IP change that triggers this.

By the way I am not using OpenVPN Server, it is the OpenVPN Client that I have issues with.

UPDATE : I haven't had any more issues since I have disabled gateway monitoring on OpenVPN Client interface. My pfSense machine has been
up for 3 days now without any kind of problem.

jarrad · Nov 27, 2017, 11:35 AM

I spoke too soon.

It broke as soon as I added a second client.

I've since rebuilt it again from scratch and now the server won't assign itself an IP so routing internally is broken but clients can communicate.

This also coincided with the upgrade to 2.4.2.

unclebacon · Nov 30, 2017, 4:04 PM

Still having this same issue. It seems to happen upon any manual changes to the routes or if OpenVPN loses it's connection and attempts to reestablish it, giving these errors:

Nov 30 10:16:28	openvpn	89059	TUN/TAP device ovpnc1 exists previously, keep at program end
Nov 30 10:16:28	openvpn	89059	TUN/TAP device /dev/tun1 opened
Nov 30 10:16:28	openvpn	89059	do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Nov 30 10:16:28	openvpn	89059	/sbin/ifconfig ovpnc1 10.4.28.44 10.4.0.1 mtu 1500 netmask 255.255.0.0 up
Nov 30 10:16:28	openvpn	89059	FreeBSD ifconfig failed: external program exited with error status: 1
Nov 30 10:16:28	openvpn	89059	Exiting due to fatal error

It appears the only way to fix it (albeit temporarily) is to reboot and wait for it to happen once more.

10.4.0.1           10.4.28.44         UGHS        lo0

This is the leftover route that seems to be causing the issue, and any attempt to remove it manually gives:

[2.4.2-RELEASE][root@pfSense.local.lan]/dev: route delete 10.4.0.1
route: writing to routing socket: Address already in use
delete host 10.4.0.1 fib 0: gateway uses the same route

I have tried multiple iterations of the above route command with no success. I've manually removed the ovpnc1 interface, tun device. No luck. Any ideas?

Slugger · Nov 30, 2017, 4:32 PM

Hitting the same issues and I'm also using AirVPN as my vpn provider. I have 3 OpenVPN connections on my 2.4.2 install:

Site to Site with shared key
Remote access
Client connection to AirVPN

Site to site and remote access haven't had any problems and the client connection to AirVPN only causes troubles when I've set an explicit monitor IP. Once I removed the monitor IP, the AirVPN connection hasn't caused me any problems. Once I add the monitor IP, the connection is fine until it drops and needs to reconnect (for whatever reason) then it won't reconnect with the same ifconfig error as reported by others. That same static route for the monitor IP hangs around as reported by others and I simply can't get it to go away nor can I get the tunnel to AirVPN to reconnect unless I:

reboot pfSense OR
Change the port I connect to AirVPN on, which then changes the link IP on the connection from 10.6.0.0/16 to say 10.4.0.0/16 which then let's everything reconnect but with that extra static route hanging around and then when it disconnects again then I now have two static routes that hang around, etc.

And to answer the questions posed earlier in the thread:

1. Is the VPN interface assigned/enabled under the Interfaces menu? Yes
2. Does the VPN gateway have an alternate monitoring IP address? Yes (when I hit this problem, but for now I've removed the explicit monitor IP and haven't had any problems)
3. Is there a DNS server set to use the VPN gateway? No
4. Are there any manually-defined static routes set to the use VPN gateway? (there should never be, but some people add them not realizing they are a problem) No
5. Any dynamic routing protocols using the VPN? No

jimp · Nov 30, 2017, 4:42 PM

So now my question is this: Is there anyone having this problem that is NOT using AirVPN?

It may be triggered by some option pushed to the client by AirVPN. Rather than focusing on the disconnection, get some logs from when AirVPN connects, maybe with an increased verb level that will show what they are pushing.

unclebacon · Nov 30, 2017, 5:05 PM

@jimp:

So now my question is this: Is there anyone having this problem that is NOT using AirVPN?

It may be triggered by some option pushed to the client by AirVPN. Rather than focusing on the disconnection, get some logs from when AirVPN connects, maybe with an increased verb level that will show what they are pushing.

Here is a full log from OpenVPN set to the highest verbosity level.

https://pastebin.com/29eWQCGY

dsp3 · Nov 30, 2017, 8:55 PM

@SirJohnEh:

Site to site and remote access haven't had any problems and the client connection to AirVPN only causes troubles when I've set an explicit monitor IP. Once I removed the monitor IP, the AirVPN connection hasn't caused me any problems. Once I add the monitor IP, the connection is fine until it drops and needs to reconnect (for whatever reason) then it won't reconnect with the same ifconfig error as reported by others. That same static route for the monitor IP hangs around as reported by others and I simply can't get it to go away nor can I get the tunnel to AirVPN to reconnect unless I:

Leave gateway monitoring enabled, but do not put in an IP address to monitor. Does that work for you?

Slugger · Nov 30, 2017, 9:16 PM

Yes it does. The only issue is it ends up monitoring its own IP address, which isn't very useful, but yes it does work (and it's what I'm actually doing now as a workaround).

unclebacon · Dec 1, 2017, 5:06 PM

That seems to be a workaround for me as well.

amires · Dec 5, 2017, 9:12 PM

For me the solution was to stop using AirVPN's gateway (10.4.0.1) as monitoring ip. I set 8.8.8.8 as the monitoring ip about two weeks ago and since then there were not any more OpenVPN crashes.

RHLinux · Jan 20, 2018, 8:11 AM

@jimp:

So now my question is this: Is there anyone having this problem that is NOT using AirVPN?

It may be triggered by some option pushed to the client by AirVPN. Rather than focusing on the disconnection, get some logs from when AirVPN connects, maybe with an increased verb level that will show what they are pushing.

Sorry for the late reply, but yes I had this issue and I am not using AirVPN, I have my own private VPN server setup and had this issue also. Seems to be linked to the monitoring IP on the remote end. After changing the remote monitoring end IP address it seems to clear the route in the routing table.

RHLinux

Warudo · Feb 28, 2019, 4:34 PM

Ran into the same issue with Mullvad.

Gabri.91 · Jul 2, 2019, 11:40 AM

Hi @jimp I have the same issue and updated the redmine: https://redmine.pfsense.org/issues/8142

As you can see I have full control over the VPN server (and options) so I can do whatever test/log is needed in order to sort out the issue.