OpenVPN Issue with 2.4 upgrade

stormforge

"I'm having maybe the same problem with the 2.4.1 upgrade (from 2.3.4-RELEASE-P1). "

Correction. Upgrade to 2.3.4-RELEASE-P1 from 2.3.3.

immy

I am also having the problem of an old route that doesn't get deleted (also on 2.4.1)

/sbin/ifconfig ovpnc1 10.4.6.92 10.4.0.1 mtu 1500 netmask 255.255.0.0 up
ifconfig fails error 1

From netstat:
10.4.0.1 10.4.3.89 UGHS lo0

Trying manually:

/sbin/ifconfig ovpnc1 10.4.6.92 10.4.0.1 mtu 1500 netmask 255.255.0.0 up
ifconfig: ioctl (SIOCAIFADDR): File exists

Trying to delete the old route:

route del 10.4.0.1 10.4.3.89
route: writing to routing socket: Address already in use
del host 10.4.0.1: gateway 10.4.3.89 fib 0: gateway uses the same route

If I reboot or flush the routes it will work again until my ADSL goes down.

cosmoxl

When will the devs pay attention to this?

RHLinux

Who knows… It's defiantly still an issue with multi client/multi WAN.

RHLinux

jimp

@cosmoxl:

When will the devs pay attention to this?

When we have some solid evidence other than vague reports that it's broken with barely any detail. I have probably 30-40 OpenVPN instances between my lab and live boxes, a mix of clients, servers, multi-WAN, routing protocols, you name it. I have yet to see this happen. About the only thing I don't have is a connection to an external VPN provider. I have simulated one internally, but I don't subscribe to any public ones.

We're going to need a lot more detail about the specific configurations in play here on both sides when it happens, more details from the logs, OS routing tables, anything that might be relevant.

cosmoxl

@jimp:

@cosmoxl:

When will the devs pay attention to this?

When we have some solid evidence other than vague reports that it's broken with barely any detail. I have probably 30-40 OpenVPN instances between my lab and live boxes, a mix of clients, servers, multi-WAN, routing protocols, you name it. I have yet to see this happen. About the only thing I don't have is a connection to an external VPN provider. I have simulated one internally, but I don't subscribe to any public ones.

We're going to need a lot more detail about the specific configurations in play here on both sides when it happens, more details from the logs, OS routing tables, anything that might be relevant.

OK. In another thread user gave me the tip that solves the problem - that is to stop specifying a different IP address to ping for gateway monitoring. That fixed the problem for me.

Perhaps that helps you understand the problem.

amires

I have this problem too. Since upgrading to pfSense 2.4.x my OpenVPN client stays connected until a change in WAN is detected (cable disconnect, IP change, etc) after that it starts giving out ifconfig errors and it wont connect anymore unless I reboot my pfsense box. The problem seems to be VPN routes are not being deleted when vpn goes down. I went back to 2.3.4 and the problem is gone. I will stay on 2.3.4 until this is fixed.

amires

@cosmoxl:

OK. In another thread user gave me the tip that solves the problem - that is to stop specifying a different IP address to ping for gateway monitoring. That fixed the problem for me.

Perhaps that helps you understand the problem.

What different IP should we use? I am using 10.4.0.1 which is gateway of AirVPN provider.

jimp

If setting a monitor IP address on the VPN gateway triggered it, then that makes me wonder if it's related to how the routes are added.

Anyone having the issue, check for:

1. Is the VPN interface assigned/enabled under the Interfaces menu?
2. Does the VPN gateway have an alternate monitoring IP address?
3. Is there a DNS server set to use the VPN gateway?
4. Are there any manually-defined static routes set to the use VPN gateway? (there should never be, but some people add them not realizing they are a problem)
5. Any dynamic routing protocols using the VPN?

jarrad

1. Is the VPN interface assigned/enabled under the Interfaces menu? - Yes. Removed the interface and no change. Readded interface, no change.
2. Does the VPN gateway have an alternate monitoring IP address? - It did (it was one of the client expected IP addresses). Removed the alternate monitoring IP, no change.
3. Is there a DNS server set to use the VPN gateway? - No
4. Are there any manually-defined static routes set to the use VPN gateway? (there should never be, but some people add them not realizing they are a problem) - No
5. Any dynamic routing protocols using the VPN? - No

I do have an OpenVPN client (to my VPN provider) running alongside and separate to the OpenVPN server.

jimp

Even if I try to intentionally break it that way, it still restarts fine for me. I do see a route hanging around for the gateway monitor IP address but it doesn't prevent anything from working, and when the VPN restarts it goes right back online. No errors.

I am using 2.4.2 to test this, however. Maybe something else changed there along the way that fixed the problem. Either that or I still don't have enough detail to reproduce the original issue.

amires

1. Is the VPN interface assigned/enabled under the Interfaces menu? Yes.
2. Does the VPN gateway have an alternate monitoring IP address? I have to check this.
3. Is there a DNS server set to use the VPN gateway? Yes, same as VPN gateway.
4. Are there any manually-defined static routes set to the use VPN gateway? No.
5. Any dynamic routing protocols using the VPN? No.

I am using AirVPN. My VPN gateway, DNS and Monitoring IP are all 10.4.0.1.

jimp

@amires:

3. Is there a DNS server set to use the VPN gateway? Yes, same as VPN gateway.
I am using AirVPN. My VPN gateway, DNS and Monitoring IP are all 10.4.0.1.

On System > General Setup, do not set a gateway on the DNS server there. Just leave the entry for 10.4.0.1 and set the gateway to "none". You're making a redundant route by doing that, which could be part of the problem. If 10.4.0.1 is your VPN gateway then you'll already have a route there. Selecting that on the DNS server list is completely unnecessary. We prevent that for static IP interfaces where the network is known, but since those VPN interfaces are dynamic it isn't prevented.

You're lucky that didn't cause a problem in the past.

amires

@jimp:

@amires:

3. Is there a DNS server set to use the VPN gateway? Yes, same as VPN gateway.
I am using AirVPN. My VPN gateway, DNS and Monitoring IP are all 10.4.0.1.

On System > General Setup, do not set a gateway on the DNS server there. Just leave the entry for 10.4.0.1 and set the gateway to "none". You're making a redundant route by doing that, which could be part of the problem. If 10.4.0.1 is your VPN gateway then you'll already have a route there. Selecting that on the DNS server list is completely unnecessary. We prevent that for static IP interfaces where the network is known, but since those VPN interfaces are dynamic it isn't prevented.

You're lucky that didn't cause a problem in the past.

Actually I have tried it both without a gateway and with VPN gateway and none did help. I just removed the gateway just to be safe.
I also removed 10.4.0.1 as monitoring ip and used 8.8.8.8 instead. I am waiting to see if the problem still exist.

jimp

@amires:

Actually I have tried it both without a gateway and with VPN gateway and none did help. I just removed the gateway just to be safe.
I also removed 10.4.0.1 as monitoring ip and used 8.8.8.8 instead. I am waiting to see if the problem still exist.

Depending on the steps you took in between attempts that may not have been a relevant test. You can use the gateway for the monitor IP address, that's what it does by default. Leave the monitor IP address field blank. Setting something else may be part of the overall issue (though it works fine here).

This may not be relevant to you, but to others: If you do need to use a DNS server across the VPN, do not set the gateway on the DNS server list, but add the DNS server IP address/32 to the remote network list of your VPN client. That way OpenVPN will manage the route. But to reiterate, that is NOT necessary if the DNS server is directly connected or especially if it is your VPN gateway.

amires

@jimp:

@amires:

Actually I have tried it both without a gateway and with VPN gateway and none did help. I just removed the gateway just to be safe.
I also removed 10.4.0.1 as monitoring ip and used 8.8.8.8 instead. I am waiting to see if the problem still exist.

Depending on the steps you took in between attempts that may not have been a relevant test. You can use the gateway for the monitor IP address, that's what it does by default. Leave the monitor IP address field blank. Setting something else may be part of the overall issue (though it works fine here).

This may not be relevant to you, but to others: If you do need to use a DNS server across the VPN, do not set the gateway on the DNS server list, but add the DNS server IP address/32 to the remote network list of your VPN client. That way OpenVPN will manage the route. But to reiterate, that is NOT necessary if the DNS server is directly connected or especially if it is your VPN gateway.

Thanks for your help. When I leave the monitoring IP blank then pfsense will use locally assigned VPN IP as the monitoring IP which is not a reliable way to check availability of the link. I need to monitor other side of the link which is 10.4.0.1 therefore I am forced to either manually enter 10.4.0.1 as monitoring IP or use a different pingable public IP but then I wouldn't be able to monitor RTT of the VPN link.

If it helps here are the relevant VPN routes from my routing table when I am using 10.4.0.1 as both DNS and monitoring IP.

Destination Gateway Flags Netif Expire
10.4.0.0/16 10.4.0.1 UGS ovpnc1
10.4.0.1 10.4.x.y UGHS ovpnc1
10.4.x.y link#9 UHS lo0

cosmoxl

Well, I tested two concurrent OVPN clients connected, both with alternate IP monitored, and stopping and reconnecting one of them.

This would have caused a problem (ifconfig and/or gateway monitoring saying gateway was down) when I was testing 2.4 a few months ago but didn't cause a problem today.

I'm at a loss, because I know others are still having problems.

amires

This morning the mentioned errors appeared again. OpenVPN wont connect and I had the following errors in the logs :

/sbin/ifconfig ovpnc1 10.4.4.12 10.4.0.1 mtu 1500 netmask 255.255.0.0 up
FreeBSD ifconfig failed: external program exited with error status: 1
Exiting due to fatal error

I checked the routing table and I noticed that I have the following VPN route :

10.4.0.1 10.4.4.12 UGHS lo0

I tried manually deleting the route :

route del 10.4.0.1
route: writing to routing socket: Address already in use
del host 10.4.0.1 fib 0: gateway uses the same route

I restarted my pfsense box at this point. After restarting everything was ok again.

jimp

@amires:

/sbin/ifconfig ovpnc1 10.4.4.12 10.4.0.1 mtu 1500 netmask 255.255.0.0 up
FreeBSD ifconfig failed: external program exited with error status: 1
Exiting due to fatal error

I tried manually deleting the route :
route del 10.4.0.1
route: writing to routing socket: Address already in use
del host 10.4.0.1 fib 0: gateway uses the same route

If that happens again, post the entire contents of "netstat -rn" and "ifconfig -a" and "ps uxaww | grep openvpn" before attempting to delete the route. Something else must still be running/using that address.

amires

@jimp:

If that happens again, post the entire contents of "netstat -rn" and "ifconfig -a" and "ps uxaww | grep openvpn" before attempting to delete the route. Something else must still be running/using that address.

It happended again and it won't connect anymore. I am posting below the information your requested.

netstat -rn

Destination        Gateway            Flags     Netif Expire
default            ADSL_GATEWAY       UGS      pppoe0
10.4.0.1           10.4.4.12          UGHS        lo0
8.8.8.8            192.168.1.1        UGHS        em0
127.0.0.1          link#4             UH          lo0
172.16.0.0/30      link#1             U           re0
172.16.0.2         link#1             UHS         lo0
ADSL_GATEWAY       link#8             UH       pppoe0
ADSL_IP            link#8             UHS         lo0
192.168.1.0/24     link#2             U           em0
192.168.1.2        link#2             UHS         lo0
192.168.2.0/24     link#3             U           em1
192.168.2.1        link#3             UHS         lo0
8.8.4.4            pppoe0             UHS      pppoe0

ifconfig -a

re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=8209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate>ether xx:xx:xx:xx:xx:xx
        hwaddr xx:xx:xx:xx:xx:xx
        inet6 fe80::6666:b3ff:fe03:5569%re0 prefixlen 64 scopeid 0x1
        inet 172.16.0.2 netmask 0xfffffffc broadcast 172.16.0.3
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether xx:xx:xx:xx:xx:xx
        hwaddr xx:xx:xx:xx:xx:xx
        inet6 fe80::7254:d2ff:feab:20a6%em0 prefixlen 64 scopeid 0x2
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
        options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether xx:xx:xx:xx:xx:xx
        hwaddr xx:xx:xx:xx:xx:xx
        inet6 fe80::7254:d2ff:feab:20a7%em1 prefixlen 64 scopeid 0x3
        inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21 <performnud,auto_linklocal>groups: lo
enc0: flags=0<> metric 0 mtu 1536
        nd6 options=21 <performnud,auto_linklocal>groups: enc
pflog0: flags=100 <promisc>metric 0 mtu 33160
        groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
        groups: pfsync
        syncpeer: 224.0.0.240 maxupd: 128 defer: on
        syncok: 1
ovpnc1: flags=8010 <pointopoint,multicast>metric 0 mtu 1500
        options=80000 <linkstate>nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn
pppoe0: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1492
        inet ADSL_IP --> ADSL_GATEWAY  netmask 0xffffffff
        inet6 xxxx::xxxx:xxxx:xxxx:xxxx%pppoe0 prefixlen 64 scopeid 0x8
        nd6 options=21<performnud,auto_linklocal></performnud,auto_linklocal></up,pointopoint,running,noarp,simplex,multicast></performnud,auto_linklocal></linkstate></pointopoint,multicast></promisc></performnud,auto_linklocal></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,linkstate></up,broadcast,running,simplex,multicast>

ps uxaww | grep openvpn

root    31186   0.0  0.0  14728   2432  0  S+   22:21     0:00.00 grep openvpn