OpenVPN Issue with 2.4 upgrade

cosmoxl · Oct 26, 2017, 12:20 PM

Yes, I had this issue when testing development builds. I reported it in the 2.4 development builds forum but it was ignored because nobody else was seeing problems.

I reverted back to 2.3.4 and all works well again. I won't upgrade to 2.4.x until this is fixed. The non-removal of dynamic routes should be an obvious fix.

RHLinux · Oct 30, 2017, 8:13 PM

Do you have the link to the issue you reported so I can also link to this issue.

Are you using two openvpn clients by any chance? I have two separate VPN Clients that are routed to two different VLANs.

cosmoxl · Oct 30, 2017, 8:55 PM

@RHLinux:

Do you have the link to the issue you reported so I can also link to this issue.

Are you using two openvpn clients by any chance? I have two separate VPN Clients that are routed to two different VLANs.

I brought up the problems I was seeing in a thread in the 2.4 development subforum. I didn't submit an official bug report.

When testing I was using two different openvpn clients but I have no VLANs created.

Do you see problems with gateway monitoring with the two openvpn clients running? I did.

RHLinux · Oct 30, 2017, 9:21 PM

@cosmoxl:

@RHLinux:

Do you have the link to the issue you reported so I can also link to this issue.

Are you using two openvpn clients by any chance? I have two separate VPN Clients that are routed to two different VLANs.

I brought up the problems I was seeing in a thread in the 2.4 development subforum. I didn't submit an official bug report.

When testing I was using two different openvpn clients but I have no VLANs created.

Do you see problems with gateway monitoring with the two openvpn clients running? I did.

Yes I had issues with the gateway monitoring… then it would not connect at all or would be erratic. I eventually notice in the openvpn log that the ifconfig command would fail and after looking at the route status in diagnostics noticed it had an old dynamic route that should of been deleted. I ended up changing the dynamic ip address of the openvpn server and that seemed to fix the problem, however the old route was still there.

cosmoxl · Oct 30, 2017, 10:28 PM

@RHLinux:

@cosmoxl:

@RHLinux:

Do you have the link to the issue you reported so I can also link to this issue.

Are you using two openvpn clients by any chance? I have two separate VPN Clients that are routed to two different VLANs.

I brought up the problems I was seeing in a thread in the 2.4 development subforum. I didn't submit an official bug report.

When testing I was using two different openvpn clients but I have no VLANs created.

Do you see problems with gateway monitoring with the two openvpn clients running? I did.

Yes I had issues with the gateway monitoring… then it would not connect at all or would be erratic. I eventually notice in the openvpn log that the ifconfig command would fail and after looking at the route status in diagnostics noticed it had an old dynamic route that should of been deleted. I ended up changing the dynamic ip address of the openvpn server and that seemed to fix the problem, however the old route was still there.

Yep, at the time I posted everybody insisted that all was good. Nobody else reported problems. I feel a little vindicated now. I won't upgrade to 2.4.x until this problem is fixed.

stormforge · Nov 1, 2017, 5:06 PM

I'm having maybe the same problem with the 2.4.1 upgrade (from 2.3.4-RELEASE-P1). Two sites connected by OpenVPN, each with multi-WAN and Multi-WAN, spanned by OpenVPN. Both sites have a road warrior VPN as well although one is for backup admin access.

VPNs still connects but with the following issues:

site 1 primary LAN traffic for site 2 primary LAN doesn't go down the tunnel despite the routing tables being correct. It's routed out the current WAN interface. Site 2 primary LAN traffic for site 1 passes normally.
road warrior traffic for site 1 appears to work normally with the caveat of #1 (traffic to site 2 isn't sent down the VPN tunnel despite the routing tables).
road warrior traffic for site 2 only reaches the firewall and goes no further.

Complicating matters is that my primary development platform, an ultrabook, is out for warranty repair again because it doesn't power on, I can't access my image backups that are in the file system backup for that from the temporary development platform although I've got config backups handy locally and with autoconfigbackup, these as on Soekris 6501's (Soekris went belly up with means I'm down to dregs for spares), and these 32-bit nano images on USB.

Worse yet, Netgate seems to oddly restrict legacy downloads of an opensource project (see comment above about access to backups; I've got basically all versions there).

And I'm remote. Previously, I had redundant firewalls at both sites but the client made a bad cost cutting move to drop to one (large retail).

Anyone know where legacy downloads are?

stormforge · Nov 1, 2017, 5:17 PM

"I'm having maybe the same problem with the 2.4.1 upgrade (from 2.3.4-RELEASE-P1). "

Correction. Upgrade to 2.3.4-RELEASE-P1 from 2.3.3.

immy · Nov 4, 2017, 10:05 AM

I am also having the problem of an old route that doesn't get deleted (also on 2.4.1)

/sbin/ifconfig ovpnc1 10.4.6.92 10.4.0.1 mtu 1500 netmask 255.255.0.0 up
ifconfig fails error 1

From netstat:
10.4.0.1 10.4.3.89 UGHS lo0

Trying manually:

/sbin/ifconfig ovpnc1 10.4.6.92 10.4.0.1 mtu 1500 netmask 255.255.0.0 up
ifconfig: ioctl (SIOCAIFADDR): File exists

Trying to delete the old route:

route del 10.4.0.1 10.4.3.89
route: writing to routing socket: Address already in use
del host 10.4.0.1: gateway 10.4.3.89 fib 0: gateway uses the same route

If I reboot or flush the routes it will work again until my ADSL goes down.

cosmoxl · Nov 4, 2017, 3:12 PM

When will the devs pay attention to this?

RHLinux · Nov 12, 2017, 2:26 PM

Who knows… It's defiantly still an issue with multi client/multi WAN.

RHLinux

jimp · Nov 13, 2017, 2:49 PM

@cosmoxl:

When will the devs pay attention to this?

When we have some solid evidence other than vague reports that it's broken with barely any detail. I have probably 30-40 OpenVPN instances between my lab and live boxes, a mix of clients, servers, multi-WAN, routing protocols, you name it. I have yet to see this happen. About the only thing I don't have is a connection to an external VPN provider. I have simulated one internally, but I don't subscribe to any public ones.

We're going to need a lot more detail about the specific configurations in play here on both sides when it happens, more details from the logs, OS routing tables, anything that might be relevant.

cosmoxl · Nov 14, 2017, 1:29 AM

@jimp:

@cosmoxl:

When will the devs pay attention to this?

When we have some solid evidence other than vague reports that it's broken with barely any detail. I have probably 30-40 OpenVPN instances between my lab and live boxes, a mix of clients, servers, multi-WAN, routing protocols, you name it. I have yet to see this happen. About the only thing I don't have is a connection to an external VPN provider. I have simulated one internally, but I don't subscribe to any public ones.

We're going to need a lot more detail about the specific configurations in play here on both sides when it happens, more details from the logs, OS routing tables, anything that might be relevant.

OK. In another thread user gave me the tip that solves the problem - that is to stop specifying a different IP address to ping for gateway monitoring. That fixed the problem for me.

Perhaps that helps you understand the problem.

amires · Nov 14, 2017, 8:32 AM

I have this problem too. Since upgrading to pfSense 2.4.x my OpenVPN client stays connected until a change in WAN is detected (cable disconnect, IP change, etc) after that it starts giving out ifconfig errors and it wont connect anymore unless I reboot my pfsense box. The problem seems to be VPN routes are not being deleted when vpn goes down. I went back to 2.3.4 and the problem is gone. I will stay on 2.3.4 until this is fixed.

amires · Nov 14, 2017, 8:31 AM

@cosmoxl:

OK. In another thread user gave me the tip that solves the problem - that is to stop specifying a different IP address to ping for gateway monitoring. That fixed the problem for me.

Perhaps that helps you understand the problem.

What different IP should we use? I am using 10.4.0.1 which is gateway of AirVPN provider.

jimp · Nov 14, 2017, 1:26 PM

If setting a monitor IP address on the VPN gateway triggered it, then that makes me wonder if it's related to how the routes are added.

Anyone having the issue, check for:

1. Is the VPN interface assigned/enabled under the Interfaces menu?
2. Does the VPN gateway have an alternate monitoring IP address?
3. Is there a DNS server set to use the VPN gateway?
4. Are there any manually-defined static routes set to the use VPN gateway? (there should never be, but some people add them not realizing they are a problem)
5. Any dynamic routing protocols using the VPN?

jarrad · Nov 14, 2017, 3:02 PM

1. Is the VPN interface assigned/enabled under the Interfaces menu? - Yes. Removed the interface and no change. Readded interface, no change.
2. Does the VPN gateway have an alternate monitoring IP address? - It did (it was one of the client expected IP addresses). Removed the alternate monitoring IP, no change.
3. Is there a DNS server set to use the VPN gateway? - No
4. Are there any manually-defined static routes set to the use VPN gateway? (there should never be, but some people add them not realizing they are a problem) - No
5. Any dynamic routing protocols using the VPN? - No

I do have an OpenVPN client (to my VPN provider) running alongside and separate to the OpenVPN server.

jimp · Nov 14, 2017, 3:33 PM

Even if I try to intentionally break it that way, it still restarts fine for me. I do see a route hanging around for the gateway monitor IP address but it doesn't prevent anything from working, and when the VPN restarts it goes right back online. No errors.

I am using 2.4.2 to test this, however. Maybe something else changed there along the way that fixed the problem. Either that or I still don't have enough detail to reproduce the original issue.

amires · Nov 14, 2017, 7:46 PM

1. Is the VPN interface assigned/enabled under the Interfaces menu? Yes.
2. Does the VPN gateway have an alternate monitoring IP address? I have to check this.
3. Is there a DNS server set to use the VPN gateway? Yes, same as VPN gateway.
4. Are there any manually-defined static routes set to the use VPN gateway? No.
5. Any dynamic routing protocols using the VPN? No.

I am using AirVPN. My VPN gateway, DNS and Monitoring IP are all 10.4.0.1.

jimp · Nov 14, 2017, 7:50 PM

@amires:

3. Is there a DNS server set to use the VPN gateway? Yes, same as VPN gateway.
I am using AirVPN. My VPN gateway, DNS and Monitoring IP are all 10.4.0.1.

On System > General Setup, do not set a gateway on the DNS server there. Just leave the entry for 10.4.0.1 and set the gateway to "none". You're making a redundant route by doing that, which could be part of the problem. If 10.4.0.1 is your VPN gateway then you'll already have a route there. Selecting that on the DNS server list is completely unnecessary. We prevent that for static IP interfaces where the network is known, but since those VPN interfaces are dynamic it isn't prevented.

You're lucky that didn't cause a problem in the past.

amires · Nov 14, 2017, 8:26 PM

@jimp:

@amires:

3. Is there a DNS server set to use the VPN gateway? Yes, same as VPN gateway.
I am using AirVPN. My VPN gateway, DNS and Monitoring IP are all 10.4.0.1.

On System > General Setup, do not set a gateway on the DNS server there. Just leave the entry for 10.4.0.1 and set the gateway to "none". You're making a redundant route by doing that, which could be part of the problem. If 10.4.0.1 is your VPN gateway then you'll already have a route there. Selecting that on the DNS server list is completely unnecessary. We prevent that for static IP interfaces where the network is known, but since those VPN interfaces are dynamic it isn't prevented.

You're lucky that didn't cause a problem in the past.

Actually I have tried it both without a gateway and with VPN gateway and none did help. I just removed the gateway just to be safe.
I also removed 10.4.0.1 as monitoring ip and used 8.8.8.8 instead. I am waiting to see if the problem still exist.