v21.02 Broke all ExpressVPN Gateways

thaddeusf

I wonder if it's possible the 'Serviced Watchdog' is causing the problem? Testing.

thaddeusf

Nothing has worked. I've just now completely reset the NetGate and starting from scratch - not even restoring from backup.

Have been working on it for hours because of the complexity of my static DHCP addressing.

I cannot BELIEVE how much crap this upgrade has caused. I do NOT recommend doing it.

I just hope the VPN Gateways work once I finishing rebuilding everything.

thaddeusf

@thaddeusf

I've flashed the system to '21.02-RELEASE-p1' and while the system is behaving better (and much faster), the Gateway problem still exists.

I've restored from 2.4 backup and turned Gateway Monitoring off on all ExpressVPNs and I seem to be back up working as before the upgrade.

Can anyone explain the specifics of disabling Gateway Monitoring?

This have been a very rough experience.

Thanks.

AB5G

+1 on this with ExpressVPN. I've been rocking ExpressVPN with a single tunnel on pfsense 2.5 from Jan . Yesterday night, I cloned the existing config and added another tunnel to a different geo. The gateway monitoring widget wasn't showing the gateway up so I added 1.1.1.1 as the monitoring IP through the tunnel - with this the gateway came up and with the Policy based routing my IOT devices got working.

Somewhere in the night, the entire network started to behave slow (main LAN and IOT). I thought it was the DNS so I restarted unbound. Didn't help, then I thought it was the ISP and went to bed. Woke up in the morning and the same issue. I traced my steps back - deleted the new tunnel, interface and the gateway and things were back to normal.

I have some openvpn logs if someone wants to look through.

thaddeusf

After flashing to 21-02-RELEASE-p1, it appeared the SG-3100 was working again. However through the night, the system completely fell apart again.

At this point I've removed all VPN configuration and rules. And for some reason the FW is only NATting IPv6 inconsistently and no IPv4.

None of this makes sense and the logs aren't providing much insight.

sigh....

thaddeusf

I've once again done a complete fresh install.

If I set it up as a pricey Linksys router, it seems to be stable.

I don't yet know what specific configuration causes it to fall over.

bod

Yep - With Gateway monitoring enabled on my Private Internet Access VPN it takes multiple OpenVPN service restarts to get it working. Anything from 1 to 10.

viktor_g

Try to install the System Patches pkg https://docs.netgate.com/pfsense/en/latest/development/system-patches.html
and apply Patch ID 969574b6dbb124e98595ca537c0d176d908707d0

See https://redmine.pfsense.org/issues/11448

bod

Thanks - didn't resolve the issue with Gateway Action. If I have Gateway Action enabled it fails to connect randomly.

thaddeusf

@viktor_g Unfortunately this does not fix the problem

Additionally I've manually loaded the identical configuration on an x86 pfsense system and it has the same problem of failed gateway monitoring.

Please prioritize - I cannot imaging this is not effecting quite a number of upgraders.

thaddeusf

I have done another completely clean install and this time it is the 2.5 pfsense community edition on a ProtectLI.

The bottom line is that I can create an ExpressVPN interface but the Gateway to the interface if marked 'Offline, Packetloss: 100%"

A monitor IP is listed and is the same as the Gateway IP.

If I turn Gateway monitoring off (system/routing/gateways) in the configuration of the ExpressVPN interface, the entire firewall stops passing traffic.

This is not precisely the same behavior I saw on the SG-3100, but it's close. The 3100 would start passing traffic with gateway monitoring off, but the performance would tank.

This is pretty obviously a bug. Is there additional information (logs, etc) I can provide?

bcruze

https://openvpn.net/community-downloads/

Under important notices states it pretty clear

The service providers are at fault not Netgate

thaddeusf

@bcruze
The only thing I am complaining about is that Netgate is not making 2.4.5 available for the SG-3100 when I specifically requested it.

It was my own damn fault for rashly upgrading it.

What could possibly be a valid reason not making it available?

bod

Well looks like I found the solution with regards to Private Internet Access. A user has found that using the config below will work. I have even turned gateway monitoring on and multiple service restarts and it works everytime.

https://www.privateinternetaccess.com/forum/discussion/38389/aes-256-gcm-with-asus-wrt-merlin-386-1-final-on-pia#latest

Force AES-256-GCM
Put in "NCP-disable" in Custom Options.

Will let you know tomorrow if it's all still going well.

bcruze

@bod said in v21.02 Broke all ExpressVPN Gateways:

Well looks like I found the solution with regards to Private Internet Access. A user has found that using the config below will work. I have even turned gateway monitoring on and multiple service restarts and it works everytime.

https://www.privateinternetaccess.com/forum/discussion/38389/aes-256-gcm-with-asus-wrt-merlin-386-1-final-on-pia#latest

Force AES-256-GCM
Put in "NCP-disable" in Custom Options.

Will let you know tomorrow if it's all still going well.

its been posted a few times it will work. then speeds will go to a grinding halt. my previous provider would also work for a while. but if I restarted the tunnel. or rebooted the router it stopped working again. please let us know either way

thaddeusf

@bod

With my configuration, I see:

GCM only does not successfully connect.
If I include 'ncp-disable' in the custom, it seems to stop OpenVPN at load (which is confusing, but I've not had time to research)

Really interested to hear your results and compare.

I'm going to try to switch to L2TP/IPsec until this can be resolved.

ALSO, Netgate finally sent me the link to 2.4.5 for the SG-3100. While continuing research on the ProtectLI, I'm going to repair the ARM appliance.

thaddeusf

I've made some progress today.

I've been able to configure an L2TP/IPSec interface (with gateways) to the ExpressVPN server. It work as without any problem, but the performance is significantly slower not to mention with compromised security.
I've been able to disable Gateway Monitoring on the OpenVPN to ExpressVPN interface. Apparently my previous attempts which proved unstable were caused by the interfaces being load balanced.

I've opened a ticket with ExpressVPN as well. Hopefully we'll make some progress.

bod

Spoke too soon. The Watchdog service (Action) broke it.

No issues with throughput on AES-256-GCM. Still getting sustained 40mbps.

thaddeusf

This post is deleted!

guido666

I'm having a similar/related issue. After upgrading to 2.5, my traffic does not get routed out the VPN gateway. It hits the proper firewall rule, and gets passed, but it just doesn't go out the VPN gateway and instead goes out the WAN gateway.

The VPN gateway (in Status) shows as "Offline, Packetloss: 100%".

If I disable Gateway monitoring (System > Routing > Gateways) for the VPN gateway, the traffic instantly starts going through the VPN gateway correctly again (matching the same firewall rule).

If related, I also get the 3 errors mentioned previously.
Mar 11 01:03:28 openvpn 49069 Options error: option 'route' cannot be used in this context ([PUSH-OPTIONS])
Mar 11 01:03:28 openvpn 49069 Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Mar 11 01:03:28 openvpn 49069 Options error: option 'redirect-gateway' cannot be used in this context ([PUSH-OPTIONS])