pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss

Gertjan

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

but in your 2.5.2 configuration you have it checked (now called "Data Encryption Negotiation").

Because (see above) :

Data Encryption Negotiation - <checked>

It's checked because "openvpn 2.5.2 needs (like mandatory) it.

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

The ExpressVPN setup instructions for 2.4.5 say to check "Don't pull routes", but you haven't.

?? Correct - both are non checked.
( probably because the server (ExpressVPN) pushed them ).

About the images that show the red 'offine' : use 8.8.8.8 as the monitoring IP : the IP's used (your IPs) do not reply to ping, so there will be a red "offline, even when the connection is online.
This one :

Can't find this one in your images ?

You have pfSense 2.5.2, right ? I can't help you with the old OpenVPN 2.4.8 used by pfSense 2.4.5p1.
The OpenVPN versions used are different. OpenVPN went from 2.4.8 to 2.5.2 Several options changed ( and that is why this subject exists ). Some minor parameter changes were needed to make it work again.

Btw : your AP should be an AP. Nothing else. Don't leave the DHCP server on Auto, shut it down ( off ). ( because : scenario : powerdown -> powerup : IF the AP boots up earlier as pfSense, the AP server will activate - moments later the sense DHCP servers activates : You've created a whole new issue)
The AP is not part of the issue right now. And if doubt, remove the AP from the equation - use a straight cable, as cables are easy to debug ;)

amdreallyfast

@gertjan

About the images that show the red 'offine' : use 8.8.8.8 as the monitoring IP

Done.

BTW, the WAN's "Monitor IP" is set to itself. Is that bad? I didn't set it as such, so it must be the default.

You have pfSense 2.5.2, right ?

I downgraded to 2.4.5 (via format -> fresh install) because I've been trying to replicate the conditions that the ExpressVPN setup instructions expect. If I update to 2.5.2, then I won't be able to go back, but since you were able to get ExpressVPN working on 2.5.2, then I hope that the upgrade shouldn't be an issue. I'll do that tonight.

Btw : your AP should be an AP. Nothing else.

Done. Turned it off. At least that's out of the way.

BTW, I'm hesitant to remove the AP entirely because then I won't have wireless. My router box only has to two ethernet ports. I can't run my PC to the router and still have wireless internet. If the issues continue to persist, I'll try doing it. In hindsight, I should have purchased a mini PC with 4 network ports, but I didn't think of that at the time.

amdreallyfast

@gertjan

Well, after upgrading to 2.5.2 and explicitly turning my TP-Link DHCP server off (from Auto) and restarting everything, the gateways are now in indefinite "Pending" mode. I'm out of ideas. Yourself?

Gertjan

@amdreallyfast

How many Expr*ssVPN clients have you running ?

Concentrate one one, then add others if needed.
I know Expr**sVPN offers 5 ( ?) simulations connections - dono if they accept 5 from the same device.

Now you have 2.5.2, get your client opvn file and post it here. I'll compare yours with mine.

amdreallyfast

@gertjan

Alright, I'll disable two of the interfaces and the OpenVPN clients that use them.

While waiting on the gateways to work themselves out, here's my config.ovpn file, unaltered in any way. It looks like upgrading to 2.5.2 changed how the config files were stored, and now I have the file structure that you had expected in prior posts.

Gertjan

@amdreallyfast

I compared you opvn file with mine.
There is 1 ( one ) difference :
local 192.168.10.3
for me, as me WAN IP is 192.168.10.3 - I have an up stream ISP router.
You have a 'real' WAN IP 68.100.115.1x9 - That must be the IP of your WAN interface.

My opvn file :

dev ovpnc2
verb 3
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_client2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA512
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.10.3
tls-client
client
lport 0
management /var/etc/openvpn/client2/sock unix
remote usa-sanfrancisco-ca-version-2.expressnetw.com 1195 udp4
auth-user-pass /var/etc/openvpn/client2/up
auth-retry nointeract
capath /var/etc/openvpn/client2/ca
cert /var/etc/openvpn/client2/cert 
key /var/etc/openvpn/client2/key 
tls-auth /var/etc/openvpn/client2/tls-auth 1
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression asym
comp-lzo adaptive
resolv-retry infinite
fast-io
sndbuf 524288
rcvbuf 524288
fast-io

remote-random
pull
comp-lzo
verify-x509-name Server name-prefix
remote-cert-tls server
key-direction 1
route-method exe
route-delay 2
tun-mtu 1500
fragment 1300
mssfix 1450

You'll see :
comp-lzo adaptive
and then
comp-lzo
in the custom option box.

The latter cancels / modifies the first, I guess.

Do not forget to add the ";".

Now, when I remove the check from 'disable' at the top, and "save" I have these logs :
Logs are in reverse order : the connection sequence starts at the bottom :

2022-01-11 08:40:16.087396+01:00 	openvpn 	73688 	Initialization Sequence Completed
2022-01-11 08:40:16.087382+01:00 	openvpn 	73688 	WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2022-01-11 08:40:16.086402+01:00 	openvpn 	73688 	/sbin/route add -net 10.46.0.1 10.46.0.133 255.255.255.255
2022-01-11 08:40:16.085418+01:00 	openvpn 	73688 	/sbin/route add -net 128.0.0.0 10.46.0.133 128.0.0.0
2022-01-11 08:40:16.084447+01:00 	openvpn 	73688 	/sbin/route add -net 0.0.0.0 10.46.0.133 128.0.0.0
2022-01-11 08:40:16.083229+01:00 	openvpn 	73688 	/sbin/route add -net 45.38.178.184 192.168.10.1 255.255.255.255
2022-01-11 08:40:14.426692+01:00 	openvpn 	73688 	/usr/local/sbin/ovpn-linkup ovpnc2 1500 1557 10.46.0.134 10.46.0.133 init
2022-01-11 08:40:14.421989+01:00 	openvpn 	73688 	/sbin/ifconfig ovpnc2 10.46.0.134 10.46.0.133 mtu 1500 netmask 255.255.255.255 up
2022-01-11 08:40:14.421967+01:00 	openvpn 	73688 	TUN/TAP device /dev/tun2 opened
2022-01-11 08:40:14.421881+01:00 	openvpn 	73688 	TUN/TAP device ovpnc2 exists previously, keep at program end
2022-01-11 08:40:14.421846+01:00 	openvpn 	73688 	ROUTE_GATEWAY 192.168.10.1/255.255.255.0 IFACE=em0 HWADDR=00:15:17:77:d1:19
2022-01-11 08:40:14.421717+01:00 	openvpn 	73688 	Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
2022-01-11 08:40:14.421688+01:00 	openvpn 	73688 	Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
2022-01-11 08:40:14.421585+01:00 	openvpn 	73688 	Data Channel: using negotiated cipher 'AES-256-GCM'
2022-01-11 08:40:14.421576+01:00 	openvpn 	73688 	OPTIONS IMPORT: data channel crypto options modified
2022-01-11 08:40:14.421559+01:00 	openvpn 	73688 	OPTIONS IMPORT: adjusting link_mtu to 1629
2022-01-11 08:40:14.421547+01:00 	openvpn 	73688 	OPTIONS IMPORT: peer-id set
2022-01-11 08:40:14.421535+01:00 	openvpn 	73688 	OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
2022-01-11 08:40:14.421524+01:00 	openvpn 	73688 	OPTIONS IMPORT: route options modified
2022-01-11 08:40:14.421512+01:00 	openvpn 	73688 	OPTIONS IMPORT: --ifconfig/up options modified
2022-01-11 08:40:14.421496+01:00 	openvpn 	73688 	OPTIONS IMPORT: compression parms modified
2022-01-11 08:40:14.421479+01:00 	openvpn 	73688 	OPTIONS IMPORT: timers and/or timeouts modified
2022-01-11 08:40:14.421389+01:00 	openvpn 	73688 	PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 10.46.0.1,comp-lzo no,route 10.46.0.1,topology net30,ping 10,ping-restart 60,ifconfig 10.46.0.134 10.46.0.133,peer-id 56,cipher AES-256-GCM'
2022-01-11 08:40:14.260127+01:00 	openvpn 	73688 	SENT CONTROL [Server-10448-0a]: 'PUSH_REQUEST' (status=1)
2022-01-11 08:40:13.011214+01:00 	openvpn 	73688 	[Server-10448-0a] Peer Connection Initiated with [AF_INET]45.38.178.184:1195
2022-01-11 08:40:13.011195+01:00 	openvpn 	73688 	Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
2022-01-11 08:40:12.834914+01:00 	openvpn 	73688 	VERIFY OK: depth=0, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10448-0a, emailAddress=support@expressvpn.com
2022-01-11 08:40:12.834906+01:00 	openvpn 	73688 	VERIFY X509NAME OK: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10448-0a, emailAddress=support@expressvpn.com
2022-01-11 08:40:12.834897+01:00 	openvpn 	73688 	VERIFY EKU OK
2022-01-11 08:40:12.834887+01:00 	openvpn 	73688 	++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
2022-01-11 08:40:12.834874+01:00 	openvpn 	73688 	Validating certificate extended key usage
2022-01-11 08:40:12.834861+01:00 	openvpn 	73688 	VERIFY KU OK
2022-01-11 08:40:12.834393+01:00 	openvpn 	73688 	VERIFY OK: depth=1, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=ExpressVPN CA, emailAddress=support@expressvpn.com
2022-01-11 08:40:12.834281+01:00 	openvpn 	73688 	VERIFY WARNING: depth=1, unable to get certificate CRL: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=ExpressVPN CA, emailAddress=support@expressvpn.com
2022-01-11 08:40:12.834232+01:00 	openvpn 	73688 	VERIFY WARNING: depth=0, unable to get certificate CRL: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10448-0a, emailAddress=support@expressvpn.com
2022-01-11 08:40:12.669575+01:00 	openvpn 	73688 	TLS: Initial packet from [AF_INET]45.38.178.184:1195, sid=afc6b917 2da3d51c
2022-01-11 08:40:12.507340+01:00 	openvpn 	73688 	UDPv4 link remote: [AF_INET]45.38.178.184:1195
2022-01-11 08:40:12.507290+01:00 	openvpn 	73688 	UDPv4 link local (bound): [AF_INET]192.168.10.3:0
2022-01-11 08:40:12.507235+01:00 	openvpn 	73688 	Socket Buffers: R=[42080->524288] S=[57344->524288]
2022-01-11 08:40:12.507137+01:00 	openvpn 	73688 	TCP/UDP: Preserving recently used remote address: [AF_INET]45.38.178.184:1195
2022-01-11 08:40:12.428036+01:00 	openvpn 	73688 	Incoming Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
2022-01-11 08:40:12.428014+01:00 	openvpn 	73688 	Outgoing Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
2022-01-11 08:40:12.427759+01:00 	openvpn 	73688 	WARNING: experimental option --capath /var/etc/openvpn/client2/ca
2022-01-11 08:40:12.426711+01:00 	openvpn 	73688 	NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2022-01-11 08:40:12.426487+01:00 	openvpn 	73688 	MANAGEMENT: unix domain socket listening on /var/etc/openvpn/client2/sock
2022-01-11 08:40:12.425588+01:00 	openvpn 	73485 	library versions: OpenSSL 1.1.1k-freebsd 25 Mar 2021, LZO 2.10
2022-01-11 08:40:12.425575+01:00 	openvpn 	73485 	OpenVPN 2.5.2 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Jun 24 2021
2022-01-11 08:40:12.425550+01:00 	openvpn 	73485 	WARNING: file '/var/etc/openvpn/client2/up' is group or others accessible
2022-01-11 08:40:12.425218+01:00 	openvpn 	73485 	WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.

There are some warnings about "Compression" - Not important right now (?).
WARNING: experimental option : nice, we want to experiment.
two "VERIFY WARNING" for the CRL : we don't care.
And a "WARNING: this configuration may cache" because we're lazy, so ok.

Do you have exactly the same log lines ?
( using "verb 3" means log level 3 - the default )

amdreallyfast

@gertjan
I changed my custom options to match yours, and then looked at Status -> System Logs -> OpenVPN.

Noticing AUTH_FAILED. That's clearly bad.

Note: This is just one AUTH_FAILED cycle. The full logs repeat this log segment many times.

Jan 11 18:09:57	openvpn	56223	NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jan 11 18:09:57	openvpn	56223	Outgoing Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Jan 11 18:09:57	openvpn	56223	Incoming Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Jan 11 18:09:57	openvpn	56223	TCP/UDP: Preserving recently used remote address: [AF_INET]45.38.178.54:1195
Jan 11 18:09:57	openvpn	56223	Socket Buffers: R=[42080->524288] S=[57344->524288]
Jan 11 18:09:57	openvpn	56223	UDPv4 link local (bound): [AF_INET]68.100.115.149:0
Jan 11 18:09:57	openvpn	56223	UDPv4 link remote: [AF_INET]45.38.178.54:1195
Jan 11 18:09:57	openvpn	56223	TLS: Initial packet from [AF_INET]45.38.178.54:1195, sid=6cbc8ab5 ec9e0d58
Jan 11 18:09:57	openvpn	56223	VERIFY WARNING: depth=0, unable to get certificate CRL: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10457-0a, emailAddress=support@expressvpn.com
Jan 11 18:09:57	openvpn	56223	VERIFY WARNING: depth=1, unable to get certificate CRL: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=ExpressVPN CA, emailAddress=support@expressvpn.com
Jan 11 18:09:57	openvpn	56223	VERIFY OK: depth=1, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=ExpressVPN CA, emailAddress=support@expressvpn.com
Jan 11 18:09:57	openvpn	56223	VERIFY KU OK
Jan 11 18:09:57	openvpn	56223	Validating certificate extended key usage
Jan 11 18:09:57	openvpn	56223	++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Jan 11 18:09:57	openvpn	56223	VERIFY EKU OK
Jan 11 18:09:57	openvpn	56223	VERIFY X509NAME OK: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10457-0a, emailAddress=support@expressvpn.com
Jan 11 18:09:57	openvpn	56223	VERIFY OK: depth=0, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10457-0a, emailAddress=support@expressvpn.com
Jan 11 18:09:57	openvpn	56223	Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
Jan 11 18:09:57	openvpn	56223	[Server-10457-0a] Peer Connection Initiated with [AF_INET]45.38.178.54:1195
Jan 11 18:09:58	openvpn	56223	SENT CONTROL [Server-10457-0a]: 'PUSH_REQUEST' (status=1)
Jan 11 18:09:58	openvpn	56223	AUTH: Received control message: AUTH_FAILED
Jan 11 18:09:58	openvpn	56223	SIGUSR1[soft,auth-failure] received, process restarting
Jan 11 18:09:58	openvpn	56223	Restart pause, 10 second(s)

And the gateway is still pending:

Gertjan

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

This is just one AUTH_FAILED cycle

That's a matter of copying correctly your credentials.
Was this message there all the time ?
This is the most logical reason the connection doesn't come up : your bank site won't let you in neither with wrong credentials..

This issue wasn't worth mentioning, as it is easy to correct.

edit : in the folder where you found the opvn client file, you will also find a file called 'up'.
Open it ;)
This explains this setting :

auth-user-pass /var/etc/openvpn/client1/up

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

And the gateway is still pending

When the login is transmitted, and it's wrong, the "PUSH_REQUEST" will not receive the route, gateway, IP etc but an error message : you saw it.
The gateway isn't reachable, as you failed authentication.
When the "auth" went well, you receive a

PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 10.46.0.1,comp-lzo no,route 10.46.0.1,topology net30,ping 10,ping-restart 60,ifconfig 10.46.0.134 10.46.0.133,peer-id 56,cipher AES-256-GCM'
2022-01-11 08:40:14.260127+01:00 openvpn 73688 SENT CONTROL [Server-10448-0a]: 'PUSH_REQUEST' (status=1)

and as you can see, the gateway, amongst others, is in there. pfSense uses all this info to set up everything.

amdreallyfast

@gertjan

Looks like it's working now. The auth problem was caused by an old username/password that went bad when the subscription recently expired. I just renewed it, entered the new username/password, and now the gateway is online.

Then I enabled the firewall rules and firewall NAT -> Outbound routing rules, and it looks like its my San Francisco VPN is now working and routing traffic correctly. I'm writing this post while my router is connected to the new VPN.

It looks like your settings worked for me too. I remember deviating from the ExpressVPN setup instructions in the VPN client setup by checking Enable NCP (2.4.5; on 2.5.2 it's called "Enable Data Encryption Negotiation") and by unchecking Don't pull routes", and by unchecking both Interfaces -> <vpn interface name> -> Reserved Networks -> Block private networks and loopback addresses and Block bogon networks. And I think that I edited the manual custom configuration options a bit so that they match yours. Current settings.

remote-random;
pull;
comp-lzo;
verify-x509-name Server name-prefix;
remote-cert-tls server;
key-direction 1;
route-method exe;
route-delay 2;
tun-mtu 1500;
fragment 1300;
mssfix 1450;
verb 3;
sndbuf 524288;
rcvbuf 524288;

Thanks for all the patience. Next I'll need to figure out custom routing rules for connections that don't like the VPN. Some places block it and I'll need to make exceptions for those.

Gertjan

@amdreallyfast
Sit back, and think with me :
What is different between these 'famous' Exprr*VPN instructions, and the ones needed to make VPN works with 2.5.2.
You'll agree with me : it's not much. Re see the post about this subject I made several month ( ? ) ago.
I'll have a look at it this weekend.

amdreallyfast

@gertjan

You'll agree with me : it's not much.

True. The problem is that network software rarely settles for "close enough". Figuring out exactly what I needed to change was the rough part.

And now, curiously enough, the docs pages for pfsense won't load. It's like my firewall (or maybe theirs) is blocking me. Back to the non-VPN configuration while I try to figure out how to pass traffic for certain websites through the non-VPN connection.

elmo1943

I had expressvpn running on 2.5.2 but it was very slow (SFO from Okinawa, Japan) also I could not change from SFO with ease. I finally installed expressvpn firmware on old external router. Tried to connect via pfsense vlan, many times, finally expressvpn help desk told me that expressvpn would not work via a vlan. Reinstalled pfsense, reinistalled pfblocker. Changed to opt1_dmz, configured new gateway and added new static address. Than I changed router from PPPoE to DHCP and expressvpn worked fine, in fact speed testing is as fast as non-vpn speeds and all expressvpn options are available. I use expressvpn for USA paid networks (prime, netflix, etc.) via wifi inside the house. wifi from expressvpn router. Your luck may be different. elmo

Txbongo

@Gertjan

Thank you, the monitor IP (8.8.8.8) and compression is what I needed to make mine work!