pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss

[amdreallyfast]

@gertjan

See 2 forum messages back, it's already there.

Are you sure? Two messages (of yours) back I see this post, and I'm not seeing a config file. Would you take a screenshot of the message (and link it if it's another forum post)?

[amdreallyfast]

@gertjan

I could ping google.com just fine (it connected).

Same. The gateway is offline with 100% packet loss, and yet the Diagnostics -> Ping tool could use that VPN client to ping. This tells me that the gateway being offline with 100% packet loss is not an issue that can be checked with the Ping tool. But how can I diagnose it then?

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

And I say "maybe" because ExpressVPN was apparently able to create .ovpn files

Expr*ssVP always exported 'ovpn' files.
These are just 'plain text' files, filled with process command line options that the process openvpn understands.
pfSense 2.5.2 uses the same format and extension 'internally'.
That is because pfSense is just an OS (sort of), with a nice GUI, and for the rest it uses the same processes (binary programs) as all the other MAC based / Window based / FreeBSD based / Lines based systems.

Dono what pfSense 2.4.5 did, that's to far in the past for me.

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

Did you have to install a specific package in order to export the config file?

Exporting what to what ?
To export a 'setup' (config opvn file + binaries) or just the opvn file from an OpenVPN Access server (server !!) then you need this

package.
See the pfSense manual or one of a thousands of OpenVPN video's, book, whatever.

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

Are you sure? Two messages (of yours) back I see this post, and I'm not seeing a config file. Would you take a screenshot of the message (and link it if it's another forum post)?

Here :

That an image from this thread.
You can't see it ?

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

This tells me that the gateway being offline with 100% packet loss is not an issue that can be checked with the Ping tool. But how can I diagnose it then?

Because some one - in your case 10.173.0.137 - doesn't answer to 'ping' this doesn't mean the gateway is offline.
What this implies is :
The path to host 10.173.0.137 doesn't exist
or
It doesn't reply to ping.
or ....

When I empty this field :

the pfSense montoring system will auto choose an IP, like your "10.173.0.137".

And I see the same thing : no ping replies. So it says 'offline'. Or, like you, I can ping 'google' just fine.
That why I fill in a IP that I "have /own / use" and that is close to the Exprss VPN exit WAN IP : I have a dedicated server near by. Or you could use what everybody does : use 8.8.8.8 as a monitoring IP.
And your ExpressVPN is now (nope, this should be : "looks like") on-line.

[amdreallyfast]

@gertjan

Expr*ssVP always exported 'ovpn' files.

...Then why can't I? This is weird. I previously mentioned that I've tried SF, LA, and DC, so I tried turning on all their VPN clients and interfaces, wondering if that might cause it to create a client1 folder to separate the three clients. Nope. My ... folder now looks like this. I don't know why OpenVPN is not creating .ovpn files. Any ideas?

That an image from this thread.
You can't see it ?

Oh. That's your file? I was looking for a file attachment (as opposed to copy-pasted values), but now I have questions about this (see next question).

I had to add these missing :
<several lines of custom options>
Now my config.opvn file looks like this :
<more lines of custom options>

Which one is your file? You said that you had to add some values, and then you say, "my config.ovpn file looks like this" and you show a list of custom options that does not include the ones that you just said you had to add. Now I'm confused about what your file looks like. Are you sure that's the entirety of the .ovpn file and that the file does not include the items that you had just added?

When I empty this field :
<screenshot of Monitor IP>
the pfSense montoring system will auto choose an IP, like your "10.173.0.137".
And I see the same thing : no ping replies. So it says 'offline'.

I don't think that we're seeing the same thing. The Monitor IP field is blank for all three of my VPN client configurations (SF, LA, DC), and all three of them do show ping replies while the gateway is offline. Please explain. Are you saying that I should leave Monitor IP blank or that I should not leave it blank?

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

Then why can't I?

Because you strugle with the wrong questions ;)

There is no such thing as exporting a "OpenVPN client" config file.

A OpenVPN server installation you have set up, benefits from the pfSense "openvpn-client-export" package. The file you exports comes from the OpenVPN (server parameters) and are used by the future OpenVPN clients, so they can access the (your) OpenVPN server for remote access.

Your subject is all about the OpenVPN client set up.
The OpenVPN server - the provider, in this case - is ExpressVPN.
So it's Express VPN that shares a 'config file' with you, what I call the opvn config file. You downloaded it.
This file has to be used as a guide line to fill in the pfSense GUI client page.
Currently, pfSense does not offer a no way to 'import' the file you got from ExpessVPN.

Now you understand (bitter) why I want you to look at the openvpn client config file that pfSense GUI has created with it's GUI.
This file has to resemble as close as possible as the file you obtained from ExpressVPN.

The
Server
Client specific overrides
Wizard
Client Export
Shared key Export

are related to the OpenVPN server.

The OpenVPN client has only one page : the one that's underlined in my image.

An important aspect is also : the GUI can only handle a subset of the possible options or settings. A GUI is useful for enforcing 'illegal' option combinations.

Got it ?! ;)

Side note : it would be nice to have the possibility to import a server side generated file, so you can set up the client easely.
But there is a thing : settings are version specific. They change all the time.
"openvpn" as a program has close to one thousand settings (options and possible combination of options combined - it's close to impossible to write a GUI to handle them all.

Big (server type) apps, like apache2, nginx, postfix and bind (named) can not be set up using a GUI.
You have to fall back to the classic 'confg files' as the shear number of combinations of settings is just daunting.

Btw : the last part is totally IMHO.

[amdreallyfast]

@gertjan

Because you strugle with the wrong questions ;)

The curse of learning. Thanks for bearing with me through this. Books and tutorials can't correct me when I understand a term badly, but you can. Thanks :).

Now you understand (bitter) why I want you to look at the openvpn client config file that pfSense GUI has created with it's GUI.
This file has to resemble as close as possible as the file you obtained from ExpressVPN.

Alright, so how do I get "the openvpn client config file that pfSense GUI has created with it's GUI"? You walked me through sftp, but that showed that I didn't have a .ovpn file at all. Is there a way to get this .ovpn file through the GUI?

It sounds like you're looking for the Client Specific Overrides. Are you sure? Mine are blank. I have done anything here.

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

It sounds like you're looking for the Client Specific Overrides

A point for you !

I meant :

All the other menu options are 'server' related.

edit :
I restarted reading you first post (see above).
I decided to enter exactly what you've mentioned.
One exception : I removed my password.

https://www.test-domaine.fr/VPN-Client.png

Compare this huge image with your settings.
Double check for differences.
( edit : I found a difference : my verbosity (log) level is 3 - level 10 is far to detailed, create hundreds of entries a second. Leaving it 10 or 3 has no other side effect )

With these settings :

I monitored 8.8.8.8 to check the VPN-Client connection - RTT is a bit high as I'm based in France, St Jose in the US is not really close.

All I had to to to make the connection works, was switching from :

"automatic mode" to "hybrid mode" and save.
I'm now connected the entire company over Expr*ssVPN ...... I wasn't expecting this, as I wanted to focus on making the connection work first, and then go for the outbound natting.

I didn't even add a 'policy' firewall rule on my LAN interface to make the connection work.
It's not activated :

So, first things first : Make your connection work.

Btw : this is the file the OpenVPN client is using right now :

/var/etc/openvpn/client2/config.ovpn

dev ovpnc2
verb 3
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_client2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA512
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.10.3
tls-client
client
lport 0
management /var/etc/openvpn/client2/sock unix
remote usa-sanfrancisco-ca-version-2.expressnetw.com 1195 udp4
auth-user-pass /var/etc/openvpn/client2/up
auth-retry nointeract
capath /var/etc/openvpn/client2/ca
cert /var/etc/openvpn/client2/cert 
key /var/etc/openvpn/client2/key 
tls-auth /var/etc/openvpn/client2/tls-auth 1
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression asym
comp-lzo adaptive
resolv-retry infinite
fast-io
sndbuf 524288
rcvbuf 524288
fast-io

persist-key

persist-tun

remote-random

pull

comp-lzo

tls-client

verify-x509-name Server name-prefix

remote-cert-tls server

key-direction 1

route-method exe

route-delay 2

tun-mtu 1500

fragment 1300

mssfix 1450

verb 3

sndbuf 524288

rcvbuf 524288

You can clearly see de double spaces lines at the bottom, some of them are present twice in the config file.
This is an issue for later, because, with these settings, it works.

I'm using :

[amdreallyfast]

@gertjan

That's a real big screenshot. Kudos on spending the time to piece that monstrosity together.

Noticed some differences:

1. The ExpressVPN setup instructions for 2.4.5 say to leave NCP (Negotiable Cryptographic Parameters) blank, but in your 2.5.2 configuration you have it checked (now called "Data Encryption Negotiation").
   

2. The ExpressVPN setup instructions for 2.4.5 say to check "Don't pull routes", but you haven't.
   

But I followed along anyway and replicated your settings. And the gateways are still offline. I haven't adjusted the firewall rules yet. I'm still trying to get these gateways online.

Note: Prior to making these screenshots, I shortened the names of the VPN clients and their interfaces, so they'll appear a little different than in prior screenshots, but they're still the same ones.

• VPN clients online
  

• Gateways offline
  

• Using ExpressVPN's IP address checker:
  

• Interface settings
  

• Gateway settings
  

• Client settings
  

[amdreallyfast]

@gertjan

At this point I'm thinking that the only way forward is for a network expert who's familiar with PfSense and OpenVPN to sit down at my computer and poke around until they can figure out what's going on. Whatever my problem is, it's not normal. I'm not a network expert, and I don't know how to diagnose this. I've replicate other peoples' working settings, and it isn't working on my end. Here's some more diagnostic info; does it give you any ideas about where to start diagnosing next?

• I can tell from a basic Wireshark scan of my network traffic that TCP failure and retransmission is not uncommon, and PfSense's Status -> System Logs -> Gateways shows me that communication with the gateways is frequently timing out. All gateways are timing out, including the WAN, which is what I'm currently using to get online and talk with you.
  
  These are all from this morning:
  

• It's rather frequent that I try to load a web page, wait several seconds, the page fails to load and the browser says that it can't find the page, and then it automatically retries and succeeds (sometimes needs to retry twice). I don't know what is causing this frequent failure, or if there's anything I can do about it. I've got a 100Mbps internet connection over cable that is usually stable and reliable once connection is actually established (ex: a long download or a big YouTube video doesn't fail once it's started). It shouldn't take 5-10sec to load a google search. The failures (when they happen) only occur when initially loading a page. If it were a random hardware failure (circuitry going bad, loose cable, etc.), I'd expect random disconnects during downloads and streaming large video, but I'm not seeing that.
  
  I've mentioned before that when I use the ExpressVPN desktop app to make a VPN connection, all those connection issues for my PC disappear. Web pages load quickly 100% of the time, no timeouts, no lost connections. That makes me wonder if there's some sort of packet monitoring and sniffing going on after the traffic leaves my network that either slows the transmission enough to time out or stops it entirely. Why would making an encrypted tunnel to ExpressVPN's servers magically make the disconnects go away?

• My connection is this: Cable -> modem -> router (mini PC running PfSense 2.4.5) -> TP-Link Archer A7 (wifi router/switch configured as access point only) -> PC.
  
  Here's the TP-Link Archer A7 set in AP mode
  

• The TP-Link Archer A7's DHCP Server is set to Auto, but the DHCP client list is empty, so I'm guessing that it was smart enough to not try to be a DHCP server when in AP mode. Should I explicitly turn that off?
  

Does this info give you any ideas of where I can start diagnosing next?

Any ideas of network diagnostic tools that can help me track down where the traffic is disappearing and why?

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

but in your 2.5.2 configuration you have it checked (now called "Data Encryption Negotiation").

Because (see above) :

Data Encryption Negotiation - <checked>

It's checked because "openvpn 2.5.2 needs (like mandatory) it.

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

The ExpressVPN setup instructions for 2.4.5 say to check "Don't pull routes", but you haven't.

?? Correct - both are non checked.
( probably because the server (ExpressVPN) pushed them ).

About the images that show the red 'offine' : use 8.8.8.8 as the monitoring IP : the IP's used (your IPs) do not reply to ping, so there will be a red "offline, even when the connection is online.
This one :

Can't find this one in your images ?

You have pfSense 2.5.2, right ? I can't help you with the old OpenVPN 2.4.8 used by pfSense 2.4.5p1.
The OpenVPN versions used are different. OpenVPN went from 2.4.8 to 2.5.2 Several options changed ( and that is why this subject exists ). Some minor parameter changes were needed to make it work again.

Btw : your AP should be an AP. Nothing else. Don't leave the DHCP server on Auto, shut it down ( off ). ( because : scenario : powerdown -> powerup : IF the AP boots up earlier as pfSense, the AP server will activate - moments later the sense DHCP servers activates : You've created a whole new issue)
The AP is not part of the issue right now. And if doubt, remove the AP from the equation - use a straight cable, as cables are easy to debug ;)

[amdreallyfast]

@gertjan

About the images that show the red 'offine' : use 8.8.8.8 as the monitoring IP

Done.

BTW, the WAN's "Monitor IP" is set to itself. Is that bad? I didn't set it as such, so it must be the default.

You have pfSense 2.5.2, right ?

I downgraded to 2.4.5 (via format -> fresh install) because I've been trying to replicate the conditions that the ExpressVPN setup instructions expect. If I update to 2.5.2, then I won't be able to go back, but since you were able to get ExpressVPN working on 2.5.2, then I hope that the upgrade shouldn't be an issue. I'll do that tonight.

Btw : your AP should be an AP. Nothing else.

Done. Turned it off. At least that's out of the way.

BTW, I'm hesitant to remove the AP entirely because then I won't have wireless. My router box only has to two ethernet ports. I can't run my PC to the router and still have wireless internet. If the issues continue to persist, I'll try doing it. In hindsight, I should have purchased a mini PC with 4 network ports, but I didn't think of that at the time.

[amdreallyfast]

@gertjan

Well, after upgrading to 2.5.2 and explicitly turning my TP-Link DHCP server off (from Auto) and restarting everything, the gateways are now in indefinite "Pending" mode. I'm out of ideas. Yourself?

[Gertjan]

@amdreallyfast

How many Expr*ssVPN clients have you running ?

Concentrate one one, then add others if needed.
I know Expr**sVPN offers 5 ( ?) simulations connections - dono if they accept 5 from the same device.

Now you have 2.5.2, get your client opvn file and post it here. I'll compare yours with mine.

[amdreallyfast]

@gertjan

Alright, I'll disable two of the interfaces and the OpenVPN clients that use them.

While waiting on the gateways to work themselves out, here's my config.ovpn file, unaltered in any way. It looks like upgrading to 2.5.2 changed how the config files were stored, and now I have the file structure that you had expected in prior posts.

[Gertjan]

@amdreallyfast

I compared you opvn file with mine.
There is 1 ( one ) difference :
local 192.168.10.3
for me, as me WAN IP is 192.168.10.3 - I have an up stream ISP router.
You have a 'real' WAN IP 68.100.115.1x9 - That must be the IP of your WAN interface.

My opvn file :

dev ovpnc2
verb 3
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_client2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA512
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.10.3
tls-client
client
lport 0
management /var/etc/openvpn/client2/sock unix
remote usa-sanfrancisco-ca-version-2.expressnetw.com 1195 udp4
auth-user-pass /var/etc/openvpn/client2/up
auth-retry nointeract
capath /var/etc/openvpn/client2/ca
cert /var/etc/openvpn/client2/cert 
key /var/etc/openvpn/client2/key 
tls-auth /var/etc/openvpn/client2/tls-auth 1
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression asym
comp-lzo adaptive
resolv-retry infinite
fast-io
sndbuf 524288
rcvbuf 524288
fast-io

remote-random
pull
comp-lzo
verify-x509-name Server name-prefix
remote-cert-tls server
key-direction 1
route-method exe
route-delay 2
tun-mtu 1500
fragment 1300
mssfix 1450

You'll see :
comp-lzo adaptive
and then
comp-lzo
in the custom option box.

The latter cancels / modifies the first, I guess.

Do not forget to add the ";".

Now, when I remove the check from 'disable' at the top, and "save" I have these logs :
Logs are in reverse order : the connection sequence starts at the bottom :

2022-01-11 08:40:16.087396+01:00 	openvpn 	73688 	Initialization Sequence Completed
2022-01-11 08:40:16.087382+01:00 	openvpn 	73688 	WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2022-01-11 08:40:16.086402+01:00 	openvpn 	73688 	/sbin/route add -net 10.46.0.1 10.46.0.133 255.255.255.255
2022-01-11 08:40:16.085418+01:00 	openvpn 	73688 	/sbin/route add -net 128.0.0.0 10.46.0.133 128.0.0.0
2022-01-11 08:40:16.084447+01:00 	openvpn 	73688 	/sbin/route add -net 0.0.0.0 10.46.0.133 128.0.0.0
2022-01-11 08:40:16.083229+01:00 	openvpn 	73688 	/sbin/route add -net 45.38.178.184 192.168.10.1 255.255.255.255
2022-01-11 08:40:14.426692+01:00 	openvpn 	73688 	/usr/local/sbin/ovpn-linkup ovpnc2 1500 1557 10.46.0.134 10.46.0.133 init
2022-01-11 08:40:14.421989+01:00 	openvpn 	73688 	/sbin/ifconfig ovpnc2 10.46.0.134 10.46.0.133 mtu 1500 netmask 255.255.255.255 up
2022-01-11 08:40:14.421967+01:00 	openvpn 	73688 	TUN/TAP device /dev/tun2 opened
2022-01-11 08:40:14.421881+01:00 	openvpn 	73688 	TUN/TAP device ovpnc2 exists previously, keep at program end
2022-01-11 08:40:14.421846+01:00 	openvpn 	73688 	ROUTE_GATEWAY 192.168.10.1/255.255.255.0 IFACE=em0 HWADDR=00:15:17:77:d1:19
2022-01-11 08:40:14.421717+01:00 	openvpn 	73688 	Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
2022-01-11 08:40:14.421688+01:00 	openvpn 	73688 	Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
2022-01-11 08:40:14.421585+01:00 	openvpn 	73688 	Data Channel: using negotiated cipher 'AES-256-GCM'
2022-01-11 08:40:14.421576+01:00 	openvpn 	73688 	OPTIONS IMPORT: data channel crypto options modified
2022-01-11 08:40:14.421559+01:00 	openvpn 	73688 	OPTIONS IMPORT: adjusting link_mtu to 1629
2022-01-11 08:40:14.421547+01:00 	openvpn 	73688 	OPTIONS IMPORT: peer-id set
2022-01-11 08:40:14.421535+01:00 	openvpn 	73688 	OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
2022-01-11 08:40:14.421524+01:00 	openvpn 	73688 	OPTIONS IMPORT: route options modified
2022-01-11 08:40:14.421512+01:00 	openvpn 	73688 	OPTIONS IMPORT: --ifconfig/up options modified
2022-01-11 08:40:14.421496+01:00 	openvpn 	73688 	OPTIONS IMPORT: compression parms modified
2022-01-11 08:40:14.421479+01:00 	openvpn 	73688 	OPTIONS IMPORT: timers and/or timeouts modified
2022-01-11 08:40:14.421389+01:00 	openvpn 	73688 	PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 10.46.0.1,comp-lzo no,route 10.46.0.1,topology net30,ping 10,ping-restart 60,ifconfig 10.46.0.134 10.46.0.133,peer-id 56,cipher AES-256-GCM'
2022-01-11 08:40:14.260127+01:00 	openvpn 	73688 	SENT CONTROL [Server-10448-0a]: 'PUSH_REQUEST' (status=1)
2022-01-11 08:40:13.011214+01:00 	openvpn 	73688 	[Server-10448-0a] Peer Connection Initiated with [AF_INET]45.38.178.184:1195
2022-01-11 08:40:13.011195+01:00 	openvpn 	73688 	Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
2022-01-11 08:40:12.834914+01:00 	openvpn 	73688 	VERIFY OK: depth=0, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10448-0a, emailAddress=support@expressvpn.com
2022-01-11 08:40:12.834906+01:00 	openvpn 	73688 	VERIFY X509NAME OK: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10448-0a, emailAddress=support@expressvpn.com
2022-01-11 08:40:12.834897+01:00 	openvpn 	73688 	VERIFY EKU OK
2022-01-11 08:40:12.834887+01:00 	openvpn 	73688 	++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
2022-01-11 08:40:12.834874+01:00 	openvpn 	73688 	Validating certificate extended key usage
2022-01-11 08:40:12.834861+01:00 	openvpn 	73688 	VERIFY KU OK
2022-01-11 08:40:12.834393+01:00 	openvpn 	73688 	VERIFY OK: depth=1, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=ExpressVPN CA, emailAddress=support@expressvpn.com
2022-01-11 08:40:12.834281+01:00 	openvpn 	73688 	VERIFY WARNING: depth=1, unable to get certificate CRL: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=ExpressVPN CA, emailAddress=support@expressvpn.com
2022-01-11 08:40:12.834232+01:00 	openvpn 	73688 	VERIFY WARNING: depth=0, unable to get certificate CRL: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10448-0a, emailAddress=support@expressvpn.com
2022-01-11 08:40:12.669575+01:00 	openvpn 	73688 	TLS: Initial packet from [AF_INET]45.38.178.184:1195, sid=afc6b917 2da3d51c
2022-01-11 08:40:12.507340+01:00 	openvpn 	73688 	UDPv4 link remote: [AF_INET]45.38.178.184:1195
2022-01-11 08:40:12.507290+01:00 	openvpn 	73688 	UDPv4 link local (bound): [AF_INET]192.168.10.3:0
2022-01-11 08:40:12.507235+01:00 	openvpn 	73688 	Socket Buffers: R=[42080->524288] S=[57344->524288]
2022-01-11 08:40:12.507137+01:00 	openvpn 	73688 	TCP/UDP: Preserving recently used remote address: [AF_INET]45.38.178.184:1195
2022-01-11 08:40:12.428036+01:00 	openvpn 	73688 	Incoming Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
2022-01-11 08:40:12.428014+01:00 	openvpn 	73688 	Outgoing Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
2022-01-11 08:40:12.427759+01:00 	openvpn 	73688 	WARNING: experimental option --capath /var/etc/openvpn/client2/ca
2022-01-11 08:40:12.426711+01:00 	openvpn 	73688 	NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2022-01-11 08:40:12.426487+01:00 	openvpn 	73688 	MANAGEMENT: unix domain socket listening on /var/etc/openvpn/client2/sock
2022-01-11 08:40:12.425588+01:00 	openvpn 	73485 	library versions: OpenSSL 1.1.1k-freebsd 25 Mar 2021, LZO 2.10
2022-01-11 08:40:12.425575+01:00 	openvpn 	73485 	OpenVPN 2.5.2 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Jun 24 2021
2022-01-11 08:40:12.425550+01:00 	openvpn 	73485 	WARNING: file '/var/etc/openvpn/client2/up' is group or others accessible
2022-01-11 08:40:12.425218+01:00 	openvpn 	73485 	WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent packets are not compressed unless "allow-compression yes" is also set.

There are some warnings about "Compression" - Not important right now (?).
WARNING: experimental option : nice, we want to experiment.
two "VERIFY WARNING" for the CRL : we don't care.
And a "WARNING: this configuration may cache" because we're lazy, so ok.

Do you have exactly the same log lines ?
( using "verb 3" means log level 3 - the default )

[amdreallyfast]

@gertjan
I changed my custom options to match yours, and then looked at Status -> System Logs -> OpenVPN.

Noticing AUTH_FAILED. That's clearly bad.

Note: This is just one AUTH_FAILED cycle. The full logs repeat this log segment many times.

Jan 11 18:09:57	openvpn	56223	NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jan 11 18:09:57	openvpn	56223	Outgoing Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Jan 11 18:09:57	openvpn	56223	Incoming Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Jan 11 18:09:57	openvpn	56223	TCP/UDP: Preserving recently used remote address: [AF_INET]45.38.178.54:1195
Jan 11 18:09:57	openvpn	56223	Socket Buffers: R=[42080->524288] S=[57344->524288]
Jan 11 18:09:57	openvpn	56223	UDPv4 link local (bound): [AF_INET]68.100.115.149:0
Jan 11 18:09:57	openvpn	56223	UDPv4 link remote: [AF_INET]45.38.178.54:1195
Jan 11 18:09:57	openvpn	56223	TLS: Initial packet from [AF_INET]45.38.178.54:1195, sid=6cbc8ab5 ec9e0d58
Jan 11 18:09:57	openvpn	56223	VERIFY WARNING: depth=0, unable to get certificate CRL: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10457-0a, emailAddress=support@expressvpn.com
Jan 11 18:09:57	openvpn	56223	VERIFY WARNING: depth=1, unable to get certificate CRL: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=ExpressVPN CA, emailAddress=support@expressvpn.com
Jan 11 18:09:57	openvpn	56223	VERIFY OK: depth=1, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=ExpressVPN CA, emailAddress=support@expressvpn.com
Jan 11 18:09:57	openvpn	56223	VERIFY KU OK
Jan 11 18:09:57	openvpn	56223	Validating certificate extended key usage
Jan 11 18:09:57	openvpn	56223	++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Jan 11 18:09:57	openvpn	56223	VERIFY EKU OK
Jan 11 18:09:57	openvpn	56223	VERIFY X509NAME OK: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10457-0a, emailAddress=support@expressvpn.com
Jan 11 18:09:57	openvpn	56223	VERIFY OK: depth=0, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10457-0a, emailAddress=support@expressvpn.com
Jan 11 18:09:57	openvpn	56223	Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
Jan 11 18:09:57	openvpn	56223	[Server-10457-0a] Peer Connection Initiated with [AF_INET]45.38.178.54:1195
Jan 11 18:09:58	openvpn	56223	SENT CONTROL [Server-10457-0a]: 'PUSH_REQUEST' (status=1)
Jan 11 18:09:58	openvpn	56223	AUTH: Received control message: AUTH_FAILED
Jan 11 18:09:58	openvpn	56223	SIGUSR1[soft,auth-failure] received, process restarting
Jan 11 18:09:58	openvpn	56223	Restart pause, 10 second(s)

And the gateway is still pending:

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

This is just one AUTH_FAILED cycle

That's a matter of copying correctly your credentials.
Was this message there all the time ?
This is the most logical reason the connection doesn't come up : your bank site won't let you in neither with wrong credentials..

This issue wasn't worth mentioning, as it is easy to correct.

edit : in the folder where you found the opvn client file, you will also find a file called 'up'.
Open it ;)
This explains this setting :

auth-user-pass /var/etc/openvpn/client1/up

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

And the gateway is still pending

When the login is transmitted, and it's wrong, the "PUSH_REQUEST" will not receive the route, gateway, IP etc but an error message : you saw it.
The gateway isn't reachable, as you failed authentication.
When the "auth" went well, you receive a

PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 10.46.0.1,comp-lzo no,route 10.46.0.1,topology net30,ping 10,ping-restart 60,ifconfig 10.46.0.134 10.46.0.133,peer-id 56,cipher AES-256-GCM'
2022-01-11 08:40:14.260127+01:00 openvpn 73688 SENT CONTROL [Server-10448-0a]: 'PUSH_REQUEST' (status=1)

and as you can see, the gateway, amongst others, is in there. pfSense uses all this info to set up everything.

[amdreallyfast]

@gertjan

Looks like it's working now. The auth problem was caused by an old username/password that went bad when the subscription recently expired. I just renewed it, entered the new username/password, and now the gateway is online.

Then I enabled the firewall rules and firewall NAT -> Outbound routing rules, and it looks like its my San Francisco VPN is now working and routing traffic correctly. I'm writing this post while my router is connected to the new VPN.

It looks like your settings worked for me too. I remember deviating from the ExpressVPN setup instructions in the VPN client setup by checking Enable NCP (2.4.5; on 2.5.2 it's called "Enable Data Encryption Negotiation") and by unchecking Don't pull routes", and by unchecking both Interfaces -> <vpn interface name> -> Reserved Networks -> Block private networks and loopback addresses and Block bogon networks. And I think that I edited the manual custom configuration options a bit so that they match yours. Current settings.

remote-random;
pull;
comp-lzo;
verify-x509-name Server name-prefix;
remote-cert-tls server;
key-direction 1;
route-method exe;
route-delay 2;
tun-mtu 1500;
fragment 1300;
mssfix 1450;
verb 3;
sndbuf 524288;
rcvbuf 524288;

Thanks for all the patience. Next I'll need to figure out custom routing rules for connections that don't like the VPN. Some places block it and I'll need to make exceptions for those.

[Gertjan]

@amdreallyfast
Sit back, and think with me :
What is different between these 'famous' Exprr*VPN instructions, and the ones needed to make VPN works with 2.5.2.
You'll agree with me : it's not much. Re see the post about this subject I made several month ( ? ) ago.
I'll have a look at it this weekend.

[amdreallyfast]

@gertjan

You'll agree with me : it's not much.

True. The problem is that network software rarely settles for "close enough". Figuring out exactly what I needed to change was the rough part.

And now, curiously enough, the docs pages for pfsense won't load. It's like my firewall (or maybe theirs) is blocking me. Back to the non-VPN configuration while I try to figure out how to pass traffic for certain websites through the non-VPN connection.