pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss

[amdreallyfast]

@lovan6

I still used Pfsense 2.4.5 P1 and never bother to upgrade. Why fix it if ain't broke. It's just me.

I tried reinstalling from scratch, this time using 2.4.5, followed the configuration exactly, and had the exact same problem. I did it three times with servers in LA, San Francisco, and DC. All three of them show "up" in Status -> OpenVPN -> Clients, all of them could ping google.com through the Diagnostics -> Ping tool, and yet all of them also had 100% packet loss and none of their connections would work. I checked the system logs and all their connections were timing out and retrying every few minutes, always with the same result.

If 2.4.5 works for you but not for me, then I did something wrong and it's probably really small; I just don't know what I did wrong and need something to compare against. Do you know if it's possible for you to export your entire configuration in a single file and share it ((sans sensitive information like private keys and passwords) so that I can compare against my setup and see what I did wrong?

[lovan6]

@amdreallyfast

Send me a PM and I will send you my ExpressVpn config.

There are also several configurations for DNS resolver, Aliases, Firewall rules, Nat including ExpressVPN media streamer DNS.

Do a clean slate. There are several configurations not offered on ExpressVPN instructions. Particularly user destined for VPN only. By default ExpressVPN instructions, users are all connected to VPN.

As I said setup Pfsense 2.4.5 P1 first. Let me know how it works if you plan to upgrade to 2.5.2. Since I don't have the time if anything goes wrong with 2.5.2. Wife will start complaining if her VPN does not work.

FYI I used a different Pfsense setup. It is an Intel Kabylake processor and an Intel i350 T4 V2 server NIC. M.2 sata formatted to ZFS. Fiber optic connection, modem in bridge mode.

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

I don't understand this sentence. Are you asking me to ssh into the router and run sftp to...do something? Please explain.

Use SSH if your not close to pfSense. Or use the console access.
IMHO : it's a shot in the dark to make a VPN connection to a server work just using the GUI.
I need to see what happens.
Comparing the file as you can download from Expr*ssVPN and the one actually created by pfSense is an important step. So, yeah, the good old command line access comes in handy here.

SFTP is FTP over SSH. Using SFTP permits you to browse the files on pfSense like you can browser files on your W10 using Explorer.

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

I've never ssh'd into this router before and it looks like it needs special setup. I tried ssh admin@192.168.1.1 (still using defaults except password until I can get this working), but the connection timed out, so ssh isn't an option until I get help fixing that.

Check "Enable Secure shell" and Save.
No special other setup is needed.

If you use Windows OS, install world's most famous SSH application : Putty.

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

Which log example?

The one that starts with

The connection logs (in reverse order) :

2021-10-22 12:39:45.217054+02:00 openvpn 7206 Initialization Sequence Completed

[amdreallyfast]

@gertjan
Working on it. After following your directions in the prior post (and after learning about sftp), I was able to find the config files, but there is no .ovpn file. There's are several other files:

I pieced together the files into something resembling the ExpressVPN .ovpn file that I got from their website, and then I used Notepad++'s Compare plugin to look at the differences. The following differences mostly come from the client1.conf file, who's contents are pasted at the top of the text on the left side of the compare:

There are a lot more items in the .conf file than at the top of the ExpressVPN-provided .ovpn file, and for the ones that are similar (near the bottom of the screenshot), there are a few that are different. It looks like the custom options specified by ExpressVPN's PfSense setup instructions are not identical to the options specified in the individual .ovpn files. Lovely.

On a hunch, I tried taking all those options from .ovpn file, splicing them together with semicolons, and then copy-pasted that chunk into the text box in VPN -> OpenVPN -> Clients -> <SF client> -> Advanced Configuration -> Custom options. And the VPN wouldn't even connect. That means that the custom options that ExpressVPN specifies in their .ovpn files are not correct. Dang it.

So which configuration is correct? Are either correct?

Then I started altering items one by one to slowly try to match the custom options specified in the ExpressVPN PfSense setup instructions, and eventually I got Status -> OpenVPN to show that it was online again, but Status -> Gateways still shows the San Francisco VPN gateway offline with 100% packet loss.

Curiously, the VPN connection succeeded even though I didn't have the exact same custom options as the ExpressVPN PfSense setup instructions. I was aiming for that, but it connected prior to reaching there. The following custom options on the right (minus the struck-through ones) seemed to work. Notice that dev tun, nobind, and auth-user-pass would all cause Status -> OpenVPN to show a failed connection.

It looks like the other differences in the screenshot don't matter for the VPN connection itself. Do you have any ideas why?

The gateway is still offline.

Would you send me your config file so that I can compare?

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

but there is no .ovpn file.

Strange.

Another view : from WINSCP :

The is a folder in /var/etc/openvpn called client1
so the path is /var/etc/openvpn/client1/ here are the files for the first openvpn client - there must be a .opvn file.

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

Would you send me your config file so that I can compare?

First thing tomorrow morning. Within 6 hours ;)

[Gertjan]

could you send me your config file so that I can compare?

Done.

See 2 forum messages back, it's al ready there.

edit !
Wait !

Check my config from above :
It says :

comp-lzo yes

as per instructions from Expr*ssVPN.

I reloaded the config, mine, for Paris, and yours, for San Franciso.
Now it says :

comp-lzo no

This is a very recent config change on their side.
I changed the option in the pfSense GUI for :

This way, my config shows :

comp-lzo no

And take note : this is 'bad' as the "comp-lzo" command is now deprecated (in openvpn 2.5.2).
It's probably better to select :

which introduced a new :

allow-compression asym

in the local config file.
This proves that Expr**ssvpn is still based on 2.4.x style of configuration.

To be clear : with

comp-lzo no

I could ping google.com just fine (it connected).

[amdreallyfast]

@gertjan

The lack of a client1 folder and the lack of an .ovpn file may be due to me using 2.4.5. Maybe. I'm intentionally using the older version because ExpressVPN's instructions are explicitly stated to be for 2.4.5, and I'm trying to avoid rocking the boat too much from the original instructions (at least until I get this working). And I say "maybe" because ExpressVPN was apparently able to create .ovpn files with 2.4.5. I just don't know how. Did you have to install a specific package in order to export the config file?

[amdreallyfast]

@gertjan

See 2 forum messages back, it's already there.

Are you sure? Two messages (of yours) back I see this post, and I'm not seeing a config file. Would you take a screenshot of the message (and link it if it's another forum post)?

[amdreallyfast]

@gertjan

I could ping google.com just fine (it connected).

Same. The gateway is offline with 100% packet loss, and yet the Diagnostics -> Ping tool could use that VPN client to ping. This tells me that the gateway being offline with 100% packet loss is not an issue that can be checked with the Ping tool. But how can I diagnose it then?

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

And I say "maybe" because ExpressVPN was apparently able to create .ovpn files

Expr*ssVP always exported 'ovpn' files.
These are just 'plain text' files, filled with process command line options that the process openvpn understands.
pfSense 2.5.2 uses the same format and extension 'internally'.
That is because pfSense is just an OS (sort of), with a nice GUI, and for the rest it uses the same processes (binary programs) as all the other MAC based / Window based / FreeBSD based / Lines based systems.

Dono what pfSense 2.4.5 did, that's to far in the past for me.

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

Did you have to install a specific package in order to export the config file?

Exporting what to what ?
To export a 'setup' (config opvn file + binaries) or just the opvn file from an OpenVPN Access server (server !!) then you need this

package.
See the pfSense manual or one of a thousands of OpenVPN video's, book, whatever.

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

Are you sure? Two messages (of yours) back I see this post, and I'm not seeing a config file. Would you take a screenshot of the message (and link it if it's another forum post)?

Here :

That an image from this thread.
You can't see it ?

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

This tells me that the gateway being offline with 100% packet loss is not an issue that can be checked with the Ping tool. But how can I diagnose it then?

Because some one - in your case 10.173.0.137 - doesn't answer to 'ping' this doesn't mean the gateway is offline.
What this implies is :
The path to host 10.173.0.137 doesn't exist
or
It doesn't reply to ping.
or ....

When I empty this field :

the pfSense montoring system will auto choose an IP, like your "10.173.0.137".

And I see the same thing : no ping replies. So it says 'offline'. Or, like you, I can ping 'google' just fine.
That why I fill in a IP that I "have /own / use" and that is close to the Exprss VPN exit WAN IP : I have a dedicated server near by. Or you could use what everybody does : use 8.8.8.8 as a monitoring IP.
And your ExpressVPN is now (nope, this should be : "looks like") on-line.

[amdreallyfast]

@gertjan

Expr*ssVP always exported 'ovpn' files.

...Then why can't I? This is weird. I previously mentioned that I've tried SF, LA, and DC, so I tried turning on all their VPN clients and interfaces, wondering if that might cause it to create a client1 folder to separate the three clients. Nope. My ... folder now looks like this. I don't know why OpenVPN is not creating .ovpn files. Any ideas?

That an image from this thread.
You can't see it ?

Oh. That's your file? I was looking for a file attachment (as opposed to copy-pasted values), but now I have questions about this (see next question).

I had to add these missing :
<several lines of custom options>
Now my config.opvn file looks like this :
<more lines of custom options>

Which one is your file? You said that you had to add some values, and then you say, "my config.ovpn file looks like this" and you show a list of custom options that does not include the ones that you just said you had to add. Now I'm confused about what your file looks like. Are you sure that's the entirety of the .ovpn file and that the file does not include the items that you had just added?

When I empty this field :
<screenshot of Monitor IP>
the pfSense montoring system will auto choose an IP, like your "10.173.0.137".
And I see the same thing : no ping replies. So it says 'offline'.

I don't think that we're seeing the same thing. The Monitor IP field is blank for all three of my VPN client configurations (SF, LA, DC), and all three of them do show ping replies while the gateway is offline. Please explain. Are you saying that I should leave Monitor IP blank or that I should not leave it blank?

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

Then why can't I?

Because you strugle with the wrong questions ;)

There is no such thing as exporting a "OpenVPN client" config file.

A OpenVPN server installation you have set up, benefits from the pfSense "openvpn-client-export" package. The file you exports comes from the OpenVPN (server parameters) and are used by the future OpenVPN clients, so they can access the (your) OpenVPN server for remote access.

Your subject is all about the OpenVPN client set up.
The OpenVPN server - the provider, in this case - is ExpressVPN.
So it's Express VPN that shares a 'config file' with you, what I call the opvn config file. You downloaded it.
This file has to be used as a guide line to fill in the pfSense GUI client page.
Currently, pfSense does not offer a no way to 'import' the file you got from ExpessVPN.

Now you understand (bitter) why I want you to look at the openvpn client config file that pfSense GUI has created with it's GUI.
This file has to resemble as close as possible as the file you obtained from ExpressVPN.

The
Server
Client specific overrides
Wizard
Client Export
Shared key Export

are related to the OpenVPN server.

The OpenVPN client has only one page : the one that's underlined in my image.

An important aspect is also : the GUI can only handle a subset of the possible options or settings. A GUI is useful for enforcing 'illegal' option combinations.

Got it ?! ;)

Side note : it would be nice to have the possibility to import a server side generated file, so you can set up the client easely.
But there is a thing : settings are version specific. They change all the time.
"openvpn" as a program has close to one thousand settings (options and possible combination of options combined - it's close to impossible to write a GUI to handle them all.

Big (server type) apps, like apache2, nginx, postfix and bind (named) can not be set up using a GUI.
You have to fall back to the classic 'confg files' as the shear number of combinations of settings is just daunting.

Btw : the last part is totally IMHO.

[amdreallyfast]

@gertjan

Because you strugle with the wrong questions ;)

The curse of learning. Thanks for bearing with me through this. Books and tutorials can't correct me when I understand a term badly, but you can. Thanks :).

Now you understand (bitter) why I want you to look at the openvpn client config file that pfSense GUI has created with it's GUI.
This file has to resemble as close as possible as the file you obtained from ExpressVPN.

Alright, so how do I get "the openvpn client config file that pfSense GUI has created with it's GUI"? You walked me through sftp, but that showed that I didn't have a .ovpn file at all. Is there a way to get this .ovpn file through the GUI?

It sounds like you're looking for the Client Specific Overrides. Are you sure? Mine are blank. I have done anything here.

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

It sounds like you're looking for the Client Specific Overrides

A point for you !

I meant :

All the other menu options are 'server' related.

edit :
I restarted reading you first post (see above).
I decided to enter exactly what you've mentioned.
One exception : I removed my password.

https://www.test-domaine.fr/VPN-Client.png

Compare this huge image with your settings.
Double check for differences.
( edit : I found a difference : my verbosity (log) level is 3 - level 10 is far to detailed, create hundreds of entries a second. Leaving it 10 or 3 has no other side effect )

With these settings :

I monitored 8.8.8.8 to check the VPN-Client connection - RTT is a bit high as I'm based in France, St Jose in the US is not really close.

All I had to to to make the connection works, was switching from :

"automatic mode" to "hybrid mode" and save.
I'm now connected the entire company over Expr*ssVPN ...... I wasn't expecting this, as I wanted to focus on making the connection work first, and then go for the outbound natting.

I didn't even add a 'policy' firewall rule on my LAN interface to make the connection work.
It's not activated :

So, first things first : Make your connection work.

Btw : this is the file the OpenVPN client is using right now :

/var/etc/openvpn/client2/config.ovpn

dev ovpnc2
verb 3
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_client2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA512
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local 192.168.10.3
tls-client
client
lport 0
management /var/etc/openvpn/client2/sock unix
remote usa-sanfrancisco-ca-version-2.expressnetw.com 1195 udp4
auth-user-pass /var/etc/openvpn/client2/up
auth-retry nointeract
capath /var/etc/openvpn/client2/ca
cert /var/etc/openvpn/client2/cert 
key /var/etc/openvpn/client2/key 
tls-auth /var/etc/openvpn/client2/tls-auth 1
data-ciphers AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC
data-ciphers-fallback AES-256-CBC
allow-compression asym
comp-lzo adaptive
resolv-retry infinite
fast-io
sndbuf 524288
rcvbuf 524288
fast-io

persist-key

persist-tun

remote-random

pull

comp-lzo

tls-client

verify-x509-name Server name-prefix

remote-cert-tls server

key-direction 1

route-method exe

route-delay 2

tun-mtu 1500

fragment 1300

mssfix 1450

verb 3

sndbuf 524288

rcvbuf 524288

You can clearly see de double spaces lines at the bottom, some of them are present twice in the config file.
This is an issue for later, because, with these settings, it works.

I'm using :

[amdreallyfast]

@gertjan

That's a real big screenshot. Kudos on spending the time to piece that monstrosity together.

Noticed some differences:

1. The ExpressVPN setup instructions for 2.4.5 say to leave NCP (Negotiable Cryptographic Parameters) blank, but in your 2.5.2 configuration you have it checked (now called "Data Encryption Negotiation").
   

2. The ExpressVPN setup instructions for 2.4.5 say to check "Don't pull routes", but you haven't.
   

But I followed along anyway and replicated your settings. And the gateways are still offline. I haven't adjusted the firewall rules yet. I'm still trying to get these gateways online.

Note: Prior to making these screenshots, I shortened the names of the VPN clients and their interfaces, so they'll appear a little different than in prior screenshots, but they're still the same ones.

• VPN clients online
  

• Gateways offline
  

• Using ExpressVPN's IP address checker:
  

• Interface settings
  

• Gateway settings
  

• Client settings
  

[amdreallyfast]

@gertjan

At this point I'm thinking that the only way forward is for a network expert who's familiar with PfSense and OpenVPN to sit down at my computer and poke around until they can figure out what's going on. Whatever my problem is, it's not normal. I'm not a network expert, and I don't know how to diagnose this. I've replicate other peoples' working settings, and it isn't working on my end. Here's some more diagnostic info; does it give you any ideas about where to start diagnosing next?

• I can tell from a basic Wireshark scan of my network traffic that TCP failure and retransmission is not uncommon, and PfSense's Status -> System Logs -> Gateways shows me that communication with the gateways is frequently timing out. All gateways are timing out, including the WAN, which is what I'm currently using to get online and talk with you.
  
  These are all from this morning:
  

• It's rather frequent that I try to load a web page, wait several seconds, the page fails to load and the browser says that it can't find the page, and then it automatically retries and succeeds (sometimes needs to retry twice). I don't know what is causing this frequent failure, or if there's anything I can do about it. I've got a 100Mbps internet connection over cable that is usually stable and reliable once connection is actually established (ex: a long download or a big YouTube video doesn't fail once it's started). It shouldn't take 5-10sec to load a google search. The failures (when they happen) only occur when initially loading a page. If it were a random hardware failure (circuitry going bad, loose cable, etc.), I'd expect random disconnects during downloads and streaming large video, but I'm not seeing that.
  
  I've mentioned before that when I use the ExpressVPN desktop app to make a VPN connection, all those connection issues for my PC disappear. Web pages load quickly 100% of the time, no timeouts, no lost connections. That makes me wonder if there's some sort of packet monitoring and sniffing going on after the traffic leaves my network that either slows the transmission enough to time out or stops it entirely. Why would making an encrypted tunnel to ExpressVPN's servers magically make the disconnects go away?

• My connection is this: Cable -> modem -> router (mini PC running PfSense 2.4.5) -> TP-Link Archer A7 (wifi router/switch configured as access point only) -> PC.
  
  Here's the TP-Link Archer A7 set in AP mode
  

• The TP-Link Archer A7's DHCP Server is set to Auto, but the DHCP client list is empty, so I'm guessing that it was smart enough to not try to be a DHCP server when in AP mode. Should I explicitly turn that off?
  

Does this info give you any ideas of where I can start diagnosing next?

Any ideas of network diagnostic tools that can help me track down where the traffic is disappearing and why?

[Gertjan]

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

but in your 2.5.2 configuration you have it checked (now called "Data Encryption Negotiation").

Because (see above) :

Data Encryption Negotiation - <checked>

It's checked because "openvpn 2.5.2 needs (like mandatory) it.

@amdreallyfast said in pfsense 2.5.2: ExpressVPN connection working by gateway has 100% packet loss:

The ExpressVPN setup instructions for 2.4.5 say to check "Don't pull routes", but you haven't.

?? Correct - both are non checked.
( probably because the server (ExpressVPN) pushed them ).

About the images that show the red 'offine' : use 8.8.8.8 as the monitoring IP : the IP's used (your IPs) do not reply to ping, so there will be a red "offline, even when the connection is online.
This one :

Can't find this one in your images ?

You have pfSense 2.5.2, right ? I can't help you with the old OpenVPN 2.4.8 used by pfSense 2.4.5p1.
The OpenVPN versions used are different. OpenVPN went from 2.4.8 to 2.5.2 Several options changed ( and that is why this subject exists ). Some minor parameter changes were needed to make it work again.

Btw : your AP should be an AP. Nothing else. Don't leave the DHCP server on Auto, shut it down ( off ). ( because : scenario : powerdown -> powerup : IF the AP boots up earlier as pfSense, the AP server will activate - moments later the sense DHCP servers activates : You've created a whole new issue)
The AP is not part of the issue right now. And if doubt, remove the AP from the equation - use a straight cable, as cables are easy to debug ;)

[amdreallyfast]

@gertjan

About the images that show the red 'offine' : use 8.8.8.8 as the monitoring IP

Done.

BTW, the WAN's "Monitor IP" is set to itself. Is that bad? I didn't set it as such, so it must be the default.

You have pfSense 2.5.2, right ?

I downgraded to 2.4.5 (via format -> fresh install) because I've been trying to replicate the conditions that the ExpressVPN setup instructions expect. If I update to 2.5.2, then I won't be able to go back, but since you were able to get ExpressVPN working on 2.5.2, then I hope that the upgrade shouldn't be an issue. I'll do that tonight.

Btw : your AP should be an AP. Nothing else.

Done. Turned it off. At least that's out of the way.

BTW, I'm hesitant to remove the AP entirely because then I won't have wireless. My router box only has to two ethernet ports. I can't run my PC to the router and still have wireless internet. If the issues continue to persist, I'll try doing it. In hindsight, I should have purchased a mini PC with 4 network ports, but I didn't think of that at the time.

[amdreallyfast]

@gertjan

Well, after upgrading to 2.5.2 and explicitly turning my TP-Link DHCP server off (from Auto) and restarting everything, the gateways are now in indefinite "Pending" mode. I'm out of ideas. Yourself?

[Gertjan]

@amdreallyfast

How many Expr*ssVPN clients have you running ?

Concentrate one one, then add others if needed.
I know Expr**sVPN offers 5 ( ?) simulations connections - dono if they accept 5 from the same device.

Now you have 2.5.2, get your client opvn file and post it here. I'll compare yours with mine.