@viragomann

Thanks Virago, the error is because a DNS problem, we fix it.

@viragomann Okay, but in the firewall I am blocking communication from the LAN to the VLAN, but the LAN still communicates with the other side of the tunnel.

Configuring my IPs:

LAN Main firewall: 10.1.1.1/24
VLAN: 172.24.0.0/24

LAN client firewall: 10.0.0.1/24

IPv4 Tunnel Network: 192.168.210.0/30

@jimp Awesome, thank you for the reassurance. We'll keep working on moving our users over but can at least take advantage of the bug fixes/etc in 23.09

I finally found the culprit. The clients that I was expecting to connect to the OpenVPN server were configured under OpenVPN > Clients. Hence the server tried to connect to itself. In combination with push "redirect-gateway autolocal def1";, that seems to have broken the routing on the pfSense.

The solution was to delete the clients from OpenVPN > Clients.

@cmkrs

Thanks for the great start.

A few items I had to add and validate to make it all work.

I was not able to publish my findings and step-by-step process - Akismit flagged it as SPAM - under this forum.

So, I published it to my web site at this link: https://d-b-s.com/documents

Credits: This is a compilation of several articles on the WEB, but it started here with this article as it had the most information. Thanks.

SOLVED

@viragomann Thanks for the ideas that got me to solve the entire thing.

I started with 2.6 using Peer to Peer (Shared Keys) on the site to site peer clients. I converted all the client sites fine with SSL/TLS but the key piece was Client Specific Overrides on the various servers I was connecting to needed. I did not need this before 2.7 to get everything working.

My various servers were 2.6 and my firewall peer clients that connected to those 2.6 is a new 2.7. It now works. I had 4 Server 2.6 I was connecting to using a new 2.7 Client firewall.

As long as you have the certs correctly set up which I did not have a problem with, you should be good. The key change or use for me was the CSO per @viragomann. CSO on the OpenVPN Server fixed the routing by populating the necessary routing / gateway configurations for my peer client connections for each corresponding sites.

Steps on OpenVPN Server pfSense firewall

1 - Create CA on Peer to Peer Server (export CA cert)
2 - Create Server Cert on Peer Server
3 - Create Client Cert for EACH Peer to Peer Client (export cert and key)
4 - Create OpenVPN Server setup selecting SSL/TLS on Peer to Peer and add the IPv4 Tunnel Network, IPv4 Local network(s), and IPv4 Remote network(s)
5 - Create Client Specific Overides for EACH peer client firewall connecting to this server
6 - Name Common Name same as the corresponding cert for the specific peer client, and fill in IPv4 Tunnel Network, IPv4 Local Network/s, IPv4 Remote Network/s

Steps on OpenVPN Peer Client pfSense firewall

1 - Import the CA (from step 1 server section above) and the corresponding peer Client cert and key (from step 3 server section above)
2 - Go to VPN / OpenVPN / Clients tab and begin adding your peer client for each Open VPN Server you need to connect to (maybe you are just connecting to one)
3 - Peer to Peer (SSL/TLS)
4 - Choose the proper port if you have several peer client setting up
5 - Select your imported CA in Peer Certificate Authority (from Step 1 in Server section) and the imported corresponding Client Certificate (from Step 3 above in Server section)
6 - Fill IPv4 Tunnel Network, IPv4 Remote network(s)

Firewall / Rules / OpenVPN

1 - Add Pass for ANY protocol on IPV4 and ANY/ANY Source / Destination to verify flow and then you can filter more if need to later

** You may need to restart the services for OpenVPNServer and OpenVPN Peer Client firewalls....connections should be made if the proper Network and Subnets were created.

Regression #13613 sounds like a valid explaination: "It looks like the problem is that we send a SIGTERM to openvpn, but don't wait until it actually exits before destroying the interface. That it turn causes it to not actually exit, breaking the subsequent openvpn instance."

Though this was for 23.01, it may have been introduced with 2.7 as well, as i did not have any such issues as long as we were on 2.6.

@CyberMinion
Worked like a charm. I had tried creating the deny rule but didnt know about the 'Do not create rules when gateway is down' setting.

Thank you!

@Bambos viragomann just refereed me to your post. Did you ever switch to Peer to Peer SSL/TLS instead of Shared Key? And if you did, did it help?

Here's my finding so far - https://forum.netgate.com/topic/183854/open-vpn-2-7-site-to-site-odd-routing-issue/11

@viragomann Thank you, sir! I will be implementing next week.

Ha, that did it :-)

Thanks a lot.

We created a new server cert, installed it and were bitten by the 'VERIFY KU ERROR' bug when restarting the openVPN :-(
The certificate had been used on both servers .....

We got that fixed and updated to 2.7.0 without a problem :-)

Now considering getting a paid licence ;-)

The solution to this issue is: read the fine print.

"Enter any additional options to add to the OpenVPN server configuration here, separated by semicolon."

So I made the following changes:

push "route 172.31.4.0 255.255.255.0"; push "route 172.31.40.0 255.255.255.0"

Mind the semicolon at the end of the first line.

Thank you for letting me use this forum as a Rubber Duck

@rustem
To assign a specific IP address to a VPN client, I use the "Client Specific Overrides" tab, it's where you can select a client by its "Common Name" (the client certificate name, ou the username for VPNs utilizing password authentication), select the VPN server in the Server List, and can use the ifconfig-push directive at the end of the page, in Advanced field.
Also, the netmask that you put in ifconfig-push seems wrong, you put 255.255.255.255 instead of 255.255.255.0 (the netmaks of your tunnel network).

8bd9b26d-a50f-441e-8a69-9367336b8157-image.png
Resolved! I added the VPN range (Add Local Network Field).

@CyberMinion That was just a manual ping from ssh. The 1100 is a little underpowered (CPU-wise), so I've noticed it can take several minutes, but sometimes it will start working. Other times, it gets hung up and won't connect. So, I believe maybe my settings are correct, but it is just a little slow to get going, plus sometimes it just has trouble and reloading the process or rebooting fixes it, but it's not very quick, so it's just difficult to troubleshoot....?

My 4100 is instantaneous and works every time. I recently also reflashed/upgraded my 1100 to see if that would help, but again, I think part of the problem is that it is underpowered. Just switching between tabs/pages is a little slow, not terrible, but an indication of it's low resources.

I'm not trying to be critical, the 1100 works fine once you get everything set, but troubleshooting is a little tedious.

There have been updates to this strategy. Since this was posted, OpenVPN has introduced the --auth-gen-token option.

All that is necessary is to add auth-gen-token; to the server's custom options. No client reconfiguration is necessary.

Here is the section from the OpenVPN documentation:

--auth-gen-token [lifetime]

After successful user/password authentication, the OpenVPN server will with this option generate a temporary authentication token and push that to client. On the following renegotiations, the OpenVPN client will pass this token instead of the users password. On the server side the server will do the token authentication internally and it will NOT do any additional authentications against configured external user/password authentication mechanisms.The lifetime argument defines how long the generated token is valid. The lifetime is defined in seconds. If lifetime is not set or it is set to 0, the token will never expire.

This feature is useful for environments which is configured to use One Time Passwords (OTP) as part of the user/password authentications and that authentication mechanism does not implement any auth-token support.