@cmkrs

Thanks for the great start.

A few items I had to add and validate to make it all work.

I was not able to publish my findings and step-by-step process - Akismit flagged it as SPAM - under this forum.

So, I published it to my web site at this link: https://d-b-s.com/documents

Credits: This is a compilation of several articles on the WEB, but it started here with this article as it had the most information. Thanks.

SOLVED

@viragomann Thanks for the ideas that got me to solve the entire thing.

I started with 2.6 using Peer to Peer (Shared Keys) on the site to site peer clients. I converted all the client sites fine with SSL/TLS but the key piece was Client Specific Overrides on the various servers I was connecting to needed. I did not need this before 2.7 to get everything working.

My various servers were 2.6 and my firewall peer clients that connected to those 2.6 is a new 2.7. It now works. I had 4 Server 2.6 I was connecting to using a new 2.7 Client firewall.

As long as you have the certs correctly set up which I did not have a problem with, you should be good. The key change or use for me was the CSO per @viragomann. CSO on the OpenVPN Server fixed the routing by populating the necessary routing / gateway configurations for my peer client connections for each corresponding sites.

Steps on OpenVPN Server pfSense firewall

1 - Create CA on Peer to Peer Server (export CA cert)
2 - Create Server Cert on Peer Server
3 - Create Client Cert for EACH Peer to Peer Client (export cert and key)
4 - Create OpenVPN Server setup selecting SSL/TLS on Peer to Peer and add the IPv4 Tunnel Network, IPv4 Local network(s), and IPv4 Remote network(s)
5 - Create Client Specific Overides for EACH peer client firewall connecting to this server
6 - Name Common Name same as the corresponding cert for the specific peer client, and fill in IPv4 Tunnel Network, IPv4 Local Network/s, IPv4 Remote Network/s

Steps on OpenVPN Peer Client pfSense firewall

1 - Import the CA (from step 1 server section above) and the corresponding peer Client cert and key (from step 3 server section above)
2 - Go to VPN / OpenVPN / Clients tab and begin adding your peer client for each Open VPN Server you need to connect to (maybe you are just connecting to one)
3 - Peer to Peer (SSL/TLS)
4 - Choose the proper port if you have several peer client setting up
5 - Select your imported CA in Peer Certificate Authority (from Step 1 in Server section) and the imported corresponding Client Certificate (from Step 3 above in Server section)
6 - Fill IPv4 Tunnel Network, IPv4 Remote network(s)

Firewall / Rules / OpenVPN

1 - Add Pass for ANY protocol on IPV4 and ANY/ANY Source / Destination to verify flow and then you can filter more if need to later

** You may need to restart the services for OpenVPNServer and OpenVPN Peer Client firewalls....connections should be made if the proper Network and Subnets were created.

Regression #13613 sounds like a valid explaination: "It looks like the problem is that we send a SIGTERM to openvpn, but don't wait until it actually exits before destroying the interface. That it turn causes it to not actually exit, breaking the subsequent openvpn instance."

Though this was for 23.01, it may have been introduced with 2.7 as well, as i did not have any such issues as long as we were on 2.6.

@CyberMinion
Worked like a charm. I had tried creating the deny rule but didnt know about the 'Do not create rules when gateway is down' setting.

Thank you!

@Bambos viragomann just refereed me to your post. Did you ever switch to Peer to Peer SSL/TLS instead of Shared Key? And if you did, did it help?

Here's my finding so far - https://forum.netgate.com/topic/183854/open-vpn-2-7-site-to-site-odd-routing-issue/11

@viragomann Thank you, sir! I will be implementing next week.

Ha, that did it :-)

Thanks a lot.

We created a new server cert, installed it and were bitten by the 'VERIFY KU ERROR' bug when restarting the openVPN :-(
The certificate had been used on both servers .....

We got that fixed and updated to 2.7.0 without a problem :-)

Now considering getting a paid licence ;-)

The solution to this issue is: read the fine print.

"Enter any additional options to add to the OpenVPN server configuration here, separated by semicolon."

So I made the following changes:

push "route 172.31.4.0 255.255.255.0"; push "route 172.31.40.0 255.255.255.0"

Mind the semicolon at the end of the first line.

Thank you for letting me use this forum as a Rubber Duck

@rustem
To assign a specific IP address to a VPN client, I use the "Client Specific Overrides" tab, it's where you can select a client by its "Common Name" (the client certificate name, ou the username for VPNs utilizing password authentication), select the VPN server in the Server List, and can use the ifconfig-push directive at the end of the page, in Advanced field.
Also, the netmask that you put in ifconfig-push seems wrong, you put 255.255.255.255 instead of 255.255.255.0 (the netmaks of your tunnel network).

8bd9b26d-a50f-441e-8a69-9367336b8157-image.png
Resolved! I added the VPN range (Add Local Network Field).

@CyberMinion That was just a manual ping from ssh. The 1100 is a little underpowered (CPU-wise), so I've noticed it can take several minutes, but sometimes it will start working. Other times, it gets hung up and won't connect. So, I believe maybe my settings are correct, but it is just a little slow to get going, plus sometimes it just has trouble and reloading the process or rebooting fixes it, but it's not very quick, so it's just difficult to troubleshoot....?

My 4100 is instantaneous and works every time. I recently also reflashed/upgraded my 1100 to see if that would help, but again, I think part of the problem is that it is underpowered. Just switching between tabs/pages is a little slow, not terrible, but an indication of it's low resources.

I'm not trying to be critical, the 1100 works fine once you get everything set, but troubleshooting is a little tedious.

There have been updates to this strategy. Since this was posted, OpenVPN has introduced the --auth-gen-token option.

All that is necessary is to add auth-gen-token; to the server's custom options. No client reconfiguration is necessary.

Here is the section from the OpenVPN documentation:

--auth-gen-token [lifetime]

After successful user/password authentication, the OpenVPN server will with this option generate a temporary authentication token and push that to client. On the following renegotiations, the OpenVPN client will pass this token instead of the users password. On the server side the server will do the token authentication internally and it will NOT do any additional authentications against configured external user/password authentication mechanisms.The lifetime argument defines how long the generated token is valid. The lifetime is defined in seconds. If lifetime is not set or it is set to 0, the token will never expire.

This feature is useful for environments which is configured to use One Time Passwords (OTP) as part of the user/password authentications and that authentication mechanism does not implement any auth-token support.

@johnpoz

@johnpoz said in openvpn causing resolver performance issue?:

its possible your vpn is causing pain as well with trying to resolve, maybe they filter other dns??

confirmed they do not filter anything I can find. pretty much just pass whatever traffic you send on through.

@johnpoz said in openvpn causing resolver performance issue?:

I would let unbound either just use your normal isp connection to resolve, or if you set on using it through your vpn. Set unbound to only use that interface for its outbound, or just set it to forward to your vpn services dns server.

ISP direct resolution would present a dns leak scenario on the vpn. not an optimal configuration. I tried changing the resolver interface binding, and it had no effect on the behavior.

@johnpoz said in openvpn causing resolver performance issue?:

But the fact of just running a vpn service on your wan would/should/could not have any effect on unbound resolving.. That don't have anything to do with each other.

I rebuilt everything from factory default last night. only difference is i setup the vpn server before i defined and configured the clients for my vpn service. everything functioning exactly as before with all dns traversing the vpn service. (no forwarding, so using root servers still)

The issue went away.

Don't really understand what was happening but would like to. I have a backup of the broken configuration. I might bring it up on a vm and investigate further. What you describe about a timeout scenario, seems to make a lot of sense. Just have no clue what would be timing out at the moment.

@Bot I personally would say go with IPsec when you can, OpenVPN is cool and all but IMO just not the same vs IPsec or WireGuard, which are my two go to options. OpenVPN certainly is overall more configurable (not to be confused with capable) than the other 2 but it ends up being harder to setup, slower, and more complex.

But yeah this should be doable either way by using NAT, it's basically the only way to get two identical subnets talking over a VPN.

@alkisg
You need to configure a VPN > OpenVPN > Client Specific Override for this client to route traffic to it.

In the CSO state a certain tunnel IP for this client and set the whole network range at "Remote Network/s". For IPv4 enter "0.0.0.0/0".

Also in the server settings enter "0.0.0.0/0" at "IPv4 Remote network(s)".

@w-hackl
Rules have to be defined on the incoming interface in pfSense.

Traffic from a client side LAN device enters the LAN interface, goes out on the OpenVPN and enters the VPN interface at the server side.

So you can either block it on the clients LAN or on the server VPN interface.

@viragomann ok, the subnet is the only parameter that I can change on the OVPN Server, I will set a /30 and I will let you know