OpenVPN relying on WAN1 when it should rely only on WAN2

abuttino

@Rico I could also report this bug and wait. Which may be better than trying to recreate my work over 4 years.

Rico

ATM we don't really know if this is a bug or some config error somewhere.
Why are you running DEV versions in production?

-Rico

abuttino

@Rico Which screenshots would you like?

I basically removed everything except for the VPN client, put the fail-over network on my USG and was too frustrated from a weeks worth of time with no results.

Here is that config:
Don't Pull
Don't Add Routes Checked (because if it's checked it takes over WAN1

tls-client;
remote-random;
tun-mtu 1500;
tun-mtu-extra 32;
mssfix 1450;
persist-key;
persist-tun;
reneg-sec 0;
remote-cert-tls server;

I had (past tense)
One outbound NAT rule in Manual mode coming from auto that created a ton of rules for NORD_VPN Interface 10.0.0.0/8 to interface address

Tried mapping it to WAN2's Alias. That didn't help.

Tried both options in Advanced>Miscellaneous for gateway, together, separate, that didn't matter.

Tried using port forwarding for 1194 to ONLY WAN2, still the same.

Iftop always had 1194 established on WAN2.

I'm willing to start over with the VPN stuff but making a whole new box will not be fun.

I am on a DEV version because I needed a feature but that's long since past and I couldn't get back to a production version.

Rico

In the OpenVPN client instance, you have Interface set to WAN2?
What happens if you pick a Failover gateway group with WAN2 Tier 1 and WAN Tier 2 ?

You don't need any WAN port forwardings with pfSense as OpenVPN client, you only need that for servers.
Outbound NAT...best is to set it to Hybrid mode, then add a mapping with Interface set to your assigned OpenVPN Client interface (do you have that added??), protocol any, source your local network(s) and destination any.

-Rico

abuttino

I read about that last night, I will try moving it to the fail-over group instead, as well as your suggested Outbound NAT Rule.

Also, I made a Hyper-V pfSense server last night and it did restore the backup. All hope it's not lost on that one.

Thanks for the continued help.

abuttino

@Rico I have had different rules added at different times, but this is a fresh start for the Outbound NAT (didn't go back yet)

Is this correct?

Rico

Looks wrong to me, Source is your local Subnet (you are not running 10.0.0.0/8 as your LAN I hope?!) and set Translation Address to Interface Address.

-Rico

abuttino

@Rico I am using 10.0.0.0/8 for LAN. I am guessing that may be an issue; the way you said it. This has never been a problem until my network started growing.

Forgot to mention "NanoStation" is actually WAN2/eth1

Only one thing is effected by it now because it runs straight into a Unifi Security Gateway that has it's NAT and Firewall disabled (basically using it for DHCP, HE.net tunnel, DPI)

If you were to tell me to change the address to a /16 address, I don't think that would be a big issue.

Bob.Dig

@abuttino Nord is using 10.8. for their Servers.

Rico

Yeah your LAN overlaps with almost any VPN providers tunnel network.

-Rico

abuttino

Fixed, testing tonight..

I presume that is why it was bringing down the entire thing, correct?

abuttino

@Rico Don't know what to try next, unplugging the cable on WAN1 still brought the VPN down..

This table continues but these are the first few entries.

WAN2 is NANOSTATION

Rico

Please post the logs here when the VPN is going down.

-Rico

abuttino

@Rico
This is from a complete cycle when I pull the plug on WAN1 to when I put it back in.

https://pastebin.com/SWHnmD3j

I fixed the issue with UPSMon, Changed IP and neglected to change the IP on the NUT server

Rico

Is your pfSense default gateway WAN2?
Try to bind the OpenVPN client Interface to localhost then.

-Rico

abuttino

@Rico No, the default gateway is WAN1. I'll give it a shot.

abuttino

@Rico

I saw that there was a LOT of errors for WAN_DHCP6 and it was trying like hell to route traffic there. I disabled it and the VPN did NOT lose connection at all. Localhost was not the trick

However, I still can't reach the internet when I am failed over for some reason. I know WAN2's connection is good because it's responsible for the VPN's connection to the server.

UPDATE:
Internet is reachable but won't use the VPN.

abuttino

I am witnessing some very odd behavior and I can't explain it.

The LAN is supposed to use the failover group but when I disconnect the cable, it goes dead.

WAN2's gateway is set to only use the right gateway for any traffic:

and, the Outbound NAT rules tell EVERYTHING how to find LAN /16:

As soon as I disconnect WAN1 everything goes down.

When I have traffic using the VPN, it's an all or nothing, it won't just use WAN2.

VPN Setup that WON'T Use the vpn on WAN1

and:
If I uncheck Don't pull routes, it will use it on WAN1, until WAN1 goes down. Then everything goes down

I am lost.

abuttino

I found that if you make a gateway group and and have plans to change it, you must remake the entire thing AND Reboot. It doesn't like changes like that. Spent an hour figuring out why it wouldn't let me fail to the bare modem.

Still can't use the VPN on WAN2 only. That seems to be the last problem which was also the original problem.

abuttino

@Rico @netgate

I believe I have uncovered 2 bugs with this.

Bug 1: OpenVPN relies on WAN Interface DHCPv6 even though it says IPv4 in the configuration for this VPN. Ignores the Interface it is supposed to look at for connection

Steps for Replication. Use an OpenVPN interface as in a Gateway group, pull plug on WAN1 where it has DHCP. Watch the VPN connection go down after the plug has been pulled.

Bug 2: The OpenVPN cannot be reached in a gateway group, uses the WAN connection instead of OpenVPN for communication but, Can be reached with a firewall rule LAN to OpenVPNGateway but NOT LAN to FailOverGateway.

Steps for Replication:
Use OpenVPN Client connection as interface, and use in a gateway Load Balance and Fail-over group. then on the firewall LAN config pick that gateway group as LAN net's gateway. It will use the connection WAN2 on to the VPN (perform a 'what's my ip' test).

I will report both of these as bugs to the bug tracking board if they are indeed bugs or if it is by design.

Thanks, in advance, for your time to look into this issue and your time to reply.