OpenVPN relying on WAN1 when it should rely only on WAN2


  • I have been at this for several days and I can't figure it out.

    Rght now, OpenVPN client is set up to use WAN2 but as soon as I pull the plug on WAN1 the failover does take place but it also takes down the OpenVPN connection.

    What can I do to stop this from happening?

    I looked at iftop and the connection is established between WAN2 and the OpenVPN connection. It just won't use it when it comes time to.


  • Ok, I'll give someone $50 if they can help me.

    Just need to know how to keep OpenVPN from relying on WAN1. If WAN1 goes down, it SHOULD NOT take WAN2's connection to the VPN Client (NordVPN) down with it.

    It's funny, cause, you can see that it is using WAN2, but when WAN1 goes down, the VPN is lost.

    @netgate is this an issue with the beta? Is that why nobody understands what is the issue, because, it should already work?

  • LAYER 8 Rebel Alliance

    Which pfSense version are you running?
    Do you have State Killing on Gateway Failure enabled in System > Advanced > Miscellaneous ?

    -Rico


  • @Rico

    2.5.0-DEVELOPMENT (amd64)
    built on Tue Aug 11 07:00:56 EDT 2020

    I checked the box, will let you know in a couple hours when I try it.

  • LAYER 8 Rebel Alliance

    It should not be checked, I would expect exactly your problem with this option enabled...this is why I'm asking.
    Same problem with 2.4.5-p1 ?

    -Rico


  • @Rico

    Ok. It was not checked I'll leave it unchecked.

    I will try p1 tonight. Can't take it down in the middle of the day.

  • LAYER 8 Rebel Alliance

    I have a lot Sites running 2.4.4-p3 and 2.4.5-p1 with Multi WAN as OpenVPN Client and never had any issue with one WAN failing taking down the whole VPN tunnel.
    Can you share your config?

    -Rico


  • @Rico what's the best way to do this? Take a backup?

  • LAYER 8 Rebel Alliance

    I think you can't revert a 2.5.0 backup back to 2.4.4 or 2.4.5 - but I never tried that.
    Maybe you have a older backup, worst case is building it again from scratch.

    -Rico


  • @Rico I meant giving you the config.

    However, that's good to know.

    HAProxy, several domains, several hosts.

    I will try to get the most recent version and see if that fixes it. Then, maybe, a start from scratch if it won't restore.

    I hope they didn't change much and can go back to p1.


  • @Rico I could also report this bug and wait. Which may be better than trying to recreate my work over 4 years.

  • LAYER 8 Rebel Alliance

    ATM we don't really know if this is a bug or some config error somewhere.
    Why are you running DEV versions in production?

    -Rico


  • @Rico Which screenshots would you like?

    I basically removed everything except for the VPN client, put the fail-over network on my USG and was too frustrated from a weeks worth of time with no results.

    Here is that config:
    Don't Pull
    Don't Add Routes Checked (because if it's checked it takes over WAN1

    tls-client;
    remote-random;
    tun-mtu 1500;
    tun-mtu-extra 32;
    mssfix 1450;
    persist-key;
    persist-tun;
    reneg-sec 0;
    remote-cert-tls server;

    I had (past tense)
    One outbound NAT rule in Manual mode coming from auto that created a ton of rules for NORD_VPN Interface 10.0.0.0/8 to interface address

    Tried mapping it to WAN2's Alias. That didn't help.

    Tried both options in Advanced>Miscellaneous for gateway, together, separate, that didn't matter.

    Tried using port forwarding for 1194 to ONLY WAN2, still the same.

    Iftop always had 1194 established on WAN2.

    I'm willing to start over with the VPN stuff but making a whole new box will not be fun.

    I am on a DEV version because I needed a feature but that's long since past and I couldn't get back to a production version.

  • LAYER 8 Rebel Alliance

    In the OpenVPN client instance, you have Interface set to WAN2?
    What happens if you pick a Failover gateway group with WAN2 Tier 1 and WAN Tier 2 ?

    You don't need any WAN port forwardings with pfSense as OpenVPN client, you only need that for servers.
    Outbound NAT...best is to set it to Hybrid mode, then add a mapping with Interface set to your assigned OpenVPN Client interface (do you have that added??), protocol any, source your local network(s) and destination any.

    -Rico


  • I read about that last night, I will try moving it to the fail-over group instead, as well as your suggested Outbound NAT Rule.

    Also, I made a Hyper-V pfSense server last night and it did restore the backup. All hope it's not lost on that one.

    Thanks for the continued help.


  • @Rico I have had different rules added at different times, but this is a fresh start for the Outbound NAT (didn't go back yet)

    Is this correct?
    f4bb3c2c-2f7b-44c6-8b99-bee82deff0e2-image.png

  • LAYER 8 Rebel Alliance

    Looks wrong to me, Source is your local Subnet (you are not running 10.0.0.0/8 as your LAN I hope?!) and set Translation Address to Interface Address.

    -Rico


  • @Rico I am using 10.0.0.0/8 for LAN. I am guessing that may be an issue; the way you said it. This has never been a problem until my network started growing.

    Forgot to mention "NanoStation" is actually WAN2/eth1

    Only one thing is effected by it now because it runs straight into a Unifi Security Gateway that has it's NAT and Firewall disabled (basically using it for DHCP, HE.net tunnel, DPI)

    If you were to tell me to change the address to a /16 address, I don't think that would be a big issue.


  • @abuttino Nord is using 10.8. for their Servers.

  • LAYER 8 Rebel Alliance

    Yeah your LAN overlaps with almost any VPN providers tunnel network.

    -Rico


  • Fixed, testing tonight..

    I presume that is why it was bringing down the entire thing, correct?


  • @Rico Don't know what to try next, unplugging the cable on WAN1 still brought the VPN down..

    97732585-9599-44a2-a778-c5d23861ab26-image.png

    This table continues but these are the first few entries.

    WAN2 is NANOSTATION

  • LAYER 8 Rebel Alliance

    Please post the logs here when the VPN is going down.

    -Rico


  • @Rico
    This is from a complete cycle when I pull the plug on WAN1 to when I put it back in.

    https://pastebin.com/SWHnmD3j

    I fixed the issue with UPSMon, Changed IP and neglected to change the IP on the NUT server

  • LAYER 8 Rebel Alliance

    Is your pfSense default gateway WAN2?
    Try to bind the OpenVPN client Interface to localhost then.

    -Rico


  • @Rico No, the default gateway is WAN1. I'll give it a shot.


  • @Rico

    I saw that there was a LOT of errors for WAN_DHCP6 and it was trying like hell to route traffic there. I disabled it and the VPN did NOT lose connection at all. Localhost was not the trick

    However, I still can't reach the internet when I am failed over for some reason. I know WAN2's connection is good because it's responsible for the VPN's connection to the server.

    UPDATE:
    Internet is reachable but won't use the VPN.


  • I am witnessing some very odd behavior and I can't explain it.

    The LAN is supposed to use the failover group but when I disconnect the cable, it goes dead.
    7724a131-449f-4e08-9405-d8a0fb234c27-image.png

    WAN2's gateway is set to only use the right gateway for any traffic:
    f9aae79d-f077-4a07-9be1-78761a5f5407-image.png

    and, the Outbound NAT rules tell EVERYTHING how to find LAN /16:
    e4514be8-6e6c-47ab-bed6-5dfc4ca35d88-image.png

    As soon as I disconnect WAN1 everything goes down.

    When I have traffic using the VPN, it's an all or nothing, it won't just use WAN2.

    VPN Setup that WON'T Use the vpn on WAN1
    9f5760db-3291-4fc9-9031-f41e1ce05e20-image.png

    and:
    If I uncheck Don't pull routes, it will use it on WAN1, until WAN1 goes down. Then everything goes down

    I am lost.


  • I found that if you make a gateway group and and have plans to change it, you must remake the entire thing AND Reboot. It doesn't like changes like that. Spent an hour figuring out why it wouldn't let me fail to the bare modem.

    Still can't use the VPN on WAN2 only. That seems to be the last problem which was also the original problem.


  • @Rico @netgate

    I believe I have uncovered 2 bugs with this.

    Bug 1: OpenVPN relies on WAN Interface DHCPv6 even though it says IPv4 in the configuration for this VPN. Ignores the Interface it is supposed to look at for connection

    Steps for Replication. Use an OpenVPN interface as in a Gateway group, pull plug on WAN1 where it has DHCP. Watch the VPN connection go down after the plug has been pulled.

    Bug 2: The OpenVPN cannot be reached in a gateway group, uses the WAN connection instead of OpenVPN for communication but, Can be reached with a firewall rule LAN to OpenVPNGateway but NOT LAN to FailOverGateway.

    Steps for Replication:
    Use OpenVPN Client connection as interface, and use in a gateway Load Balance and Fail-over group. then on the firewall LAN config pick that gateway group as LAN net's gateway. It will use the connection WAN2 on to the VPN (perform a 'what's my ip' test).

    I will report both of these as bugs to the bug tracking board if they are indeed bugs or if it is by design.

    Thanks, in advance, for your time to look into this issue and your time to reply.


  • @Netgate-Steve can you take a look at this and tell me if it warrants a bug report, please.