Openvpn with multi-lan



  • Hi every body. I'm starting to work with PfSense. I have configured everything I wanted: NAT, Openvpn Server, Openvpn Clients etc. Everything is working. But here is the question: I have two WAN (two diferents ISP's) and I want to connect to Openvpn server with Multi-lan, just like that:

    link1 (WAN1) –---> |                      |
                                | pfSense Router |-----------> LAN
    link2 (WAN2) -----> | Openvpn Server|

    The gateway default is WAN1. When I connect to the Openvpn through WAN1, the connection is made sucessful. But if I try to connect through WAN2, the connection doesn't work. In packet capture, I can see the packet going out through WAN1 but with the address of WAN2.
    For Example:
    WAN1 has the address 10.172.191.2
    WAN2 has the address 10.172.190.2
    In packet capture I see the packet going out with the address 10.172.190.2 but through WAN1. About static routes, I can't use it because the clients have a dynamic address.

    Someone can help me? Is it possible to do?



  • I'm not sure how you configured your openvpn but i think there are 2 scenarios available:

    • create multiple openvpn servers (one for each wan)
    • run your openvpn server on the lan interface and not on the wan interface, then portforward from both wan ips to lan ip ?

    i've never attempted the second thing, but i don't see any reason why that wouldn't work


  • Rebel Alliance Developer Netgate

    multiple servers should not be needed.

    Running with an interface of 'any' should work, but TCP would probably have a better chance of avoiding the traffic issue you're seeing.

    The option of binding to LAN and using two port forwards would work fine, I've set it up that way before with success and it works great.

    You can also forward more than just port 1194 in as well, in case your clients have trouble getting in on that port, you could also forward in 53, 80, 443, 15000, or whatever else you want.



  • Running with an interface of 'any' did not work. To put my openvpn server in the lan in unavailable becouse my lan configuration. There is more services running there. Any more sugestions?


  • Rebel Alliance Developer Netgate

    It doesn't have to run on a machine in your LAN, just pick "LAN" as the interface to bind.

    Then add port forwards for each WAN to forward from the wan IPs to the LAN IP on that port.



  • This looks ridiculous: the router can balance and failover any pass-through traffic over any number and type of WAN connections, but seems to be unable to do the same for its own traffic, either self-originated or in response to incoming requests, with either TCP, or UDP, or ICMP, or whatever. All outgoing packets generated by pfSense strictly follow the system routing table, which contains no more that 1 default route — via the gateway that is marked as the default one. This default route is not affected by gateway monitoring and is removed only when the link physically goes down, rather than replaced with another route immediately; no more gateways can be marked default, nor 0.0.0.0/0 static route can be added manually. As a result, pfSense can't act as a server (DNS, proxy, VPN) in multi-WAN configuration.

    @jimp:

    @bilizoi:

    Packets go out through WAN1, but with the address of WAN2.

    Running with an interface of “any” should work, but TCP would probably have a better chance of avoiding the traffic issue you're seeing.

    It doesn't matter on which addresses does a pfSense-hosted server listen: on any address, or on single LAN address, or on multiple addresses, — its outgoing traffic obeys the routing table. Even when a proper entry exists in the state table, even when “sticky connections” are enabled, — all reply packets go through the default route, although with correct source IP address in their header (which makes this packet to be discarded by ISP, by the way, as it spoofs foreign addresses).

    @heper:

    Run your OpenVPN server on a LAN interface rather than a WAN interface, then portforward from both WAN IPs to the LAN IP. I've never attempted that, but I don't see any reason why that wouldn't work.

    Funny, I thought the opposite: there is no reason for that to work, as the internal traffic should go through quick loopback path rather than processed thoroughly. Strangely enough, your method works indeed for both TCP and UDP traffic — such as OpenVPN server and authoritative DNS server (BIND).

    Unfortunately, this trick is not applicable to self-originated traffic — such as when OpenVPN acts as a client or in peer-to-peer mode, or DNS server in resolver mode, or Squid proxy server. As for the latter, there is a how-to posted here, which similarly recommends to use LAN or Loopback interface:@DimitriS:

    http://securite-ti.com/pfSense_Web_Proxy_with_multi-WAN_links.pdf

    • Configure Squid with “tcp_outgoing_address 127.0.0.1” directive.
    • Create floating firewall rules for outbound direction on WAN interfaces, which will redistribute traffic among gateway-group members.

    I don't know how this is supposed to work (without creating an infinite routing loop), maybe it worked in 1.2 or 2.0-RC1, but currently it has no effect, either with 127.0.0.1 or LAN address as the source, — outgoing requests are always directed via the default gateway.

    So the question is: why is not the routing table populated with all alternative routes? Neither is it done automatically, nor can it be configured manually.


  • Rebel Alliance Developer Netgate

    @Samsonov:

    This looks ridiculous: the router can balance and failover any pass-through traffic over any number and type of WAN connections, but seems to be unable to do the same for its own traffic, either self-originated or in response to incoming requests, with either TCP, or UDP, or ICMP, or whatever. All outgoing packets generated by pfSense strictly follow the system routing table, which contains no more that 1 default route — via the gateway that is marked as the default one. This default route is not affected by gateway monitoring and is removed only when the link physically goes down, rather than replaced with another route immediately; no more gateways can be marked default, nor 0.0.0.0/0 static route can be added manually. As a result, pfSense can't act as a server (DNS, proxy, VPN) in multi-WAN configuration.

    None of your previous messages in this thread mentioned outgoing traffic, just acting as a server, so nobody offered any advice on that. It's quite possible, and it does work.

    One way is under System > Advanced on the Misc. tab: "Allow default gateway switching" - the default will switch to the next available gateway if it goes down.

    The way I do it is with outbound NAT + floating rules. I make sure outbound NAT applies to my OpenVPN client connection (rule for each wan, source 'any', destination of the far side of the VPN on the applicable VPN port). Then a floating rule catches the outbound VPN traffic and makes it use a failover gateway group. Interface for OpenVPN in this case is just 'any' so it uses the right source.

    For other ideas just search this forum for one of the dozens of threads about squid+multi-wan.

    @Samsonov:

    @jimp:

    @bilizoi:

    Packets go out through WAN1, but with the address of WAN2.

    Running with an interface of “any” should work, but TCP would probably have a better chance of avoiding the traffic issue you're seeing.

    It doesn't matter on which addresses does a pfSense-hosted server listen: on any address, or on single LAN address, or on multiple addresses, — its outgoing traffic obeys the routing table. Even when a proper entry exists in the state table, even when “sticky connections” are enabled, — all reply packets go through the default route, although with correct source IP address in their header (which makes this packet to be discarded by ISP, by the way, as it spoofs foreign addresses).

    Yes that is correct. Sticky has nothing to do with that, btw.

    @Samsonov:

    @heper:

    Run your OpenVPN server on a LAN interface rather than a WAN interface, then portforward from both WAN IPs to the LAN IP. I've never attempted that, but I don't see any reason why that wouldn't work.

    Funny, I thought the opposite: there is no reason for that to work, as the internal traffic should go through quick loopback path rather than processed thoroughly. Strangely enough, your method works indeed for both TCP and UDP traffic — such as OpenVPN server and authoritative DNS server (BIND).

    Not that surprising. Doing it that way makes it obey the reply-to aspect of the incoming firewall rules and nat, and because it's bound to the LAN IP, it can't just decide to source the traffic from the wrong WAN, so the usual automatic outbound nat applies. That method has worked fine since the 1.2.x days.

    @Samsonov:

    Unfortunately, this trick is not applicable to self-originated traffic — such as when OpenVPN acts as a client or in peer-to-peer mode, or DNS server in resolver mode, or Squid proxy server. As for the latter, there is a how-to posted here, which similarly recommends to use LAN or Loopback interface:@DimitriS:

    http://securite-ti.com/pfSense_Web_Proxy_with_multi-WAN_links.pdf

    • Configure Squid with “tcp_outgoing_address 127.0.0.1” directive.
    • Create floating firewall rules for outbound direction on WAN interfaces, which will redistribute traffic among gateway-group members.

    I don't know how this is supposed to work (without creating an infinite routing loop), maybe it worked in 1.2 or 2.0-RC1, but currently it has no effect, either with 127.0.0.1 or LAN address as the source, — outgoing requests are always directed via the default gateway.

    So the question is: why is not the routing table populated with all alternative routes? Neither is it done automatically, nor can it be configured manually.

    See above, just because you didn't know how to do it doesn't mean it isn't possible.

    Also, switching your default manually is easy. Edit the other gateway, check default, save/apply. The default is now the new gateway. That works fine for extended outages.



  • @jimp:

    None of messages in this thread mentioned outgoing traffic, just acting as a server, so nobody offered any advice on that.

    By “outgoing” traffic I meant the reply packets also, not just self-originated requests, — at least in the first half of my message. Original poster wrote about the same thing, as long as I understand.

    Under System > Advanced on the Misc. tab: “Allow default gateway switching” — the default will switch to the next available gateway, if it goes down.

    Yes, we already discussed that with you. And the gateway switching does work. The problem is when does it consider the default gateway to be down: if a cable is physically disconnected, so PPP session is timed out, a gateway switch indeed occurs; but if the link is logically alive, then the routing table is not altered — even though gateway monitoring considers the gateway to be offline.

    Also, switching your default manually is easy. That works fine for extended outages.

    My idea was to build a completely automatic router, which doesn't require human interaction for months. Manual reconfiguration might be appropriate for corporate environments with dedicated 24x7 administration, but is quiet tough for home use with unstable links that can break a couple of times per day for a matter of several minutes to hours.

    The way I do it is with outbound NAT + floating rules for each WAN: source “any”, destination of the far side of the VPN on the applicable VPN port. Then a floating rule catches the outbound VPN traffic and makes it use a failover gateway group.

    It looks exactly what I was trying, but noticed no effect at all: neither was it helping to balance the self-originated traffic, nor did it break the pass-through forwarding. So I am wondering what's the difference between our setups? Maybe some additional sophisticated rules are required for manual outbound NAT?

    When I enable logging for those floating rules, I see corresponding lines in firewall log:

    @65 pass out log quick on WANifcgrp route-to { (pppoe0 …address...), (pppoe1 ...address...) } round-robin sticky-address inet from any to ! <net_std_private:4>flags S/SA keep state label "USER_RULE: Balance to WAN"</net_std_private:4>

    So the rules are matched, but they are always logged as “captured on interface: WAN1” (is this the expected behavior?) and obviously do not affect the traffic, as connections are still made via WAN1 — any website with “What is my IP?” page proves it.



  • @heper:

    Run your OpenVPN server on a LAN interface rather than a WAN interface, then portforward from both WAN IPs to the LAN IP.

    By the way, this isn't applicable to ICMP pings, as pfSense doesn't support NAT for ICMP; so all ping replies from WAN2 actually go via WAN1.

    @jimp:

    Doing it that way makes it obey the reply-to aspect of the incoming firewall rules and NAT…

    Just to be sure, I double-checked the “Disable reply-to on WAN rules” option.

    With Multi-WAN you generally want to ensure traffic leaves the same interface it arrives on,
    hence reply-to is added automatically by default. When using bridging, you must disable this behavior
    if the WAN gateway IP is different from the gateway IP of the hosts behind the bridged interface.
    

    Of course, it was not checked now nor ever before. So why doesn't it work as expected? Why is NAT needed for pfSense-hosted services, instead of plain firewall rules? (This option is located in “Firewall Advanced” section rather than “Network Address Translation”, so I assume that it refers to all traffic, not just NAT'ted one.)



  • @Samsonov:

    @jimp:

    Under System > Advanced on the Misc. tab: “Allow default gateway switching” — the default will switch to the next available gateway, if it goes down.

    Yes, we already discussed that with you. And the gateway switching does work. The problem is when does it consider the default gateway to be down: if a cable is physically disconnected, so PPP session is timed out, a gateway switch indeed occurs; but if the link is logically alive, then the routing table is not altered — even though gateway monitoring considers the gateway to be offline.

    Today I found out that even that case was not the worst. Consider this situation:

    • MAN1 (the underlying Ethernet link for WAN1) is up, although its intra-city gateway is not reachable — but it's pure separate thing and affects local traffic only.
    • WAN1 (PPPoE over MAN1) is down, because PPP session can't be established — as its PPPoE server is probably having the same problems as the MAN1 gateway, 'cause they are located next to each other.
    • MAN2 and WAN2 are perfectly up and running.
    • WAN3 (tier 2) is also up.

    In this situation, while the gateway group “(WAN1+WAN2) or (WAN3)” perfectly routes the traffic via WAN2, but the default route via WAN1 is deleted and not inserted for WAN2 or WAN3! And without any default rule, pfSense stops functioning as a DNS resolver, HTTP proxy or VPN client.



  • Hi everyone (again). It seens a bug of PFSense, no?? Does anyone have a sugestion do make this cenario work?


  • Rebel Alliance Developer Netgate

    For lack of a clearer way to convey it:

    It's not a bug, it's just how the network functions in the OS.

    When the traffic tries to leave, it follows the routing table to pick the interface the traffic will leave from. If there is no static route for the other IP, it goes out the default gateway. pf is not yet involved.

    As the traffic tries to leave via the default gateway, pf can catch it, and using floating rules, it can redirect that traffic to use a different gateway.

    There is no magic "go this way" direction to give a packet before it consults the routing table. Even if a service is bound to an IP on a different interface/WAN, that doesn't mean it will leave that way. If a service is bound to 'any' then for TCP it will reply from the address that the client connected to. If it's UDP, it will usually reply from the "closest" interface, which in most cases is the default route.

    So unless you have default route switching enabled (which is still buggy, but mostly works, there is an open ticket, notably it has issues with ppp links), if your default route WAN is down, it will still attempt to send firewall-initiated traffic that way, or if there is no default, the OS behaves badly. Most of that is FreeBSD's fault, though we have tried to work around it over the years.

    The NAT is required because, in the case of things like UDP or a TCP connection initiated from that interface, in order for pf to shove it out the other WAN, it needs to have the IP of that WAN applied as it leaves that way. You can't send traffic out WAN2 with WAN1's IP, and it wouldn't come back anyhow because (a) the ISP on WAN2 would probably drop it and (b) WAN1 is down so it can't receive the traffic.


Log in to reply