Loadbalancing + OpenVPN (Policy based routing); incoming connections not working
-
Hi guys,
I read just about every thread regarding Multi-WAN-OpenVPN-Loadbalancing but was not able to find a solution to my problem.
I have several networks with clients behind an OpenVPN-Server (Debian + OpenVPN 2.3.4). Now I want to setup site-to-site OpenVPN connections from pfSense routers with more than one WAN connection. There are many tutorials for this on the internet like: https://nguvu.org/pfsense/pfsense-multi-vpn-wan/
In short: it's creating one OpenVPN client for every WAN connection, creating an interface for every OpenVPN client, creating a gateway for every OpenVPN interface, creating a gateway group out of those gateways with the same tier and creating a firewall filter rule (with the gateway group specified) for traffic that should go to the networks behind the OpenVPN server. What I understand is that then I only have to create outbound NAT rules.. so that the sender address of the packets is the OpenVPN client address and the reply comes the same way the request was made (correct if I'm wrong).
For loadbalancing to function I have to tell my OpenVPN clients that they shouldn't pull routes from the server (I did that in the client config), because we don't want to use the routing table.. we want the firewall filter rule to direct the traffic into the right gateway group. I found out that because of this I shouldn't use monitoring IPs in the networks behind the OpenVPN server, because pfSense then adds a static route to the routing table. For this I added two or more virtual interfaces on the OpenVPN server just for monitoring IPs.
So much for that.. outgoing loadbalancing through multiple OpenVPN connection is working! :)
And now the problem: incoming traffic through the VPN tunnels reaches the client, the client replies, the packet goes back to the pfSense.. but this time the pfSense doesn't care about the firewall filter rule where I said that the packet should go out through the OpenVPN gateway group. :( I tested that with tcpdump on all hops.
Maybe I forgot something or I just don't understand what I'm doing. I hope somebody can point me in the right direction! :)
-
Policy routing is for connections being established into the interface the policy routing is on. (ie from a LAN client into a LAN interface with the policy routing rules)
If you want connections coming in from OpenVPN from arbitrary locations (locations for which there isn't a kernel route for the reply traffic with that OpenVPN server as the gateway) then you need an assigned interface on the OpenVPN instance in question.
You also need to be sure that the inbound connection is not matched by the rules on the OpenVPN tab, but is matched by rules on the assigned interface tab. That will get reply-to functioning for the reply traffic.
I cannot recommend a pfSense Gold subscription, with video lectures (Hangouts) from the guys who literally wrote the book on pfSense, enough. There is one called Advanced OpenVPN Topics (or something close to that) that covers all of this and more. And that is but one of dozens.
If that is too steep the aforementioned book is < $25. See my sig.
-
Hi Derelict,
thank you for your answer and for your recommendation. I thought about that too, but this month there's no money for that. I've got a book about pfSense already called "The definitive Guide", but theres only one section about that topic: "11.3.3. Local Services and Multi-WAN".
There it says: "In the case of traffic initiated on the Internet destined for an OPT WAN interface, pfSense automatically uses pf's reply-to directive in all WAN and OPT WAN rules, which ensures the reply traffic is routed back out the correct WAN interface."
I don't really feel this "automatically"-thing.. but maybe it's because of OpenVPN in this case? I don't know.
I tried to follow your instructions: There is an assigned interface on both OpenVPN instances, because I needed them for creating a gateway group, right? Then I checked the OpenVPN tab under firewall rules: there was no rule. Then I went to the tabs for the OpenVPN assigned interfaces and created one rule on both, where I matched the incoming traffic with an alias for those networks behind the OpenVPN server. But I don't know how to further setup this rule. Should I specify a gateway in this rule? The interface itself? Or the gateway group? I tried both.. but it didn't work.
-
I've now purchased the Gold Subscription and there it says under "Assigning OpenVPN Interfaces" that this enables several beneficial changes like: "Adds reply-to to rules on the VPN interface tab to help with return routing". But there is no auto created rule on that OpenVPN interface tab.. and I can't find any further infos about that.
Then there is a section "OpenVPN Site-to-Site with Multi-WAN and OSPF" but this a extra package is required on both sites.. and the OpenVPN server is not a pfSense, it's Debian system.
-
In order to get reply-to for connections coming into pfSense through OpenVPN, the incoming traffic MUST NOT match on the OpenVPN tab. If it does, that will prevail and that is where the state will be created and, being an interface group, not an interface, there will be no reply-to.
The traffic MUST NOT be matched by the OpenVPN tab and MUST be passed by a rule on the assigned interface tab.
How traffic is load balanced for connections coming into pfSense is determined by the other side.
-
Thank you for your patience. I found the source you were talking about: it's in the Hangout archive in the video "Advanced OpenVPN Concepts" at 22:07.
What I understood: the incoming traffic first hits the "OpenVPN" tab and then goes through the interface tabs of the corresponding OpenVPN instance. The reply-to will be added by the first rule that matches. If it matches a rule on the "OpenVPN" tab, then it gets a reply-to for the default gateway. If it matches a rule on the corresponding OpenVPN interface tab, then it gets a reply-to for this interface (exactly what we want).
The thing that wasn't clear for me was that EVERY rule will do that, no matter how it is configured. So I checked "Floating rules" and the "OpenVPN" tab, deleted all rules there and created a "pass any to any" rule on every OpenVPN instance interface tab.
Problem solved! ;)
-
Just for clarity, rules that match the OpenVPN tab do not get reply-to at all so the replies are routed according to the routing table. That usually means they go out the default gateway.
Rules matching the assigned interface tab (which means they weren't matched by the OpenVPN tab or processing would have stopped there) get reply-to on the states.
Glad it's working.
