Different Interfaces/Gateways Using Same IP Address
-
@stephenw10 @viragomann @Bob-Dig Thanks for chiming in here. As it appears that there might be some conflicting opinions here I think it might me best if I can elaborate on what I tried so far. Maybe if I'm more explicit/detailed it can help clear things up.
So in pfsense I was going to set up 3 Wirerguard tunnels to 3 different VPN provider endpoints. Each of the endpoints are located in different states in the US, they are all housed at different datacenters, and all of the datacenters are managed by different providers. Each of the servers has a unique public IP address.
The Wireguard Tunnels are easy to setup and are essentially identical other than the Listen Ports being unique to each tunnel as well as the keys.
The Wireguard Peers are also essentially the same with the only differences being the Endpoint IP address which are set to the unique IP address of the VPN servers at each of the data centers. Obviously the Public Key is unique in each peer. The allowed IPs is set to 0.0.0.0/0.
I set up the initial interface for the first Wireguard tunnel using a static IP address of 10.2.0.2/32 which is what the VPN provider indicated I should use based in their wireguard configuration. I then created an Upstream Gateway using the same IP address (10.2.0.2) as the interface. I then bound the Gateway to the Interface.
Finally, I created a outbound NAT rule for the new interface, using my LAN address as the Source network and "Any" as the destination.
The above steps all work correctly. I can policy route out the Wireguard tunnel with no issues via the new interface and gateway.
The problems all start when I try to set up the second or third tunnels. Specifically, the IP address that the VPN provider specifies to use for the Interface and Gateway for tunnel 2 & 3 is also 10.2.0.2/32. In trying to set up the additional Interfaces I get an error indicating that 10.2.0.2/32 is being used by, or overlaps with, the same IP range assigned to the first tunnel.
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
Is the gateway IP also the same? That would break the routing in pfSense if so.
I'm not sure which gateway IP you're referring to here. If it's the WAN gateway to my ISP, then yes all 3 tunnels are going out that gateway. It's the only gateway I have to the internet.
If it's the IP of the gateway's for the 3 wireguard tunnels, I am able to set gateways up with 3 different IP addresses. But the problem I'm having is that I still can't set up the interfaces for tunnels 2 & 3 because they require the 10.2.0.2 IP address and therefore I can't bind the gateways (2 or 3) to the appropriate interfaces.
What I was hoping could be done would be that I could set up the interfaces/gateways for tunnels 2 and 3 with different IP ranges, say 10.2.0.3/32 and 10.2.0.4/32. Then have the outgoing traffic translated (somehow???) to 10.2.0.2 as it goes out the gateway. If this is foolish thinking please excuse my ignorance. I'm definitely a bit over my head here. But I'm certainly grateful for either knowing it can't be done or, learning any new thing that you guys can point me to in making this work.
-
@dma_pf said in Different Interfaces/Gateways Using Same IP Address:
I'm not sure which gateway IP you're referring to here.
It's the WG tunnel gateways. You can't policy route to 10.2.0.2 if that is 3 gateways. They need to be different.
Wireguard doesn't actually need/use a tunnel subnet though it uses cryptokey routing. So if you can define different local IPs and gateways that satisfy the FreeBSD routing requirements you might be able to just change the outbound NAT rule to to use 10.2.0.2 on all of them. Try it.
Steve
-
@dma_pf said in Different Interfaces/Gateways Using Same IP Address:
I am able to set gateways up with 3 different IP addresses.
That sounds promising. Please keep us updated because it is a hassle with many of the privacy VPNs using WireGuard. There are exceptions like mullvad though.
-
@bob-dig said in Different Interfaces/Gateways Using Same IP Address:
That sounds promising. Please keep us updated because it is a hassle with many of the privacy VPNs using WireGuard. There are exceptions like mullvad though.
I've been using IVPN for years (since the wireguard package came out) with no issues whatsoever. But they provide a unique IP address for each interface.
I just switched some email accounts to a paid Protonmail plan and they include 10 connections via their VPN as part of the email package. So I'm trying to see if I can get their VPN working with the thought of possibly dropping IVPN and saving $100/year.
So I do have an update.
I configured 3 different Interfaces and Gateways. I used these IP addresses for them:
10.2.0.2/32 (the one provided by proton)
10.2.0.3/32
10.2.0.4/32I can see in the Wireguard status page that all of the tunnels to the endpoints are shown as up. I did a little bit of digging around on cryptokey routing and from my reading (and Steve's earlier comments) this would be expected as the keys for the tunnel/peer would control the establishment of the tunnel. The IP addresses have noting to do with the actual tunnel being connected. But once the tunnel is connected then the IP address is checked and if it is on their allowed IP list then traffic can enter their server.
With that in mind, 10.2.0.2/32 (the one provided by proton) has no issue routing traffic. The other 2 do not with a typical NAT Rule.
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
Wireguard doesn't actually need/use a tunnel subnet though it uses cryptokey routing. So if you can define different local IPs and gateways that satisfy the FreeBSD routing requirements you might be able to just change the outbound NAT rule to to use 10.2.0.2 on all of them. Try it.
Steve
Since I was not able to route via the typical NAT rule I tried modifying the outbound NAT rule as follows:
For the interface I used the 10.2.0.3
For Source I used my LAN network IP range.
For Desination I used Any
For NAT Address I did the following:- I first created a new Host Alias with a single IP of 10.2.0.2 (the IP given by Proton)
- In the NAT Outbound rule I changed Translation/Address to the Host Alias I created above. Translation Pool Options was left at "Default" and Port or Range is blank.
I then saved/applied the rule and flushed the states. I then policy routed a laptop on the LAN to route out the 10.2.0.3 gateway and it can't get to any website. I verified from the command prompt that the laptop can resolve domain names. I tried to ping from the command prompt on the laptop and it is always successful getting the first ping and then fails on the remaining 3 tries. That happened on all the IP addresses/domains I tried to ping.
I tried rebooting pfsense just in case something was stuck with all the fooling around I've been doing on it. But no luck.
That's as far as I've gotten.
-
Check the state whilst you do that. Make sure traffic is actually going to that interface and that it's being translated correctly.
There are a number of ways this could fail really....
-
@stephenw10 The Proton_NY gateway is the 10.2.0.3 IP address range. So it looks to me that the NAT rule is routing it out on 10.2.0.2 which is what Proton would be expecting to see.
-
Well there is some traffic coming back. I'd try running a pcap on that interface and see what's coming back there. Or in fact if that traffic is actually using that interface.
-
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
Well there is some traffic coming back. I'd try running a pcap on that interface and see what's coming back there. Or in fact if that traffic is actually using that interface.
I did some further testing as you suggested. I set the laptop to policy route out the 10.2.0.3 (tunnel #2 Proton_NY) interface. I then ran a pcap on that interface for for ICMP packets by doing a ping test from the laptop. The pcap showed absolutely no packets on the 10.2.0.3 interface.
I then did the same test, but this time I ran the pcap on the 10.2.0.2 interface and sure enough the ping packets are showing up as routing out that interface.
I double checked all my setting and am pretty sure they are correct. But I'll post some pictures to make sure.
Pcap on the 10.2.0.2 interface
Policy Routing Laptop to the 10.2.0.3 Interface
10.2.0.2 Interface Config
10.2.0.2 Gateway Config
10.2.0.3 Interface Config
10.2.0.3 Gateway Config
Outbound NAT Rules for 10.2.0.2 and 10.2.0.3 Interfaces
Detail Of Outbound NAT Rule Config For 10.2.0.3 Interface
Proton_Interface_Address Alias Config
Thanks for your help. Let me know what else I can do to help figure this out!
-
You may be hitting a route-to problem. Outbound traffic sourced from 10.2.0.2 will always be forced via the 10.2.0.2 gateway if route-to is applied. However this situation is complex. Check the rules file /tmp/rules.debug to see if and where that is applied.
You might also check the two states that are created when you run that ping to see what rules created them. Especially the outbound rule. You will likely have to run
pfctl -vvssto see that. That can be a lot of output!Steve
-
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
You may be hitting a route-to problem. Outbound traffic sourced from 10.2.0.2 will always be forced via the 10.2.0.2 gateway if route-to is applied. However this situation is complex. Check the rules file /tmp/rules.debug to see if and where that is applied.
I checked the rules.debug file and do see the "route-to" applied to the 10.2.0.2 gateway:
GWProton_NJ_WGV4 = " route-to ( tun_wg4 10.2.0.2 ) "I can't play with it right now, but do you think if I was to change that gateway/interface to another unique IP, say 10.2.0.5, it would work? At that point there would not be any gateway/interface with a "route-to" rule for 10.2.0.2.
-
I would certainly try that first, yes.
-
@stephenw10 Hey Steve, I switched the 10.2.0.2 interface to 10.2.0.5 and it worked! I now have 3 different tunnels that are all routing traffic at 780-820 Mbits/sec up and down. I've bound them all together as a Gateway Group and policy routing is working very well.
Thanks for your help. I would never have figured that out on my own!
The only outstanding issue I have is that I if I use the 3 interfaces as the outgoing interfaces for DNS Resolver (not forwarding) it will not resolve any domains. I have added those interfaces in the DNS Resolver Access List. I have it set up that way with the interfaces from my prior VPN provider (IVPN) and I have no issues using IVPN to resolve via the root servers. But I can't seem to make it happen via Proton.
-
Aha, that's a great result!
I would check the states and also run a pcap on the WAN for DNS traffic.
You may well be hitting this: https://redmine.pfsense.org/issues/13420
Or at least something related to that. It's fixed now in 23.01/2.7.Steve
-
Hi There,
I'm trying to do the same thing with Nord.
Can you possibly post your final configuration? I've tried following your example but seems secondary connections wont work for me.
If I can get this working, I will mostly likely post a guide to reddit, as a few people are trying to do the same thing.
Thanks!
-
@dma_pf I also wouldn't mine a short summary, what you have to do differently to a normal setup, to get this working.
-
You outbound NAT the traffic to the IP the remote side is expecting.
The key here is that you have to outbound NAT all the tunnels to that. None of them can be using that IP natively because doing so will cause route-to tagging to pass all traffic via that.
Steve
-
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
You outbound NAT the traffic to the IP the remote side is expecting.
Interesting, so only outbound NAT in that way has to be applied, nice.
Will have to try that for myself, maybe I can abandon my fleet of OpenWRT-VMs.
-
@lex33 Everything is configured as shown in my pictures above with the exception as follows:
The key thing that worked for me is that the 3 interfaces/gateways have to have unique IP addresses and they can't be the IP address that the VPN provider wants you to use.
So in my case, ProtonVPN wants all connections to all their servers to use 10.2.0.2/32. So I set my 3 interfaces/gateways to use the IPs of 10.2.0.3/32, 10.2.0.4/32 & 10.2.0.5/32. Then set the NAT for each Interface as I showed in my picture above.
In my case, using the 10.2.0.2 IP for any of the interfaces messed up the NAT due to the "reply-to" rule that's automatically applied to that interface. The reply-to rule preempts the custom NAT rules and would return packets back to the 10.2.0.2 interface. Big kudos to @stephenw10 for figuring that out!
(Way over my pay grade)I'm still having the issue with the DNS, but I haven't had the time to mess with it. For right now I'm still routing resolver out through my account with IVPN (I paid for a yearly subscription and it ends in March).
I have discovered another issue since my last post. For some reason when I stream music, via either a single connection or through a gateway group, the songs play perfectly for about half the song. But then somewhere in the last half of the song it will just jump to the next song in the playlist. I've definitely been able to isolate this to the VPN connections to ProtonVPN. I haven't had the time to see if this happens when streaming video. I have no issues whatsoever when streaming via IVPN.
-
Maybe the streaming service detecting you're coming from a VPN IP?
-
@stephenw10 I don't think that's it. I've tried 3 different ProtonVPN servers in distinctly different geographical areas. In addition when I streamed through IVPN I used it with their servers located in the exact same data centers as ProtonVPN's servers.
I'm also having the same exact issues with 2 different streaming services, Tidal and Qobuz.
As best as I can tell, it seems like packets are just not getting to the devices. With Tidal it will just jump to the next song, like as if the current song had ended. With Qobuz I can see the song's remaining time indicator kind of wiggle back and forth a little. It will do that for a bit and then it's like the data catches up with it and it starts playing again from where it left off.
I reached out to ProtonVPN about it and they suggested I install their app on my device...lol.
