Different Interfaces/Gateways Using Same IP Address
-
@bob-dig said in Different Interfaces/Gateways Using Same IP Address:
That sounds promising. Please keep us updated because it is a hassle with many of the privacy VPNs using WireGuard. There are exceptions like mullvad though.
I've been using IVPN for years (since the wireguard package came out) with no issues whatsoever. But they provide a unique IP address for each interface.
I just switched some email accounts to a paid Protonmail plan and they include 10 connections via their VPN as part of the email package. So I'm trying to see if I can get their VPN working with the thought of possibly dropping IVPN and saving $100/year.
So I do have an update.
I configured 3 different Interfaces and Gateways. I used these IP addresses for them:
10.2.0.2/32 (the one provided by proton)
10.2.0.3/32
10.2.0.4/32I can see in the Wireguard status page that all of the tunnels to the endpoints are shown as up. I did a little bit of digging around on cryptokey routing and from my reading (and Steve's earlier comments) this would be expected as the keys for the tunnel/peer would control the establishment of the tunnel. The IP addresses have noting to do with the actual tunnel being connected. But once the tunnel is connected then the IP address is checked and if it is on their allowed IP list then traffic can enter their server.
With that in mind, 10.2.0.2/32 (the one provided by proton) has no issue routing traffic. The other 2 do not with a typical NAT Rule.
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
Wireguard doesn't actually need/use a tunnel subnet though it uses cryptokey routing. So if you can define different local IPs and gateways that satisfy the FreeBSD routing requirements you might be able to just change the outbound NAT rule to to use 10.2.0.2 on all of them. Try it.
Steve
Since I was not able to route via the typical NAT rule I tried modifying the outbound NAT rule as follows:
For the interface I used the 10.2.0.3
For Source I used my LAN network IP range.
For Desination I used Any
For NAT Address I did the following:- I first created a new Host Alias with a single IP of 10.2.0.2 (the IP given by Proton)
- In the NAT Outbound rule I changed Translation/Address to the Host Alias I created above. Translation Pool Options was left at "Default" and Port or Range is blank.
I then saved/applied the rule and flushed the states. I then policy routed a laptop on the LAN to route out the 10.2.0.3 gateway and it can't get to any website. I verified from the command prompt that the laptop can resolve domain names. I tried to ping from the command prompt on the laptop and it is always successful getting the first ping and then fails on the remaining 3 tries. That happened on all the IP addresses/domains I tried to ping.
I tried rebooting pfsense just in case something was stuck with all the fooling around I've been doing on it. But no luck.
That's as far as I've gotten.
-
Check the state whilst you do that. Make sure traffic is actually going to that interface and that it's being translated correctly.
There are a number of ways this could fail really....
-
@stephenw10 The Proton_NY gateway is the 10.2.0.3 IP address range. So it looks to me that the NAT rule is routing it out on 10.2.0.2 which is what Proton would be expecting to see.
-
Well there is some traffic coming back. I'd try running a pcap on that interface and see what's coming back there. Or in fact if that traffic is actually using that interface.
-
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
Well there is some traffic coming back. I'd try running a pcap on that interface and see what's coming back there. Or in fact if that traffic is actually using that interface.
I did some further testing as you suggested. I set the laptop to policy route out the 10.2.0.3 (tunnel #2 Proton_NY) interface. I then ran a pcap on that interface for for ICMP packets by doing a ping test from the laptop. The pcap showed absolutely no packets on the 10.2.0.3 interface.
I then did the same test, but this time I ran the pcap on the 10.2.0.2 interface and sure enough the ping packets are showing up as routing out that interface.
I double checked all my setting and am pretty sure they are correct. But I'll post some pictures to make sure.
Pcap on the 10.2.0.2 interface
Policy Routing Laptop to the 10.2.0.3 Interface
10.2.0.2 Interface Config
10.2.0.2 Gateway Config
10.2.0.3 Interface Config
10.2.0.3 Gateway Config
Outbound NAT Rules for 10.2.0.2 and 10.2.0.3 Interfaces
Detail Of Outbound NAT Rule Config For 10.2.0.3 Interface
Proton_Interface_Address Alias Config
Thanks for your help. Let me know what else I can do to help figure this out!
-
You may be hitting a route-to problem. Outbound traffic sourced from 10.2.0.2 will always be forced via the 10.2.0.2 gateway if route-to is applied. However this situation is complex. Check the rules file /tmp/rules.debug to see if and where that is applied.
You might also check the two states that are created when you run that ping to see what rules created them. Especially the outbound rule. You will likely have to run
pfctl -vvssto see that. That can be a lot of output!Steve
-
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
You may be hitting a route-to problem. Outbound traffic sourced from 10.2.0.2 will always be forced via the 10.2.0.2 gateway if route-to is applied. However this situation is complex. Check the rules file /tmp/rules.debug to see if and where that is applied.
I checked the rules.debug file and do see the "route-to" applied to the 10.2.0.2 gateway:
GWProton_NJ_WGV4 = " route-to ( tun_wg4 10.2.0.2 ) "I can't play with it right now, but do you think if I was to change that gateway/interface to another unique IP, say 10.2.0.5, it would work? At that point there would not be any gateway/interface with a "route-to" rule for 10.2.0.2.
-
I would certainly try that first, yes.
-
@stephenw10 Hey Steve, I switched the 10.2.0.2 interface to 10.2.0.5 and it worked! I now have 3 different tunnels that are all routing traffic at 780-820 Mbits/sec up and down. I've bound them all together as a Gateway Group and policy routing is working very well.
Thanks for your help. I would never have figured that out on my own!
The only outstanding issue I have is that I if I use the 3 interfaces as the outgoing interfaces for DNS Resolver (not forwarding) it will not resolve any domains. I have added those interfaces in the DNS Resolver Access List. I have it set up that way with the interfaces from my prior VPN provider (IVPN) and I have no issues using IVPN to resolve via the root servers. But I can't seem to make it happen via Proton.
-
Aha, that's a great result!
I would check the states and also run a pcap on the WAN for DNS traffic.
You may well be hitting this: https://redmine.pfsense.org/issues/13420
Or at least something related to that. It's fixed now in 23.01/2.7.Steve
-
Hi There,
I'm trying to do the same thing with Nord.
Can you possibly post your final configuration? I've tried following your example but seems secondary connections wont work for me.
If I can get this working, I will mostly likely post a guide to reddit, as a few people are trying to do the same thing.
Thanks!
-
@dma_pf I also wouldn't mine a short summary, what you have to do differently to a normal setup, to get this working.
-
You outbound NAT the traffic to the IP the remote side is expecting.
The key here is that you have to outbound NAT all the tunnels to that. None of them can be using that IP natively because doing so will cause route-to tagging to pass all traffic via that.
Steve
-
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
You outbound NAT the traffic to the IP the remote side is expecting.
Interesting, so only outbound NAT in that way has to be applied, nice.
Will have to try that for myself, maybe I can abandon my fleet of OpenWRT-VMs.
-
@lex33 Everything is configured as shown in my pictures above with the exception as follows:
The key thing that worked for me is that the 3 interfaces/gateways have to have unique IP addresses and they can't be the IP address that the VPN provider wants you to use.
So in my case, ProtonVPN wants all connections to all their servers to use 10.2.0.2/32. So I set my 3 interfaces/gateways to use the IPs of 10.2.0.3/32, 10.2.0.4/32 & 10.2.0.5/32. Then set the NAT for each Interface as I showed in my picture above.
In my case, using the 10.2.0.2 IP for any of the interfaces messed up the NAT due to the "reply-to" rule that's automatically applied to that interface. The reply-to rule preempts the custom NAT rules and would return packets back to the 10.2.0.2 interface. Big kudos to @stephenw10 for figuring that out!
(Way over my pay grade)I'm still having the issue with the DNS, but I haven't had the time to mess with it. For right now I'm still routing resolver out through my account with IVPN (I paid for a yearly subscription and it ends in March).
I have discovered another issue since my last post. For some reason when I stream music, via either a single connection or through a gateway group, the songs play perfectly for about half the song. But then somewhere in the last half of the song it will just jump to the next song in the playlist. I've definitely been able to isolate this to the VPN connections to ProtonVPN. I haven't had the time to see if this happens when streaming video. I have no issues whatsoever when streaming via IVPN.
-
Maybe the streaming service detecting you're coming from a VPN IP?
-
@stephenw10 I don't think that's it. I've tried 3 different ProtonVPN servers in distinctly different geographical areas. In addition when I streamed through IVPN I used it with their servers located in the exact same data centers as ProtonVPN's servers.
I'm also having the same exact issues with 2 different streaming services, Tidal and Qobuz.
As best as I can tell, it seems like packets are just not getting to the devices. With Tidal it will just jump to the next song, like as if the current song had ended. With Qobuz I can see the song's remaining time indicator kind of wiggle back and forth a little. It will do that for a bit and then it's like the data catches up with it and it starts playing again from where it left off.
I reached out to ProtonVPN about it and they suggested I install their app on my device...lol.
-
@dma_pf Maybe IPv6 is used somehow on your side?
I want to test it but have to change so much other things right now, don't have time, yet. And I will not delete my fleet of Vms for sure. -
@dma_pf said in Different Interfaces/Gateways Using Same IP Address:
I'm still having the issue with the DNS, but I haven't had the time to mess with it. For right now I'm still routing resolver out through my account with IVPN (I paid for a yearly subscription and it ends in March).
Maybe because you only have one subnet as source for the outbound-NAT?
And one other thought:
The best practice is to use strict rules when utilizing static port to avoid any potential conflict if two local hosts use the same source port to talk to the same remote server and port using the same external IP address.
WireGuard on pfSense is using a static Source Port by default. I don't know if this is a must for WG in general. Maybe this has to be considered. So I would try to use static port outbound NAT.
-
Having DNS queries going out over a different VPN could definitely cause issues like this. Services that 'detect' VPN use often use DNS queries to do it.
Steve
