Different Interfaces/Gateways Using Same IP Address
-
Check the state whilst you do that. Make sure traffic is actually going to that interface and that it's being translated correctly.
There are a number of ways this could fail really....
-
@stephenw10 The Proton_NY gateway is the 10.2.0.3 IP address range. So it looks to me that the NAT rule is routing it out on 10.2.0.2 which is what Proton would be expecting to see.
-
Well there is some traffic coming back. I'd try running a pcap on that interface and see what's coming back there. Or in fact if that traffic is actually using that interface.
-
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
Well there is some traffic coming back. I'd try running a pcap on that interface and see what's coming back there. Or in fact if that traffic is actually using that interface.
I did some further testing as you suggested. I set the laptop to policy route out the 10.2.0.3 (tunnel #2 Proton_NY) interface. I then ran a pcap on that interface for for ICMP packets by doing a ping test from the laptop. The pcap showed absolutely no packets on the 10.2.0.3 interface.
I then did the same test, but this time I ran the pcap on the 10.2.0.2 interface and sure enough the ping packets are showing up as routing out that interface.
I double checked all my setting and am pretty sure they are correct. But I'll post some pictures to make sure.
Pcap on the 10.2.0.2 interface
Policy Routing Laptop to the 10.2.0.3 Interface
10.2.0.2 Interface Config
10.2.0.2 Gateway Config
10.2.0.3 Interface Config
10.2.0.3 Gateway Config
Outbound NAT Rules for 10.2.0.2 and 10.2.0.3 Interfaces
Detail Of Outbound NAT Rule Config For 10.2.0.3 Interface
Proton_Interface_Address Alias Config
Thanks for your help. Let me know what else I can do to help figure this out!
-
You may be hitting a route-to problem. Outbound traffic sourced from 10.2.0.2 will always be forced via the 10.2.0.2 gateway if route-to is applied. However this situation is complex. Check the rules file /tmp/rules.debug to see if and where that is applied.
You might also check the two states that are created when you run that ping to see what rules created them. Especially the outbound rule. You will likely have to run
pfctl -vvssto see that. That can be a lot of output!Steve
-
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
You may be hitting a route-to problem. Outbound traffic sourced from 10.2.0.2 will always be forced via the 10.2.0.2 gateway if route-to is applied. However this situation is complex. Check the rules file /tmp/rules.debug to see if and where that is applied.
I checked the rules.debug file and do see the "route-to" applied to the 10.2.0.2 gateway:
GWProton_NJ_WGV4 = " route-to ( tun_wg4 10.2.0.2 ) "I can't play with it right now, but do you think if I was to change that gateway/interface to another unique IP, say 10.2.0.5, it would work? At that point there would not be any gateway/interface with a "route-to" rule for 10.2.0.2.
-
I would certainly try that first, yes.
-
@stephenw10 Hey Steve, I switched the 10.2.0.2 interface to 10.2.0.5 and it worked! I now have 3 different tunnels that are all routing traffic at 780-820 Mbits/sec up and down. I've bound them all together as a Gateway Group and policy routing is working very well.
Thanks for your help. I would never have figured that out on my own!
The only outstanding issue I have is that I if I use the 3 interfaces as the outgoing interfaces for DNS Resolver (not forwarding) it will not resolve any domains. I have added those interfaces in the DNS Resolver Access List. I have it set up that way with the interfaces from my prior VPN provider (IVPN) and I have no issues using IVPN to resolve via the root servers. But I can't seem to make it happen via Proton.
-
Aha, that's a great result!
I would check the states and also run a pcap on the WAN for DNS traffic.
You may well be hitting this: https://redmine.pfsense.org/issues/13420
Or at least something related to that. It's fixed now in 23.01/2.7.Steve
-
Hi There,
I'm trying to do the same thing with Nord.
Can you possibly post your final configuration? I've tried following your example but seems secondary connections wont work for me.
If I can get this working, I will mostly likely post a guide to reddit, as a few people are trying to do the same thing.
Thanks!
-
@dma_pf I also wouldn't mine a short summary, what you have to do differently to a normal setup, to get this working.
-
You outbound NAT the traffic to the IP the remote side is expecting.
The key here is that you have to outbound NAT all the tunnels to that. None of them can be using that IP natively because doing so will cause route-to tagging to pass all traffic via that.
Steve
-
@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:
You outbound NAT the traffic to the IP the remote side is expecting.
Interesting, so only outbound NAT in that way has to be applied, nice.
Will have to try that for myself, maybe I can abandon my fleet of OpenWRT-VMs.
-
@lex33 Everything is configured as shown in my pictures above with the exception as follows:
The key thing that worked for me is that the 3 interfaces/gateways have to have unique IP addresses and they can't be the IP address that the VPN provider wants you to use.
So in my case, ProtonVPN wants all connections to all their servers to use 10.2.0.2/32. So I set my 3 interfaces/gateways to use the IPs of 10.2.0.3/32, 10.2.0.4/32 & 10.2.0.5/32. Then set the NAT for each Interface as I showed in my picture above.
In my case, using the 10.2.0.2 IP for any of the interfaces messed up the NAT due to the "reply-to" rule that's automatically applied to that interface. The reply-to rule preempts the custom NAT rules and would return packets back to the 10.2.0.2 interface. Big kudos to @stephenw10 for figuring that out!
(Way over my pay grade)I'm still having the issue with the DNS, but I haven't had the time to mess with it. For right now I'm still routing resolver out through my account with IVPN (I paid for a yearly subscription and it ends in March).
I have discovered another issue since my last post. For some reason when I stream music, via either a single connection or through a gateway group, the songs play perfectly for about half the song. But then somewhere in the last half of the song it will just jump to the next song in the playlist. I've definitely been able to isolate this to the VPN connections to ProtonVPN. I haven't had the time to see if this happens when streaming video. I have no issues whatsoever when streaming via IVPN.
-
Maybe the streaming service detecting you're coming from a VPN IP?
-
@stephenw10 I don't think that's it. I've tried 3 different ProtonVPN servers in distinctly different geographical areas. In addition when I streamed through IVPN I used it with their servers located in the exact same data centers as ProtonVPN's servers.
I'm also having the same exact issues with 2 different streaming services, Tidal and Qobuz.
As best as I can tell, it seems like packets are just not getting to the devices. With Tidal it will just jump to the next song, like as if the current song had ended. With Qobuz I can see the song's remaining time indicator kind of wiggle back and forth a little. It will do that for a bit and then it's like the data catches up with it and it starts playing again from where it left off.
I reached out to ProtonVPN about it and they suggested I install their app on my device...lol.
-
@dma_pf Maybe IPv6 is used somehow on your side?
I want to test it but have to change so much other things right now, don't have time, yet. And I will not delete my fleet of Vms for sure. -
@dma_pf said in Different Interfaces/Gateways Using Same IP Address:
I'm still having the issue with the DNS, but I haven't had the time to mess with it. For right now I'm still routing resolver out through my account with IVPN (I paid for a yearly subscription and it ends in March).
Maybe because you only have one subnet as source for the outbound-NAT?
And one other thought:
The best practice is to use strict rules when utilizing static port to avoid any potential conflict if two local hosts use the same source port to talk to the same remote server and port using the same external IP address.
WireGuard on pfSense is using a static Source Port by default. I don't know if this is a must for WG in general. Maybe this has to be considered. So I would try to use static port outbound NAT.
-
Having DNS queries going out over a different VPN could definitely cause issues like this. Services that 'detect' VPN use often use DNS queries to do it.
Steve
-
I can confirm that it is not working. After some minutes tunnels will go down and will come back but there are disconnects. Also it doesn't matter if I NAT to that one IP or to the Interface IP.
Using my separate VMs for that, no problems.
