Different Interfaces/Gateways Using Same IP Address

[Bob.Dig]

@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:

You outbound NAT the traffic to the IP the remote side is expecting.

Interesting, so only outbound NAT in that way has to be applied, nice.
Will have to try that for myself, maybe I can abandon my fleet of OpenWRT-VMs.

[dma_pf]

@lex33 Everything is configured as shown in my pictures above with the exception as follows:

The key thing that worked for me is that the 3 interfaces/gateways have to have unique IP addresses and they can't be the IP address that the VPN provider wants you to use.

So in my case, ProtonVPN wants all connections to all their servers to use 10.2.0.2/32. So I set my 3 interfaces/gateways to use the IPs of 10.2.0.3/32, 10.2.0.4/32 & 10.2.0.5/32. Then set the NAT for each Interface as I showed in my picture above.

In my case, using the 10.2.0.2 IP for any of the interfaces messed up the NAT due to the "reply-to" rule that's automatically applied to that interface. The reply-to rule preempts the custom NAT rules and would return packets back to the 10.2.0.2 interface. Big kudos to @stephenw10 for figuring that out! (Way over my pay grade)

I'm still having the issue with the DNS, but I haven't had the time to mess with it. For right now I'm still routing resolver out through my account with IVPN (I paid for a yearly subscription and it ends in March).

I have discovered another issue since my last post. For some reason when I stream music, via either a single connection or through a gateway group, the songs play perfectly for about half the song. But then somewhere in the last half of the song it will just jump to the next song in the playlist. I've definitely been able to isolate this to the VPN connections to ProtonVPN. I haven't had the time to see if this happens when streaming video. I have no issues whatsoever when streaming via IVPN.

[stephenw10]

Maybe the streaming service detecting you're coming from a VPN IP?

[dma_pf]

@stephenw10 I don't think that's it. I've tried 3 different ProtonVPN servers in distinctly different geographical areas. In addition when I streamed through IVPN I used it with their servers located in the exact same data centers as ProtonVPN's servers.

I'm also having the same exact issues with 2 different streaming services, Tidal and Qobuz.

As best as I can tell, it seems like packets are just not getting to the devices. With Tidal it will just jump to the next song, like as if the current song had ended. With Qobuz I can see the song's remaining time indicator kind of wiggle back and forth a little. It will do that for a bit and then it's like the data catches up with it and it starts playing again from where it left off.

I reached out to ProtonVPN about it and they suggested I install their app on my device...lol.

[Bob.Dig]

@dma_pf Maybe IPv6 is used somehow on your side?
I want to test it but have to change so much other things right now, don't have time, yet. And I will not delete my fleet of Vms for sure.

[Bob.Dig]

@dma_pf said in Different Interfaces/Gateways Using Same IP Address:

I'm still having the issue with the DNS, but I haven't had the time to mess with it. For right now I'm still routing resolver out through my account with IVPN (I paid for a yearly subscription and it ends in March).

Maybe because you only have one subnet as source for the outbound-NAT?

And one other thought:

The best practice is to use strict rules when utilizing static port to avoid any potential conflict if two local hosts use the same source port to talk to the same remote server and port using the same external IP address.

WireGuard on pfSense is using a static Source Port by default. I don't know if this is a must for WG in general. Maybe this has to be considered. So I would try to use static port outbound NAT.

[stephenw10]

Having DNS queries going out over a different VPN could definitely cause issues like this. Services that 'detect' VPN use often use DNS queries to do it.

Steve

[Bob.Dig]

I can confirm that it is not working. After some minutes tunnels will go down and will come back but there are disconnects. Also it doesn't matter if I NAT to that one IP or to the Interface IP.
Using my separate VMs for that, no problems.

[dma_pf]

@bob-dig said in Different Interfaces/Gateways Using Same IP Address:

I can confirm that it is not working. After some minutes tunnels will go down and will come back but there are disconnects. Also it doesn't matter if I NAT to that one IP or to the Interface IP.

Do you have a keep alive value in the wireguard peer setting? I have mine set to 25 on all my peers. I haven't experienced any disconnects in my tunnels. Who is your provider and what does their config look like?

I've been trying to troubleshoot my streaming and DNS issues.

I resolved my music streaming issue. I'm using Logitech Media Server (LMS) on a Windows server to feed music to different players which is where I was having the streaming issues. I was able to modify some settings in LMS related to how it connects to the network and that seems to have resolved the issue.

In doing some of the testing I ran speedtests from the command line and saw that there was a big variance in latency with ProtonVPN as compared to IVPN. I think that might have been the issue and adjusting the network connection properties in the media player overcame that.

The DNS issue is interesting. I've absolutely narrowed it down to an issue in the interface IP which might be related to the NAT issue. As I mentioned before, ProtonVPN want's all connections to each of their different servers to come in on 10.2.0.2. I isolated my testing to getting DNS to work on just 1 interface, 10.2.0.3. What I found is that I can't resolve or forward (208.67.222.222) at all over that interface regardless of what I tried to do. ProtonVPN confirmed with me that they do not block port 53 so normal resolver, or resolver in forwarding mode, should work.

So I reverted the interface to the 10.2.0.2 IP that ProtonVPN wants. Bang! Resolver and forwarding works perfectly. Something strange is happening and I don't know what it is.

I ran a pcap on the ProtonVPN tunnel for a failed DNS request when the interface was set to 10.2.0.3:

You can see in the blue circle that there is traffic going in/out of the 10.2.0.3 interface. The translating NAT rule is sending it out of the source as 10.2.0.2 as seen by the destination server is returning the packets to 10.2.0.2.

But for some reason the query in the red circle is messed up when it goes out the 10.2.0.3 interface. Compare it to the picture below which is a pcap from a successful DNS request sent out the IVPN tunnel. The only change between the two is selecting the IVPN interface in Resolver Settings/Outgoing Network Interfaces, clearing all states and restarting resolver.

I'm guessing this might be a NAT issue but I'm just not knowledgeable enough about NAT to understand what's going on.

I'm not sure it this helps you but I wanted to pass along the info in case it could.

[Bob.Dig]

@dma_pf said in Different Interfaces/Gateways Using Same IP Address:

Do you have a keep alive value in the wireguard peer setting? I have mine set to 25 on all my peers. I haven't experienced any disconnects in my tunnels. Who is your provider and what does their config look like?

No, but I had gateway monitoring running on them. My providers config is the same for every tunnel: 10.14.0.2/16.

Now back on my OpenWrt-VMs it is perfect, no high pings or losses.

Spoiler

So I still think it is not possible to do that with WireGuards Cryptokey Routing on pfSense. It is strange though, that it is working for some limited time and I only have experience with one privacy-VPN. And there are other providers like Mullvad, where every tunnel gets a different IP.

[dma_pf]

@bob-dig said in Different Interfaces/Gateways Using Same IP Address:

And there are other providers like Mullvad, where every tunnel gets a different IP.

IVPN is like that as well. I've used them for years. It's been pretty much rock solid, set it and forget it. Good speeds, and I've found their tech support to be very good. I really have no reason to move away from them other than my Proton email account includes 10 VPN connections. So possibly saving $100/yr.