Different Interfaces/Gateways Using Same IP Address

stephenw10

You may be hitting a route-to problem. Outbound traffic sourced from 10.2.0.2 will always be forced via the 10.2.0.2 gateway if route-to is applied. However this situation is complex. Check the rules file /tmp/rules.debug to see if and where that is applied.

You might also check the two states that are created when you run that ping to see what rules created them. Especially the outbound rule. You will likely have to run pfctl -vvss to see that. That can be a lot of output!

Steve

dma_pf

@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:

You may be hitting a route-to problem. Outbound traffic sourced from 10.2.0.2 will always be forced via the 10.2.0.2 gateway if route-to is applied. However this situation is complex. Check the rules file /tmp/rules.debug to see if and where that is applied.

I checked the rules.debug file and do see the "route-to" applied to the 10.2.0.2 gateway:

GWProton_NJ_WGV4 = " route-to ( tun_wg4 10.2.0.2 ) "

I can't play with it right now, but do you think if I was to change that gateway/interface to another unique IP, say 10.2.0.5, it would work? At that point there would not be any gateway/interface with a "route-to" rule for 10.2.0.2.

stephenw10

I would certainly try that first, yes.

dma_pf

@stephenw10 Hey Steve, I switched the 10.2.0.2 interface to 10.2.0.5 and it worked! I now have 3 different tunnels that are all routing traffic at 780-820 Mbits/sec up and down. I've bound them all together as a Gateway Group and policy routing is working very well.

Thanks for your help. I would never have figured that out on my own!

The only outstanding issue I have is that I if I use the 3 interfaces as the outgoing interfaces for DNS Resolver (not forwarding) it will not resolve any domains. I have added those interfaces in the DNS Resolver Access List. I have it set up that way with the interfaces from my prior VPN provider (IVPN) and I have no issues using IVPN to resolve via the root servers. But I can't seem to make it happen via Proton.

stephenw10

Aha, that's a great result!

I would check the states and also run a pcap on the WAN for DNS traffic.
You may well be hitting this: https://redmine.pfsense.org/issues/13420
Or at least something related to that. It's fixed now in 23.01/2.7.

Steve

lex33

Hi There,

I'm trying to do the same thing with Nord.

Can you possibly post your final configuration? I've tried following your example but seems secondary connections wont work for me.

If I can get this working, I will mostly likely post a guide to reddit, as a few people are trying to do the same thing.

Thanks!

Bob.Dig

@dma_pf I also wouldn't mine a short summary, what you have to do differently to a normal setup, to get this working.

stephenw10

You outbound NAT the traffic to the IP the remote side is expecting.

The key here is that you have to outbound NAT all the tunnels to that. None of them can be using that IP natively because doing so will cause route-to tagging to pass all traffic via that.

Steve

Bob.Dig

@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:

You outbound NAT the traffic to the IP the remote side is expecting.

Interesting, so only outbound NAT in that way has to be applied, nice.
Will have to try that for myself, maybe I can abandon my fleet of OpenWRT-VMs.

dma_pf

@lex33 Everything is configured as shown in my pictures above with the exception as follows:

The key thing that worked for me is that the 3 interfaces/gateways have to have unique IP addresses and they can't be the IP address that the VPN provider wants you to use.

So in my case, ProtonVPN wants all connections to all their servers to use 10.2.0.2/32. So I set my 3 interfaces/gateways to use the IPs of 10.2.0.3/32, 10.2.0.4/32 & 10.2.0.5/32. Then set the NAT for each Interface as I showed in my picture above.

In my case, using the 10.2.0.2 IP for any of the interfaces messed up the NAT due to the "reply-to" rule that's automatically applied to that interface. The reply-to rule preempts the custom NAT rules and would return packets back to the 10.2.0.2 interface. Big kudos to @stephenw10 for figuring that out! (Way over my pay grade)

I'm still having the issue with the DNS, but I haven't had the time to mess with it. For right now I'm still routing resolver out through my account with IVPN (I paid for a yearly subscription and it ends in March).

I have discovered another issue since my last post. For some reason when I stream music, via either a single connection or through a gateway group, the songs play perfectly for about half the song. But then somewhere in the last half of the song it will just jump to the next song in the playlist. I've definitely been able to isolate this to the VPN connections to ProtonVPN. I haven't had the time to see if this happens when streaming video. I have no issues whatsoever when streaming via IVPN.

stephenw10

Maybe the streaming service detecting you're coming from a VPN IP?

dma_pf

@stephenw10 I don't think that's it. I've tried 3 different ProtonVPN servers in distinctly different geographical areas. In addition when I streamed through IVPN I used it with their servers located in the exact same data centers as ProtonVPN's servers.

I'm also having the same exact issues with 2 different streaming services, Tidal and Qobuz.

As best as I can tell, it seems like packets are just not getting to the devices. With Tidal it will just jump to the next song, like as if the current song had ended. With Qobuz I can see the song's remaining time indicator kind of wiggle back and forth a little. It will do that for a bit and then it's like the data catches up with it and it starts playing again from where it left off.

I reached out to ProtonVPN about it and they suggested I install their app on my device...lol.

Bob.Dig

@dma_pf Maybe IPv6 is used somehow on your side?
I want to test it but have to change so much other things right now, don't have time, yet. And I will not delete my fleet of Vms for sure.

Bob.Dig

@dma_pf said in Different Interfaces/Gateways Using Same IP Address:

I'm still having the issue with the DNS, but I haven't had the time to mess with it. For right now I'm still routing resolver out through my account with IVPN (I paid for a yearly subscription and it ends in March).

Maybe because you only have one subnet as source for the outbound-NAT?

And one other thought:

The best practice is to use strict rules when utilizing static port to avoid any potential conflict if two local hosts use the same source port to talk to the same remote server and port using the same external IP address.

WireGuard on pfSense is using a static Source Port by default. I don't know if this is a must for WG in general. Maybe this has to be considered. So I would try to use static port outbound NAT.

stephenw10

Having DNS queries going out over a different VPN could definitely cause issues like this. Services that 'detect' VPN use often use DNS queries to do it.

Steve

Bob.Dig

I can confirm that it is not working. After some minutes tunnels will go down and will come back but there are disconnects. Also it doesn't matter if I NAT to that one IP or to the Interface IP.
Using my separate VMs for that, no problems.

dma_pf

@bob-dig said in Different Interfaces/Gateways Using Same IP Address:

I can confirm that it is not working. After some minutes tunnels will go down and will come back but there are disconnects. Also it doesn't matter if I NAT to that one IP or to the Interface IP.

Do you have a keep alive value in the wireguard peer setting? I have mine set to 25 on all my peers. I haven't experienced any disconnects in my tunnels. Who is your provider and what does their config look like?

I've been trying to troubleshoot my streaming and DNS issues.

I resolved my music streaming issue. I'm using Logitech Media Server (LMS) on a Windows server to feed music to different players which is where I was having the streaming issues. I was able to modify some settings in LMS related to how it connects to the network and that seems to have resolved the issue.

In doing some of the testing I ran speedtests from the command line and saw that there was a big variance in latency with ProtonVPN as compared to IVPN. I think that might have been the issue and adjusting the network connection properties in the media player overcame that.

The DNS issue is interesting. I've absolutely narrowed it down to an issue in the interface IP which might be related to the NAT issue. As I mentioned before, ProtonVPN want's all connections to each of their different servers to come in on 10.2.0.2. I isolated my testing to getting DNS to work on just 1 interface, 10.2.0.3. What I found is that I can't resolve or forward (208.67.222.222) at all over that interface regardless of what I tried to do. ProtonVPN confirmed with me that they do not block port 53 so normal resolver, or resolver in forwarding mode, should work.

So I reverted the interface to the 10.2.0.2 IP that ProtonVPN wants. Bang! Resolver and forwarding works perfectly. Something strange is happening and I don't know what it is.

I ran a pcap on the ProtonVPN tunnel for a failed DNS request when the interface was set to 10.2.0.3:

You can see in the blue circle that there is traffic going in/out of the 10.2.0.3 interface. The translating NAT rule is sending it out of the source as 10.2.0.2 as seen by the destination server is returning the packets to 10.2.0.2.

But for some reason the query in the red circle is messed up when it goes out the 10.2.0.3 interface. Compare it to the picture below which is a pcap from a successful DNS request sent out the IVPN tunnel. The only change between the two is selecting the IVPN interface in Resolver Settings/Outgoing Network Interfaces, clearing all states and restarting resolver.

I'm guessing this might be a NAT issue but I'm just not knowledgeable enough about NAT to understand what's going on.

I'm not sure it this helps you but I wanted to pass along the info in case it could.

Bob.Dig

@dma_pf said in Different Interfaces/Gateways Using Same IP Address:

Do you have a keep alive value in the wireguard peer setting? I have mine set to 25 on all my peers. I haven't experienced any disconnects in my tunnels. Who is your provider and what does their config look like?

No, but I had gateway monitoring running on them. My providers config is the same for every tunnel: 10.14.0.2/16.

Now back on my OpenWrt-VMs it is perfect, no high pings or losses.

So I still think it is not possible to do that with WireGuards Cryptokey Routing on pfSense. It is strange though, that it is working for some limited time and I only have experience with one privacy-VPN. And there are other providers like Mullvad, where every tunnel gets a different IP.

dma_pf

@bob-dig said in Different Interfaces/Gateways Using Same IP Address:

And there are other providers like Mullvad, where every tunnel gets a different IP.

IVPN is like that as well. I've used them for years. It's been pretty much rock solid, set it and forget it. Good speeds, and I've found their tech support to be very good. I really have no reason to move away from them other than my Proton email account includes 10 VPN connections. So possibly saving $100/yr.