Different Interfaces/Gateways Using Same IP Address

dma_pf · Dec 19, 2022, 1:33 PM

I'm not sure if this can be accomplished with pfsense, but I'm hoping it can be.

I'm trying to set up multiple connections to a VPN provider. The VPN provider provides configuration files to access their servers via Wireguard. Each server has a unique public endpoint IP address.

Unfortunately the Wireguard configuration for each connection to the VPN provider's endpoints specify the exact same interface IP address (10.2.0.2/32) for each connection. That is preventing me from setting up multiple connections to the different endpoints as I can't setup the Interfaces/Gateways in pfsense with the same IP address (10.2.0.2/32).

Is there some way in pfsense that I can create unique interface/gateway IP addresses but have the outgoing packets translated to show that the source is 10.2.0.2/32?

stephenw10 · Dec 19, 2022, 3:18 PM

No, probably not. But...

Is the gateway IP also the same? That would break the routing in pfSense if so.

You can change the outbound NAT on the WG interface to the same IP address. Normally I would expect that to break the routing at the server end but if they are separate servers they have presumably allowed for that. Easy enough to try that. If the gateway is different.

Steve

viragomann · Dec 19, 2022, 4:02 PM

@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:

You can change the outbound NAT on the WG interface to the same IP address.

I would not expect that this will work. The server will await that the IP is assigned to the clients interface, but pfSense doesn't allow to assign the same IP to multiple interfaces.

Bob.Dig · Dec 19, 2022, 4:15 PM

Often in these setups the interface address is the gateway address, so there is no hope.

dma_pf · Dec 20, 2022, 1:33 PM

@stephenw10 @viragomann @Bob-Dig Thanks for chiming in here. As it appears that there might be some conflicting opinions here I think it might me best if I can elaborate on what I tried so far. Maybe if I'm more explicit/detailed it can help clear things up.

So in pfsense I was going to set up 3 Wirerguard tunnels to 3 different VPN provider endpoints. Each of the endpoints are located in different states in the US, they are all housed at different datacenters, and all of the datacenters are managed by different providers. Each of the servers has a unique public IP address.

The Wireguard Tunnels are easy to setup and are essentially identical other than the Listen Ports being unique to each tunnel as well as the keys.

The Wireguard Peers are also essentially the same with the only differences being the Endpoint IP address which are set to the unique IP address of the VPN servers at each of the data centers. Obviously the Public Key is unique in each peer. The allowed IPs is set to 0.0.0.0/0.

I set up the initial interface for the first Wireguard tunnel using a static IP address of 10.2.0.2/32 which is what the VPN provider indicated I should use based in their wireguard configuration. I then created an Upstream Gateway using the same IP address (10.2.0.2) as the interface. I then bound the Gateway to the Interface.

Finally, I created a outbound NAT rule for the new interface, using my LAN address as the Source network and "Any" as the destination.

The above steps all work correctly. I can policy route out the Wireguard tunnel with no issues via the new interface and gateway.

The problems all start when I try to set up the second or third tunnels. Specifically, the IP address that the VPN provider specifies to use for the Interface and Gateway for tunnel 2 & 3 is also 10.2.0.2/32. In trying to set up the additional Interfaces I get an error indicating that 10.2.0.2/32 is being used by, or overlaps with, the same IP range assigned to the first tunnel.

@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:

Is the gateway IP also the same? That would break the routing in pfSense if so.

I'm not sure which gateway IP you're referring to here. If it's the WAN gateway to my ISP, then yes all 3 tunnels are going out that gateway. It's the only gateway I have to the internet.

If it's the IP of the gateway's for the 3 wireguard tunnels, I am able to set gateways up with 3 different IP addresses. But the problem I'm having is that I still can't set up the interfaces for tunnels 2 & 3 because they require the 10.2.0.2 IP address and therefore I can't bind the gateways (2 or 3) to the appropriate interfaces.

What I was hoping could be done would be that I could set up the interfaces/gateways for tunnels 2 and 3 with different IP ranges, say 10.2.0.3/32 and 10.2.0.4/32. Then have the outgoing traffic translated (somehow???) to 10.2.0.2 as it goes out the gateway. If this is foolish thinking please excuse my ignorance. I'm definitely a bit over my head here. But I'm certainly grateful for either knowing it can't be done or, learning any new thing that you guys can point me to in making this work.

stephenw10 · Dec 20, 2022, 1:33 PM

@dma_pf said in Different Interfaces/Gateways Using Same IP Address:

I'm not sure which gateway IP you're referring to here.

It's the WG tunnel gateways. You can't policy route to 10.2.0.2 if that is 3 gateways. They need to be different.

Wireguard doesn't actually need/use a tunnel subnet though it uses cryptokey routing. So if you can define different local IPs and gateways that satisfy the FreeBSD routing requirements you might be able to just change the outbound NAT rule to to use 10.2.0.2 on all of them. Try it.

Steve

Bob.Dig · Dec 20, 2022, 2:56 PM

@dma_pf said in Different Interfaces/Gateways Using Same IP Address:

I am able to set gateways up with 3 different IP addresses.

That sounds promising. Please keep us updated because it is a hassle with many of the privacy VPNs using WireGuard. There are exceptions like mullvad though.

dma_pf · Dec 20, 2022, 6:31 PM

@bob-dig said in Different Interfaces/Gateways Using Same IP Address:

That sounds promising. Please keep us updated because it is a hassle with many of the privacy VPNs using WireGuard. There are exceptions like mullvad though.

I've been using IVPN for years (since the wireguard package came out) with no issues whatsoever. But they provide a unique IP address for each interface.

I just switched some email accounts to a paid Protonmail plan and they include 10 connections via their VPN as part of the email package. So I'm trying to see if I can get their VPN working with the thought of possibly dropping IVPN and saving $100/year.

So I do have an update.

I configured 3 different Interfaces and Gateways. I used these IP addresses for them:

10.2.0.2/32 (the one provided by proton)
10.2.0.3/32
10.2.0.4/32

I can see in the Wireguard status page that all of the tunnels to the endpoints are shown as up. I did a little bit of digging around on cryptokey routing and from my reading (and Steve's earlier comments) this would be expected as the keys for the tunnel/peer would control the establishment of the tunnel. The IP addresses have noting to do with the actual tunnel being connected. But once the tunnel is connected then the IP address is checked and if it is on their allowed IP list then traffic can enter their server.

With that in mind, 10.2.0.2/32 (the one provided by proton) has no issue routing traffic. The other 2 do not with a typical NAT Rule.

@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:

Wireguard doesn't actually need/use a tunnel subnet though it uses cryptokey routing. So if you can define different local IPs and gateways that satisfy the FreeBSD routing requirements you might be able to just change the outbound NAT rule to to use 10.2.0.2 on all of them. Try it.

Steve

Since I was not able to route via the typical NAT rule I tried modifying the outbound NAT rule as follows:

For the interface I used the 10.2.0.3
For Source I used my LAN network IP range.
For Desination I used Any
For NAT Address I did the following:

I first created a new Host Alias with a single IP of 10.2.0.2 (the IP given by Proton)
In the NAT Outbound rule I changed Translation/Address to the Host Alias I created above. Translation Pool Options was left at "Default" and Port or Range is blank.

I then saved/applied the rule and flushed the states. I then policy routed a laptop on the LAN to route out the 10.2.0.3 gateway and it can't get to any website. I verified from the command prompt that the laptop can resolve domain names. I tried to ping from the command prompt on the laptop and it is always successful getting the first ping and then fails on the remaining 3 tries. That happened on all the IP addresses/domains I tried to ping.

I tried rebooting pfsense just in case something was stuck with all the fooling around I've been doing on it. But no luck.

That's as far as I've gotten.

stephenw10 · Dec 20, 2022, 6:59 PM

Check the state whilst you do that. Make sure traffic is actually going to that interface and that it's being translated correctly.

There are a number of ways this could fail really....

dma_pf · Dec 20, 2022, 7:09 PM

@stephenw10 The Proton_NY gateway is the 10.2.0.3 IP address range. So it looks to me that the NAT rule is routing it out on 10.2.0.2 which is what Proton would be expecting to see.

stephenw10 · Dec 21, 2022, 6:51 PM

Well there is some traffic coming back. I'd try running a pcap on that interface and see what's coming back there. Or in fact if that traffic is actually using that interface.

dma_pf · Dec 21, 2022, 6:51 PM

@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:

Well there is some traffic coming back. I'd try running a pcap on that interface and see what's coming back there. Or in fact if that traffic is actually using that interface.

I did some further testing as you suggested. I set the laptop to policy route out the 10.2.0.3 (tunnel #2 Proton_NY) interface. I then ran a pcap on that interface for for ICMP packets by doing a ping test from the laptop. The pcap showed absolutely no packets on the 10.2.0.3 interface.

I then did the same test, but this time I ran the pcap on the 10.2.0.2 interface and sure enough the ping packets are showing up as routing out that interface.

I double checked all my setting and am pretty sure they are correct. But I'll post some pictures to make sure.

Pcap on the 10.2.0.2 interface

Policy Routing Laptop to the 10.2.0.3 Interface

10.2.0.2 Interface Config

10.2.0.2 Gateway Config

10.2.0.3 Interface Config

10.2.0.3 Gateway Config

Outbound NAT Rules for 10.2.0.2 and 10.2.0.3 Interfaces

Detail Of Outbound NAT Rule Config For 10.2.0.3 Interface

Proton_Interface_Address Alias Config

Thanks for your help. Let me know what else I can do to help figure this out!

stephenw10 · Dec 22, 2022, 1:56 AM

You may be hitting a route-to problem. Outbound traffic sourced from 10.2.0.2 will always be forced via the 10.2.0.2 gateway if route-to is applied. However this situation is complex. Check the rules file /tmp/rules.debug to see if and where that is applied.

You might also check the two states that are created when you run that ping to see what rules created them. Especially the outbound rule. You will likely have to run pfctl -vvss to see that. That can be a lot of output!

Steve

dma_pf · Dec 22, 2022, 4:08 PM

@stephenw10 said in Different Interfaces/Gateways Using Same IP Address:

You may be hitting a route-to problem. Outbound traffic sourced from 10.2.0.2 will always be forced via the 10.2.0.2 gateway if route-to is applied. However this situation is complex. Check the rules file /tmp/rules.debug to see if and where that is applied.

I checked the rules.debug file and do see the "route-to" applied to the 10.2.0.2 gateway:

GWProton_NJ_WGV4 = " route-to ( tun_wg4 10.2.0.2 ) "

I can't play with it right now, but do you think if I was to change that gateway/interface to another unique IP, say 10.2.0.5, it would work? At that point there would not be any gateway/interface with a "route-to" rule for 10.2.0.2.

stephenw10 · Dec 22, 2022, 5:52 PM

I would certainly try that first, yes.

dma_pf · Jan 5, 2023, 9:10 AM

@stephenw10 Hey Steve, I switched the 10.2.0.2 interface to 10.2.0.5 and it worked! I now have 3 different tunnels that are all routing traffic at 780-820 Mbits/sec up and down. I've bound them all together as a Gateway Group and policy routing is working very well.

Thanks for your help. I would never have figured that out on my own!

The only outstanding issue I have is that I if I use the 3 interfaces as the outgoing interfaces for DNS Resolver (not forwarding) it will not resolve any domains. I have added those interfaces in the DNS Resolver Access List. I have it set up that way with the interfaces from my prior VPN provider (IVPN) and I have no issues using IVPN to resolve via the root servers. But I can't seem to make it happen via Proton.

stephenw10 · Dec 23, 2022, 8:16 PM

Aha, that's a great result!

I would check the states and also run a pcap on the WAN for DNS traffic.
You may well be hitting this: https://redmine.pfsense.org/issues/13420
Or at least something related to that. It's fixed now in 23.01/2.7.

Steve

lex33 · Jan 5, 2023, 2:54 AM

Hi There,

I'm trying to do the same thing with Nord.

Can you possibly post your final configuration? I've tried following your example but seems secondary connections wont work for me.

If I can get this working, I will mostly likely post a guide to reddit, as a few people are trying to do the same thing.

Thanks!

Bob.Dig · Jan 5, 2023, 9:10 AM

@dma_pf I also wouldn't mine a short summary, what you have to do differently to a normal setup, to get this working.

stephenw10 · Jan 5, 2023, 2:18 PM

You outbound NAT the traffic to the IP the remote side is expecting.

The key here is that you have to outbound NAT all the tunnels to that. None of them can be using that IP natively because doing so will cause route-to tagging to pass all traffic via that.

Steve