Gateway duplicates usage example

[jazzl0ver]

I'm wondering how to properly set up pfsense with Gateway duplicates option as well.

In my case pfSenseA is connected to 3 different ISPs and I'd like to create tunnels to the remote pfSenseB on top of all 3 WAN channels. pfSenseB has a single WAN. So, I create 3 Phase1 entries with the same remote endpoint IP address (pfSenseB) and tick Gateway duplicates option. All parameters are the same on both sides, besides the PSKs.
According to this Gateway duplicates option's docs:

This option also disables automatic static routes to the peer via specific WAN gateways. Traffic will follow the default route, not the selected tunnel interface, unless manual static routes redirect the traffic.

How in this case should I tell pfsense to use individual interfaces to start ipsec on?
Currently, this kind of setup ends up with randomly non-working one or two tunnels out of 3.

States look weird:

(notice WAN1 and WAN3 IPs are sources on WAN2)

All tunnels are up:

# ipsec status
Shunted Connections:
   bypasslan:  172.26.0.0/16|/0 === 172.26.0.0/16|/0 PASS
Security Associations (7 up, 0 connecting):

Some of them are not working:

# ifconfig ipsec3
ipsec3: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400
        tunnel inet WAN2IP --> x.y.169.90
        inet6 fe80::204:76ff:fe24:34f0%ipsec3 prefixlen 64 scopeid 0x11
        inet 10.6.107.6 --> 10.6.107.5 netmask 0xfffffffc
        groups: ipsec
        reqid: 5003
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

# ping -c 1 10.6.107.5
PING 10.6.107.5 (10.6.107.5): 56 data bytes

--- 10.6.107.5 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss

Some are working:

# ifconfig ipsec2
ipsec2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400
        tunnel inet WAN3IP --> x.y.169.90
        inet6 fe80::204:76ff:fe24:34f0%ipsec2 prefixlen 64 scopeid 0xe
        inet 10.6.107.2 --> 10.6.107.1 netmask 0xfffffffc
        groups: ipsec
        reqid: 5002
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
# ping -c 1 10.6.107.1
PING 10.6.107.1 (10.6.107.1): 56 data bytes
64 bytes from 10.6.107.1: icmp_seq=0 ttl=64 time=34.368 ms

--- 10.6.107.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 34.368/34.368/34.368/0.000 ms

Both pfsenses are 2.6.0 and phase 2s are in VTI mode.

Advanced IPSec settings (if needed):

Any ideas and help are highly appreciated!

If I disable the working tunnel and clear the states, the other two tunnels start working (pings are working). If I then enable the 1st tunnel, I can ping all ipsec IPs.

[jazzl0ver]

This post is deleted!

[mcado_dealtis]

@jazzl0ver its a same problem for me, for the moment, i configure all phase 2 on one phase 1. I don't find a solution for the moment :(

[jazzl0ver]

@jimp please, take a look at the problem described at https://forum.netgate.com/post/1099326
any ideas?

[jimp]

You'd have to rely on using a gateway group for the default route and only one tunnel would ever work at a time.

I have never been a fan of that option because it's inherently going to be broken in some way no matter what you do, you can't route to the same destination out multiple paths. Some people think it's easier for them to manage in some way, which is a matter of personal preference I suppose, but it never surprises me when it fails to work how someone expects.

You're better off with one single tunnel using a gateway group for the interface so it can switch as you see fit. Yes, failover can be slow that way, but it does work.

That, or if the remote end can have multiple IP addresses, setup tunnels to different destination addresses for each WAN.

[mcado_dealtis]

@jimp Hello thank you for your answer, but what is the usage of the option: duplicate gateway.
That option is made then, right?
Currently, it is not possible for pfsense+ to have 3 tunnels with the same destination (same IP)?

[jimp]

It might work if and only if the remote system initiates the tunnels -- never the firewall with multiple WANs. Also assuming it's VTI, you could never have more than one tunnel with overlapping tunnel mode P2s work. Set all tunnels to responder only and configure the remote end to initiate. Only setup keep alives on the remote, and none locally. Even then it may only work if your WANs are properly setup for reply-to to function.

AFAIK that options was more about having the tunnels configured and in place so they could work one at a time, not all at once, people just didn't want to have to keep editing and changing the tunnels when they could just enable/disable them easily.

[mcado_dealtis]

@jimp ok, i understand now.
In my case, my pfsense is set as reponder only, with 3 tunnel, 2 are ok, the third loop (status down > status up, etc)
the remote firewall is a fortinet. My client imposes on me this configuration, i hunderstood there are no solution ?

[jazzl0ver]

thank you very much for the prompt reply, @jimp !

you can't route to the same destination out multiple paths

can you please elaborate on this? isn't there a way to accomplish this with the policy routing?

It might work if and only if the remote system initiates the tunnels -- never the firewall with multiple WANs. Also assuming it's VTI, you could never have more than one tunnel with overlapping tunnel mode P2s work. Set all tunnels to responder only and configure the remote end to initiate. Only setup keep alives on the remote, and none locally. Even then it may only work if your WANs are properly setup for reply-to to function.

it would be great to put this in the docs

AFAIK that options was more about having the tunnels configured and in place so they could work one at a time, not all at once, people just didn't want to have to keep editing and changing the tunnels when they could just enable/disable them easily.

it would be much easier to reach that goal by simply avoiding remote endpoints checks for disabled tunnels, wouldn't it?
just found the original request: https://redmine.pfsense.org/issues/10214. the purpose was exactly to have multiple tunnels to the same endpoints

[jimp]

@jazzl0ver said in Gateway duplicates usage example:

thank you very much for the prompt reply, @jimp !

you can't route to the same destination out multiple paths

can you please elaborate on this? isn't there a way to accomplish this with the policy routing?

It's as simple as what I said. You can't have multiple routes to the same remote destination out different paths like that.

Policy routing is different, but also policy routing from the firewall itself is not as simple as it is for traffic passing through. Source address selection can give it some grief trying to find the shortest path out before PF has a chance to do anything with policy routing.

AFAIK that options was more about having the tunnels configured and in place so they could work one at a time, not all at once, people just didn't want to have to keep editing and changing the tunnels when they could just enable/disable them easily.

it would be much easier to reach that goal by simply avoiding remote endpoints checks for disabled tunnels, wouldn't it?

It's not that simple.

just found the original request: https://redmine.pfsense.org/issues/10214. the purpose was exactly to have multiple tunnels to the same endpoints

That original issue is so vague you can't make an assumption from the description about whether they wanted them all up/active at the same time.

[jazzl0ver]

@jimp said in Gateway duplicates usage example:

It's as simple as what I said. You can't have multiple routes to the same remote destination out different paths like that.

Policy routing is different, but also policy routing from the firewall itself is not as simple as it is for traffic passing through. Source address selection can give it some grief trying to find the shortest path out before PF has a chance to do anything with policy routing.

Will multiple routing tables (setfib) help to workaround this?

[jimp]

@jazzl0ver said in Gateway duplicates usage example:

Will multiple routing tables (setfib) help to workaround this?

No because at least on FreeBSD, last I saw, setfib works on a per-process level and it wouldn't let individual tunnels in the same process choose a different FIB.

Not unless there is some feature in strongSwan itself that has support for that, but that would require quite a lot of things we don't have now to support (e.g. the ability to manage multiple FIBs)

[jazzl0ver]

@jimp yeah, i understand setfib works on per-process basis. probably, the ipsec tunnels with gateway duplicates option enabled could be started as different processes?

i really don't know how rare or frequent my case is, but from the pfsense features perspective, it would be very useful to cover the described situation and multiple routing tables support sounds like a great feature not only for ipsec tunnels.
do you mind if I create a feature request in redmine or github?

[jimp]

The way IPsec works, they cannot be separate processes. It's not like OpenVPN.

[jazzl0ver]

@jimp thanks for your answers and patience!

[jazzl0ver]

@jazzl0ver said in Gateway duplicates usage example:

It might work if and only if the remote system initiates the tunnels -- never the firewall with multiple WANs. Also assuming it's VTI, you could never have more than one tunnel with overlapping tunnel mode P2s work. Set all tunnels to responder only and configure the remote end to initiate. Only setup keep alives on the remote, and none locally. Even then it may only work if your WANs are properly setup for reply-to to function.

Hi @jimp . I've refactored my setup and the pfsense holding multiple connections to the different remotes is now set as responder only along with no local keep alives.

After recent reboot, all tunnels came up but one and I still can't get it to run. After I click on Connect P1 and P2 on the remote, tcpdump shows the outbound isakmp packets on the remote and inbound isakmp packets on the local system, but nothing happens.

on the remote:
# pfctl -sa | grep aa.bb.254.42
...
all udp xx.yy.169.90:500 -> aa.bb.254.42:500       SINGLE:NO_TRAFFIC
...

on the local:
# pfctl -sa | grep xx.yy.169.90 | grep aa.bb.254.42
#

The only guess I have is that reply-to function was not set correctly. Can you please give me some more details on the proper reply-to configuration?

[jazzl0ver]

JFYI. I've ended up with adding two extra pfsenses (for HA) that deals with ISP channels only