LAN TO LAN WITH 4 VPN TUNNEL (REDUNDANT)

hoba

The problem rather is having same source and destination subnets for all tunneldefinitions which is causing a conflict. And even if implementing some logic that is reconfiguring the tunnels on failure of an ISP, how should the other end know that one ISP at the remote site has failed to switch it's tunnel definition? This only could work if the both ends are exchanging information about their WAN states. You could make it one way redundant with a carp setup at one end and the other end waitung for it to tunnel in as "mobile client". However this only would provide redundancy at one end.

LAN
|
+–---+------+
| | (This would need ifdepd to bring the Master down on ISP1 failure)
pfSense pfSense (Master has tunnel to ISP1 at remote end)
Master Slave (Slave has tunnel to ISP2 at remote end)
| | | |
+-----------+ |
| | |
| +----------+
| |
ISP1 ISP2
| |
| |
(Internet)
| |
ISP1 ISP2
| |
pfSense (awaiting mobile clients IPSEC at ISP1 and ISP2)
|
|
LAN

(Don't blame me if this won't work, never tried such a setup yet ;D )

sbyoon

Dear Hoba,

Could you give the method in more detail how to make one way redundant with a carp setup?
And I want to know how the two pfsense should be connected for carp setup. Which one is correct in below?

LAN
|
+–---+-------------------+
| |
pfSense pfSense
Master Slave
WAN OPT1 WAN OPT1
| | | |
+---------------------+ |
| | |
| +--------------------+
| |
ISP1 ISP2
| |

LAN
|
+-----+-------------------+
| |
pfSense pfSense
Master Slave
WAN OPT1 OPT1 WAN
| | | |
+---------------------+ |
| | |
| +--------------------+
| |
ISP1 ISP2
| |

I don't think the first one can be possible because we cannot make tunnel at OPT1 with pfsense. It means we cannot make the tunnel on ISP2.
Pls adv how to configure the carp in detail if the second one is correct. Thank you.

hoba

Having IPSEC tunnels at OPT interfaces needs fixing: http://cvstrac.pfsense.com/tktview?tn=1028,6 but after that it should work.

In case you don't want to use the both ISPs at the same time and have the second one only for failover you can set it up even easier:

sbyoon

Thank you, Hoba,

Noted now I should use two pfsense until IPsec will be available at OPT1. And I don't need two ISP at once. I need to use just one ISP on failure of another one. So your suggestion will be helpful. I think I know how to share Carp IP as gateway at LAN but I don't know how ifdepd bring LAN down on ISP failure. I don't even know what ifdepd is. Could you give me more explanation about it? Thank you.

hoba

It's a package. You can add dependencies between interfaces with it like "if interface wan is down then down interface lan". This is needed to make the slave machine become master while ISP1 is down at the original master.

_ifdepd implements dependencies between network interfaces in a
reliable way. If any of the source interfaces fails, ifdepd sets all
destination interfaces to state down. If all source interfaces are active,
ifdepd sets all destination interfaces to state up.

For example, it can be used with carp(4) to provide failover functionality
on gateways/firewalls._

martinc_77

Hi hoba is me again, tks for the info of redundant tunnel, very cool.
I' am another problem , load balancing run ok but if disconnect wan interface or Wan ISP is down, then not ping OPT interface (internet ping), not internet access (from my lan) and not tunnel is establish for OPT.
But if i copy the OPT gateway in the Wan gateway of the WAN interface (when wan isp is down) all run ok.

What's happend???

route default is the problem or is a bug?

Sorry but my english is bad.

Tks for all

hoba

There is still an open bug for IPSECs at OPTs: http://cvstrac.pfsense.com/tktview?tn=1028,6 which needs to be fixed.

sbyoon

Dear Hoba,

I tested ifdepd yesterday. And it works well with below diagram. I can use internet automatically with another pfsense when WAN at master pfsense is down.

But it does not work with IPSec. I think "Failover IPSEC" on IPSec configuration page would help solving this problem. I think it would be used for replacing IPSEC tunneling from pfsense1-pfsense3 to pfsense2-pfsense3 automatically. But I don't have any information how to configure it. Could you advise me which field on "Failover IPSEC" tab means?

"Failover IPSEC" has five fields to be filled as below.

1. "Enable" - I think it should be checked.

2. "Interface" - The hint says "Select the carp interface to use." And I can select one of r r r p l p c c. Which one should I select?

3. "Failover IP" - The hint says "Enter the IP address you would like to use for failover. HINT: You normally want to use a public CARP ip". Should I fill it with the Carp1 IP address? Carp1 IP address is the virtual IP address on LAN.

4. "Peer IP" - The hint says "Enter the peers ip address. HINT: You normally want to enter the peers REAL LAN IP here." I really don't know what IP address should be used for this field. pfSense2's LAN IP address or pfSense3' LAN address????

5. "Shared key" - The hint says "The shared AES key used to encrypt messages between sasyncd hosts. This configuration setting is required and must be either 16, 24 or 32 bytes long (corresponding to AES using a 128, 192 or 256 bit key)." I think it is a new shared key and it does not involve with the shared key that was used to make IPSEC tunneling.

And one more question. I think "Failover IPSEC" should be configured both of pfsense1 and pfsense2. Is it correct?

Thank you.

hoba

Failover IPSEC is for use with CARP on WAN to use the CARP IP instead of the real WAN IP. This doesn't apply to your setup and won't help you.

sbyoon

Noted. I should wait until IPSec on OPT1 will be fixed. Then, do you think IPSec failover will be possible with outgoing loadbalancing if OPT1 IPSec is available? Of course in this case we can use only one pfsense for failover.

Thank you.

hoba

It should just work the way it's drawn without failover IPSEC settings. Just noticed you only have one WAN (ISP3) at the opposite end and not like before 2 isps.

sbyoon

I think it does not work because of the conflict of same local subnet from pfsenes1 and pfsense2 at the remote end network. In order to succeed in IPSec failover, the first tunnelling of pfsense1-pfsense3 should be remove clearly before the second tunnelling of pfsense2-pfsense3 will be established. In case of my above diagram, the first and second tunnelling are established at once and one of those two tunnels does not work even if one of ISP disconnets and I can access to internet through another pfsense.

Is there any way to remove the first tunnel automatically when another one is trying to establish tunnel? And only one IPSec tunnel should be allowed when both of ISP1 and ISP2 are connected.

Thank you.

hoba

pfSense3 is waiting for mobile clients. pfSense1's WAN is failing and pfSense2 is establishing the IPSEC as there no w is traffic for the remote subnet due to becoming the gateway. For pfSense3 this should just look like the IP of pfSense 1 has changed. I have setups like this where one pfSense sits at a dynamic IP and it works fine. Maybe http://www.pfsense.org/mirror.php?section=tutorials/mobile_ipsec/ helps how to configure it.

sbyoon

Of course, I tested with pfsense3 wainting for mobile client. The problem is that pfsense3 cannot figure out disconneting of pfsense1 and it remember the tunnel with pfsense1 even after pfsense1 disappear. So the LAN subnet of pfsense1 and pfsense2 conflicts on pfsense3 because pfsense1 and pfsense2 has same LAN subnet and then the tunnel of pfsense2-pfsense3 does not work. Is there any solution of this problem?

hoba

Try "prefer old IPSEC SA" setting at system>advanced at all ends and see if that makes a difference. If that doesn't help try using smaller lifetimes (like 300 seconds). This way a tunnel should expire after 5 minutes and the backupmachine then will hopefully be able to connect though this means some downtime until the tunnel will be estblished again on failover.

sbyoon

Noted, I will try and let you know of the result. Thank you, Hoba.

martinc_77

I test it and run fine, work's tunnel in OPT-WAN Interface !

LAN
|
(PfSense 1)
| |
ISP1(WAN) ISP2 (OPT-WAN)
| |
| |
( Internet )
|
|
ISP3
|
|
pfSense2 (waiting for mobile clients)
|
LAN

I test this configuration and run ok, but don't automatically.

Both pfsense have static ip.
pfsense 1 have load-balancer
The tunnel is stablish with ISP1 and ISP3 using in pfsense3 mobile clients.
If ISP1 is down i enter in pfsense1 and only change in ipsec WAN Interface to OPT-WAN Interface, and in WAN Gateway write the
OPT-WAN GATEWAY
save and go to diagnostic-ping and all run fine, now vpn is stablished from OPT-WAN PFsense1 to WAN movile client of PFsense 2

the unique problem in this configuration is thats is manuall, but for now work's for me

hoba

Might be easier to have both tunnels configured and just disable the one or the other tunnel for the manual failover.

sbyoon

I've not tested ipsec redundant with carp and ifdepd because I need it at Wrap but the package of ifdepd is not available at embedded system. I'm waiting 1.0 version which will be available package installation at embedded system.

And according to Martinc_77's message the IPsec problem at OPT1 interface was already solved. I tested it with Wrap but IPsec at OPT1 interface does not work yet. Is there anyone who tested it with Wrap? I'd like to know whether only I have this problem. I used pfsense RC2 for testing.

Thank you.

hoba

pfSense 1.0 won't have packagesupport on embedded platforms. Where did you read that? I have tested IPSEC on OPT Interfaces during the hackathon, however I have not tested with hosts coming not from the OPT subnet. In that case it might be possible to add a static route for the remote IPSEC endpoint through the OPT interface gateway.