2 HA pfsense boxes with 1 public IP working but…

[francesco1]

For some reason after a few hours it seems to stop working. This is what happens: everything seems fine until I unplug the master WAN port and fail-over works but when I plug the master WAN port back in, the internet stops working. It appears the master role goes back to the master pfsense but nothing goes through. If I unplug the master WAN it fails over to the slave and internet works.

If I go to Interfaces -> WAN and click on SAVE (on the master), internet comes back online but I lose connectivity to the slave pfsense web interface. The only way I could find to solve that problem, is to reboot the slave and all is good after that (for a few hours at least). Happened twice so far.

This is the setup:

WAN                      LAN                      PFSYNC
PF1  10.10.10.1/24  192.168.111.1/24  172.16.222.1/30
PF2  10.11.11.1/24  192.168.111.2/24  172.16.222.2/30
CARP 1 Public IP        192.168.111.3/24
GW    Public IP

It works… for a few hours. Then I unplug the master WAN and, when I plug it back in, internet just stops working as explained above.

I'm using v2.3.2

What would cause this?

[francesco1]

BTW, I am using v2.3.2

[francesco1]

I wiped out 2.3.2 and installed 2.2.6 from scratch and same thing.

What I noticed is that  the problem only happens when the WAN goes down. If I unplug LAN, WAN and SYNC, I have to plug in WAN first then SYNC and LAN. Also, when the VM cannot get on the internet, neither the laptop connected to the LAN can.

????

[Derelict]

Define "stops working." Do some Connectivity Troubleshooting and determine what is actually failing and report back.

[francesco1]

Stops working as in no longer being able to get on the internet. Says ping timeout or host unreachable.

[Derelict]

Need more details such as what can't ping what from where. You are the one at the site who can properly troubleshoot your network. Nobody else is.

[dotdash]

You should have your WAN interfaces on the same bogus network, your setup shows them in separate /24s. When I did it, I  used a private /30 as the public IP was a /30. That's probably not needed.

[francesco1]

@Derelict: I was pinging various valid IP addresses on the internet.

@dotdash: I used a /24 subnet because that is what the ISP is giving here in the test lab. I final setup supposed to be a /30.

Today, I changed the setup. I took the whole thing off the internet and setup "my own internet" with public IP addresses and set up pfSense to route public IP addresses from the WAN to a DMZ. I got it to work but displays the same behavior as the previous setup.

Basically, if I unplug the master firewall's WAN cable, routing switches to the slave firewall properly within a second. When I see the ping timeout and then resumes receiving replies I plug the WAN cable back in to the master firewall. Sometimes it switches back to the master without a problem but sometimes it does not.

When it does not come back successfully, I unplug the master firewall's WAN cable again and the slave takes over successfully. It seems like I have to unplug all of the cables and plug them in an order where the WAN is plugged in first and then the rest.

[Derelict]

For starters, stop unplugging cables. Yes, you should test eventually but start with basic functionality.

All of your CARP VIPs should be MASTER on primary and BACKUP on secondary. If not, fix that.

Use the Temporarily Disable CARP button on Status > CARP on the primary. All of your CARP VIPs should go MASTER on the secondary (Again, Status > CARP on the secondary). If they do not, fix that.

What documentation did you follow in setting up your HA cluster?

[francesco1]

I used this document:
https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_%28CARP%29

The IP, CARP and HA part works; just not 100% of the time. All CARP related settings, rules etc., copy from master to slave.

I’m unplugging cables to test fault tolerance. What if the WAN connection goes down for 3 seconds and comes back up? If the master doesn’t come back online properly, then everything going through the firewalls will go down. Is unplugging the WAN not a valid test? This behavior that I am seeing with pfSense HA something acceptable?

You can see what I’m talking about in this video https://youtu.be/pepq4VLOUHE

This is the current setup:

“ISP” SIDE CONFIGURATION

66.66.77.10/24 Windows Server

“ISP“ Router
66.66.77.1/24 Public DMZ Interface
66.66.66.1/30 WAN Interface

MY SIDE CONFIGURATION

PFS1
10.10.10.1/30 WAN Interface Dummy IP
66.66.66.2/30 WAN CARP IP
77.77.77.2/24 Public DMZ Interface
77.77.77.1/24 Public DMZ Interface CARP IP

PFS2
10.11.11.1/30 WAN Interface Dummy IP
66.66.66.2/30 WAN CARP IP
77.77.77.3/24 Public DMZ Interface
77.77.77.1/24 Public DMZ Interface CARP IP

77.77.77.40/24 Windows Server

[Derelict]

What is a "Dummy IP?"

So you are trying to do HA without the required 3 public IP addresses (minimum /29) on WAN?

You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

[dotdash]

@Francesco:

MY SIDE CONFIGURATION

PFS1
10.10.10.1/30 WAN Interface Dummy IP

PFS2
10.11.11.1/30 WAN Interface Dummy IP

I pointed this out before, but you need to have them in the same subnet. Why not try 10.10.10.1/30 and 10.10.10.2/30 ??

[francesco1]

@Derelict:

What is a "Dummy IP?"

So you are trying to do HA without the required 3 public IP addresses (minimum /29) on WAN?

You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

Yes, I am trying to do HA without 3 public IP addresses on WAN. What other info do you need? Dummy IP = any IP placed in the IPv4 Address field of the WAN Interface that is not going to conflict with any other interface, host etc. The "Dummy IP" is required because PFS requires you to enter an IP address in that field. And, as I said before, I do NOT have the option of using more than 1 public IP address and overall seems to work.

@dotdash:

@Francesco:

MY SIDE CONFIGURATION

PFS1
10.10.10.1/30 WAN Interface Dummy IP

PFS2
10.11.11.1/30 WAN Interface Dummy IP

I pointed this out before, but you need to have them in the same subnet. Why not try 10.10.10.1/30 and 10.10.10.2/30 ??

Good question. I tried it before and it didn't make a difference. I will try it again.

[francesco1]

@dotdash:

@Francesco:

MY SIDE CONFIGURATION

PFS1
10.10.10.1/30 WAN Interface Dummy IP

PFS2
10.11.11.1/30 WAN Interface Dummy IP

I pointed this out before, but you need to have them in the same subnet. Why not try 10.10.10.1/30 and 10.10.10.2/30 ??

Just tried it again and same problem.

[Derelict]

Again - pretty sparse with the details.

[francesco1]

@Derelict:

Again - pretty sparse with the details.

Again, what more details do you need??? I asked what details you need before. I'll provide the details needed.

[Derelict]

You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

Help us help you.

[francesco1]

@Derelict:

You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

Help us help you.

I have no idea what details you are looking for. All I know right now is this:

1. I start ping command on server on both ends
2. Unplug Master Firewall WAN connection
3. I see Slave Firewall takes over when a packet is dropped and ping resumes
4. Right after that I plug the Master Firewall WAN connection
5. Master Firewall takes over but all pings fail

[Derelict]

Details like what are you pinging from where when you're testing. Specifics, like interfaces and IP addresses.

[francesco1]

I am pinging 77.77.77.40 from 66.66.77.10. I am also pinging 66.66.77.10 from 77.77.77.40.

Diagram of setup is attached. FYI, this is a closed lab setup.