2 HA pfsense boxes with 1 public IP working but…

francesco1

For some reason after a few hours it seems to stop working. This is what happens: everything seems fine until I unplug the master WAN port and fail-over works but when I plug the master WAN port back in, the internet stops working. It appears the master role goes back to the master pfsense but nothing goes through. If I unplug the master WAN it fails over to the slave and internet works.

If I go to Interfaces -> WAN and click on SAVE (on the master), internet comes back online but I lose connectivity to the slave pfsense web interface. The only way I could find to solve that problem, is to reboot the slave and all is good after that (for a few hours at least). Happened twice so far.

This is the setup:

WAN LAN PFSYNC
PF1 10.10.10.1/24 192.168.111.1/24 172.16.222.1/30
PF2 10.11.11.1/24 192.168.111.2/24 172.16.222.2/30
CARP 1 Public IP 192.168.111.3/24
GW Public IP

It works… for a few hours. Then I unplug the master WAN and, when I plug it back in, internet just stops working as explained above.

I'm using v2.3.2

What would cause this?

francesco1

BTW, I am using v2.3.2

francesco1

I wiped out 2.3.2 and installed 2.2.6 from scratch and same thing.

What I noticed is that the problem only happens when the WAN goes down. If I unplug LAN, WAN and SYNC, I have to plug in WAN first then SYNC and LAN. Also, when the VM cannot get on the internet, neither the laptop connected to the LAN can.

????

Derelict

Define "stops working." Do some Connectivity Troubleshooting and determine what is actually failing and report back.

francesco1

Stops working as in no longer being able to get on the internet. Says ping timeout or host unreachable.

Derelict

Need more details such as what can't ping what from where. You are the one at the site who can properly troubleshoot your network. Nobody else is.

dotdash

You should have your WAN interfaces on the same bogus network, your setup shows them in separate /24s. When I did it, I used a private /30 as the public IP was a /30. That's probably not needed.

francesco1

@Derelict: I was pinging various valid IP addresses on the internet.

@dotdash: I used a /24 subnet because that is what the ISP is giving here in the test lab. I final setup supposed to be a /30.

Today, I changed the setup. I took the whole thing off the internet and setup "my own internet" with public IP addresses and set up pfSense to route public IP addresses from the WAN to a DMZ. I got it to work but displays the same behavior as the previous setup.

Basically, if I unplug the master firewall's WAN cable, routing switches to the slave firewall properly within a second. When I see the ping timeout and then resumes receiving replies I plug the WAN cable back in to the master firewall. Sometimes it switches back to the master without a problem but sometimes it does not.

When it does not come back successfully, I unplug the master firewall's WAN cable again and the slave takes over successfully. It seems like I have to unplug all of the cables and plug them in an order where the WAN is plugged in first and then the rest.

Derelict

For starters, stop unplugging cables. Yes, you should test eventually but start with basic functionality.

All of your CARP VIPs should be MASTER on primary and BACKUP on secondary. If not, fix that.

Use the Temporarily Disable CARP button on Status > CARP on the primary. All of your CARP VIPs should go MASTER on the secondary (Again, Status > CARP on the secondary). If they do not, fix that.

What documentation did you follow in setting up your HA cluster?

francesco1

I used this document:
https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_%28CARP%29

The IP, CARP and HA part works; just not 100% of the time. All CARP related settings, rules etc., copy from master to slave.

I’m unplugging cables to test fault tolerance. What if the WAN connection goes down for 3 seconds and comes back up? If the master doesn’t come back online properly, then everything going through the firewalls will go down. Is unplugging the WAN not a valid test? This behavior that I am seeing with pfSense HA something acceptable?

You can see what I’m talking about in this video https://youtu.be/pepq4VLOUHE

This is the current setup:

“ISP” SIDE CONFIGURATION

66.66.77.10/24 Windows Server

“ISP“ Router
66.66.77.1/24 Public DMZ Interface
66.66.66.1/30 WAN Interface

MY SIDE CONFIGURATION

PFS1
10.10.10.1/30 WAN Interface Dummy IP
66.66.66.2/30 WAN CARP IP
77.77.77.2/24 Public DMZ Interface
77.77.77.1/24 Public DMZ Interface CARP IP

PFS2
10.11.11.1/30 WAN Interface Dummy IP
66.66.66.2/30 WAN CARP IP
77.77.77.3/24 Public DMZ Interface
77.77.77.1/24 Public DMZ Interface CARP IP

77.77.77.40/24 Windows Server

Derelict

What is a "Dummy IP?"

So you are trying to do HA without the required 3 public IP addresses (minimum /29) on WAN?

You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

dotdash

@Francesco:

MY SIDE CONFIGURATION

PFS1
10.10.10.1/30 WAN Interface Dummy IP

PFS2
10.11.11.1/30 WAN Interface Dummy IP

I pointed this out before, but you need to have them in the same subnet. Why not try 10.10.10.1/30 and 10.10.10.2/30 ??

francesco1

@Derelict:

What is a "Dummy IP?"

So you are trying to do HA without the required 3 public IP addresses (minimum /29) on WAN?

You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

Yes, I am trying to do HA without 3 public IP addresses on WAN. What other info do you need? Dummy IP = any IP placed in the IPv4 Address field of the WAN Interface that is not going to conflict with any other interface, host etc. The "Dummy IP" is required because PFS requires you to enter an IP address in that field. And, as I said before, I do NOT have the option of using more than 1 public IP address and overall seems to work.

@dotdash:

@Francesco:

MY SIDE CONFIGURATION

PFS1
10.10.10.1/30 WAN Interface Dummy IP

PFS2
10.11.11.1/30 WAN Interface Dummy IP

I pointed this out before, but you need to have them in the same subnet. Why not try 10.10.10.1/30 and 10.10.10.2/30 ??

Good question. I tried it before and it didn't make a difference. I will try it again.

francesco1

@dotdash:

@Francesco:

MY SIDE CONFIGURATION

PFS1
10.10.10.1/30 WAN Interface Dummy IP

PFS2
10.11.11.1/30 WAN Interface Dummy IP

I pointed this out before, but you need to have them in the same subnet. Why not try 10.10.10.1/30 and 10.10.10.2/30 ??

Just tried it again and same problem.

Derelict

Again - pretty sparse with the details.

francesco1

@Derelict:

Again - pretty sparse with the details.

Again, what more details do you need??? I asked what details you need before. I'll provide the details needed.

Derelict

You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

Help us help you.

francesco1

@Derelict:

You are still not providing the necessary details to properly help you diagnose your problem. Run through the connectivity troubleshooting steps and say what actually fails and where.

Help us help you.

I have no idea what details you are looking for. All I know right now is this:

I start ping command on server on both ends
Unplug Master Firewall WAN connection
I see Slave Firewall takes over when a packet is dropped and ping resumes
Right after that I plug the Master Firewall WAN connection
Master Firewall takes over but all pings fail

Derelict

Details like what are you pinging from where when you're testing. Specifics, like interfaces and IP addresses.

francesco1

I am pinging 77.77.77.40 from 66.66.77.10. I am also pinging 66.66.77.10 from 77.77.77.40.

Diagram of setup is attached. FYI, this is a closed lab setup.

Derelict

What address is 77.77.77.1/X routed to on 66.66.66 ?

francesco1

@Derelict:

What address is 77.77.77.1/X routed to on 66.66.66 ?

I do not understand your question.

Derelict

How do the 77.77.77.X addresses get from the ISP to you? They have to be routed to you somehow.

francesco1

@Derelict:

How do the 77.77.77.X addresses get from the ISP to you? They have to be routed to you somehow.

If you are asking me where the 77.77.77.0/24 addresses come from, that does not matter because this is a lab designed for testing only with no real connection to the Internet. It is only for testing PFS HA functionality.

If you are asking what am I using to route IP traffic between 66.66.66.0/30 and 77.77.77.0/24, I am using a separate installation of PFS.

Derelict

OK it doesn't matter if it is routed to the CARP VIP or not. I'm done.

francesco1

@Derelict:

OK it doesn't matter if it is routed to the CARP VIP or not. I'm done.

That's fine. You weren't reading the details, exaggerating information needed and making things more difficult than they really are.

dotdash

@Francesco:

That's fine. You weren't reading the details, exaggerating information needed and making things more difficult than they really are.

No, he was trying to point out that you could have a dozen different things wrong with your lab setup which no one can easily sort out. e.g. something on the WAN side by default can't ping your LAN, so the fact that 77 whatever can't ping 66 whatever is probably irrelevant; the 'isp router' config is unknown, etc… I'm not sure what you are trying to test with your methodology either, someone unplugging the WAN on the master seems an unlikely event. The HA failures I've dealt with usually involve failed hardware. If I was going to test, I'd pull power on the master and see what happens. Anyway, you are seeking free assistance from strangers on the Internet. If you don't want to work with someone who steps up, fine, but don't be offended if no one else wants to spend time trying to figure out what's wrong with your setup.