Backup Node Normal Behavior
-
Hi,
I configured pfSense in HA configuration with 2 WANs and 2 LANs, and I noticed that backup node is unable to ping any of the interface gateways until the master node has that interface disabled. I'm not able to ping the master node from backup node either. Is that normal? Or am I supposed to be able to ping gateway, master/backup node and internet IPs from both pfSense units at all times?
Thanks in advance.
-
@CaptainKeyboard
Not clear, what you mean with gateway here
But you should be able to ping any interface IP of the master as well as the CARP VIPs from the backup node, as long as the firewall rules on the master allow this -
Hmm... that's clearly not happening, so something is wrong on my side. By Gateways, I meant the gateway that is on the WAN side provided by the each ISP, and on LAN side the CARP which is supposed to be gateway for DHCP clients that are assigned IPs on LAN side.
Does the behaviour I described suggest multicast related problem or pfSense misconfiguration?
-
@CaptainKeyboard
Pinging a device in the local subnet doesn't reqire multicast.How did you configure the interfaces?
Did you set the network mask properly?The only case, where you cannot access the WAN gateway from the backup node, is when your CARP is your only single public IP. Because the backup has no IP in this subnet then.
-
@viragomann This is how I have things set up.
I'll go over WAN1 but other three networks (WAN2, LAN1 and LAN2) are set up the same way.
pfSenseA and pfSenseB are identical hardware (Dell servers), CPU, RAM Dimms, Network cards, firmware, HDDs are all identical. Ports are assigned to each interface in identical manner.
pfSenseA has two network ports configured as LAGG (failover). pfSenseB has the same LAGG configured on it. Each of the active ports from each LAGG is connected to a Dell Force10 switch. On pfSense side, I have unique public static IP set on each pfSense. For example, pfSenseA is x.y.z.201, pfSenseB is x.y.z.202, and the CARP I created between them for WAN1 is x.y.z.200. CARP status shows master/backup status correctly. WAN1's ISP said that gateway is x.y.z.199. On each pfSense's WAN interface, that is the gateway set. I also have an outbound NAT entry to send all outbound traffic via CARP (x.y.z.200). Firewall rules on WAN1 have entry to allow access from LAN1 and LAN2.
I can ping x.y.z.199 from pfSenseA, but I cannot ping pfSenseB's WAN1 IP (x.y.z.202). From pfSenseB, I can't ping anything. On the Dell switch, I have igmp snooping enabled on those interfaces, flood limit set to 200. Interfaces on the switch are part of a VLAN, not sure if that matters.
When I change LAGG to LACP from failover, it has no impact as things are still half broken (i.e. pfSenseB seems disconnected from the network). The only way I can bring pfSenseB to ping the internet and x.y.z.199 is by disabling the WAN1 interface on pfSenseA.
-
@CaptainKeyboard said in Backup Node Normal Behavior:
For example, pfSenseA is x.y.z.201, pfSenseB is x.y.z.202, and the CARP I created between them for WAN1 is x.y.z.200. CARP status shows master/backup status correctly. WAN1's ISP said that gateway is x.y.z.199. On each pfSense's WAN interface, that is the gateway set.
So both should have access to the gateway.
I also have an outbound NAT entry to send all outbound traffic via CARP (x.y.z.200).
Did you set this also for 127.0.0.0/8?
Firewall rules on WAN1 have entry to allow access from LAN1 and LAN2.
The WAN interface should not see traffic form LANs. So this rule wouldn't make any sense.
-
@viragomann Very Interesting....
I removed the unecessary firewall rules. I thought it was necessary to allow LAN to have internet access.
127.0.0.0/8 is part of Automatic Rules.
-
@CaptainKeyboard said in Backup Node Normal Behavior:
I removed the unecessary firewall rules. I thought it was necessary to allow LAN to have internet access.
Firewall rule have to be added to the interface, where the traffic is coming in.
So to allow internet access for LAN devices, you need a rule on LAN.127.0.0.0/8 is part of Automatic Rules.
This is correct. So each node use its interface address, e.g. its WAN address to access to WAN gateway.
-
@viragomann I'm not sure how to troubleshoot this.
-
@viragomann When I do packet capture on both pfSense units for the duration of ping. I see packets leaving from CARP IP. Could that be a problem? Maybe the response to all traffic from pfSenseB is leaving from CARP and returned to pfSenseA, which in turn discards it?
-
@viragomann After changing the Outbound NAT to not have "any" for Source, that improved the situation. Not pfSenseB can ping the WAN gateway and access the web. It still cannot ping pfSenseA's WAN IP or CARP IP.
-
@CaptainKeyboard said in Backup Node Normal Behavior:
When I do packet capture on both pfSense units for the duration of ping. I see packets leaving from CARP IP. Could that be a problem?
If you ping from one pfSense node to the other, the source should be the interface address, not the CARP.
After changing the Outbound NAT to not have "any" for Source, that improved the situation.
Yeah, you must not use any here. Manual rules override automatic rules.
-
@viragomann I needed outbound traffic to go through WAN CARP as public facing domains would be pointing to that IP, which is why I had the NAT rule for outbound NATing outgoing traffic to CARP. I disabled it for now. There is no change in behaviour. pfSenseB and A can ping WAN gateway and the web, but not eachother. I'm pretty sure this used to work until I set up CARP on WAN and set up Gateway Group for Load Balancing outgoing traffic across WAN1 and WAN2.
-
@CaptainKeyboard
Yes, special outbound NAT rules are needed, when running HA.We usually set the outbound NAT into manual mode.
pfSense then converts all automatic rules into manual.
This ends up, that you have a separate rule for each network you've defined on any internal interface and for pfSense itself (127.0.0.0/8), each one rule for ISAKMP and one for the rest.Then you have to chance all rules for your internal networks to CARP VIP, but keep the 127/8.
-
@viragomann Ah, ok. I had NAT rules set in Hybrid mode. I changed it to Manual just now and saw pfSense merge them to Automatically generated rules. It won't let me ping the other pfSense. It does let me ping the WAN gateway so that's good. At least the second unit is no longer offline.
-
@viragomann It's working now. I added a Firewall Rule to allow ICMP traffic and that fixed it! I can't believe I forgot that.
Thanks a lot for your help in troubleshooting this. I greatly appreciate it!
-
@CaptainKeyboard
The hint to consider rule was in my first post.But glad, that's working now.