pfSense HA LAN Interfaces Only
-
@viragomann When try to ping 1.0.0.1 from 2nd; i get all fails
PING 1.0.0.1 (1.0.0.1): 56 data bytes 92 bytes from 127.0.0.1: Time to live exceeded Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 ee77 0 0000 01 01 0000 127.0.0.1 1.0.0.1 92 bytes from 127.0.0.1: Time to live exceeded Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 4453 0 0000 01 01 0000 127.0.0.1 1.0.0.1 92 bytes from 127.0.0.1: Time to live exceeded Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 071e 0 0000 01 01 0000 127.0.0.1 1.0.0.1 --- 1.0.0.1 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet lossAnd when i run this ping while doing packet cap on master LAN; i dont see the entries come over
-
@iptvcld said in pfSense HA LAN Interfaces Only:
This is the state table after pinging from 2nd
The last two lines look well for only one ping packet. One state entry for LAN and one for WAN.
But the other lines are showing faults at all. The packets column shows the packets for each direction (request, respond). So it shows many requests, but few responds.Not clear, what's going wrong here, now.
Maybe there are some hints in the system log? -
@iptvcld said in pfSense HA LAN Interfaces Only:
When try to ping 1.0.0.1 from 2nd; i get all fails
This IP might be routed out to WAN, since there is no special route for it. But WAN is not connected, hence it will fail.
-
@viragomann This makes sense..
Its just so odd that 1 get 1/3 ping request to pass when doing a ping from 2nd pf. It should almost be all or none..
-
@viragomann So just an over view of what I have done to try to get this going
-
Master Node: Disabled Static Route configuration under HA Sync setting
-
Master Node: Outbound NAT changed to Hybrid and added a mapping for source 127.0.0.0/8 to ANY and NAT Address is LAN address (this synced over to Secondary node)
-
Secondary Node: System-Advanced-Miscellaneous; enabled State Killing on Gateway Failure
-
Secondary Node: System-Routing; Created a new GW using LAN interface with the IP of LAN interface from Master node as the Gateway and 1.1.1.1 for the monitoring IP.
-
Secondary Node: Created a GW Group as PPPOE WAN Tier1, Above noted GW as Tier 2 and VPN GW as Tier 3
-
Secondary Node: The new GW group was set to default under IPv4
-
-
@iptvcld said in pfSense HA LAN Interfaces Only:
Secondary Node: Created a GW Group as PPPOE WAN Tier1, Above noted GW as Tier 2 and VPN GW as Tier 3
The VPN GW has to be set to "never", so that it is no member of this gateway group.
All other settings should be correct.
When you run a ping with 3 requests to 1.1.1.1 on the secondary the packets should go out to the masters LAN and be forwarded to the internet, since the gw monitoring has set the static route for this IP.
When you sniff the traffic filter for ICMP and 1.1.1.1, you should see 3 ICMP request packets (and 3 responds if it works) on
- both masters and secondarys LAN: 192.168.2.81 > 1.1.1.1
- on the masters WAN: WAN IP > 1.1.1.1
If you look into the masters state table you should see
- one entry for LAN: 192.168.2.81 > 1.1.1.1
- and one for WAN: WAN IP > 1.1.1.1
Both should show 3/3 in the packets column.
Not clear, why it doesn't behave this way on your setup.
-
@viragomann I removed all the settings we added, rebooted 2nd node and re-added all the settings as per above and still no go.. I am not able to see the ping req from master packet cap now even when sending a ping from 2nd node. But on the master wan, i can see the 1/1 request which shows that in the ping log on the 2nd node as well 1/3 goes
-
@viragomann I think were on to something... I just did this and i was able to ping 1.1.1.1 3/3 and my 2nd node now has internet..
As a test, i change the Outbound NAT rule from Source 127.0.0.0/8 to any..
bolded textBut.. I know this is not the correct way to leave it; what do you think the issue was with 127.0.0.0/8 as the source?
-
@iptvcld said in pfSense HA LAN Interfaces Only:
I think were on to something... I just did this and i was able to ping 1.1.1.1 3/3 and my 2nd node now has internet..
Interestingly.
I know this is not the correct way to leave it; what do you think the issue was with 127.0.0.0/8 as the source?
It's not really a good idea to have any source natted, at least if you have incoming connections.
Imagine you have a port forwarding to your web server. So packets form the IP 1.2.3.4 is forwarded to the internal server. However, since pfSense translated the source into its LAN address, the web server sees the packets coming from pfSense and you're not able to determine the real source address.Don't know, why 127.0.0.0/8 doesn't work here as source. But as a workaround you can try to set the destination to non-RFC1918 networks (add a proper alias first). So you rule will only be applied to packets which go to the internet.
-
@viragomann
This is very odd for sure!
Even when i change the source to This Firewall - the internet works..
So for your workaround you suggest to put the source back to 127.0.0.0/8 and set the destination to non-RFC1918 via Alias?
Would this still be a good solution if it works? As in, would it cause the issues you noted above?
-
@iptvcld said in pfSense HA LAN Interfaces Only:
Even when i change the source to This Firewall - the internet works..
Nice! That's all you need at all.
Didn't think of this option. -
@viragomann
Amazing Sir... I left it as This Firewall for the source and internet is up and running on backup node!!!Later tonight, I will try swinging the wan over to make sure the tier 1 GW Group takes over as i currently have the group action trigger set to Link Down.
-
@viragomann
I just tested this evening and everything works as i visioned!LAN interfaces are using CARP and what i did was shutdown master node and my backup node became master for the carp LAN interfaces.. Great.. I then saw the GW from master go offline (which is good). I then swang the WAN cable over from master to backup node and right away WAN interface went UP and my GW changed to the PPPOE WAN Tier 1!! which was great.
I then powered on master node and saw my carp LAN interfaces switch back to the master node and once i swang my wan cable back to the master pf node, link went up there and also noticed on my backup node, the GW from master went ONLINE and internet access was flowing!!
One thing i noticed which may be normal after my CARP was active on my backup pf node for a short period before my master taking over again, but I noticed some of devices are showing the DHCP server IP of my backup node when i run ipconfig /all and if i do a release and renew, it will jump back to my master pf node. Is this normal and does this effect anything?
Thank you for all your great work, I have learned ALOT!!!
-
@iptvcld
Did you set the "Failover peer IP" in the DHCP settings on both nodes?The DHCP leases are bound to the hardware interfaces. So the interfaces on both have to have the same name. If you have different hardware or one bare metal installation and one virtual you can work around this with LAGG interfaces to abstract the hardware.
You can find some hints in the pfSense docs:
Modifying the DHCP Server
Troubleshooting High Availability DHCP Failover -
@viragomann
Morning..
Yes my enabled DHCP Server settings have my CARP VIP for the DNS, Gateway and for the Failover peer IP, i have it set to the interface IP of the backup node. (In the case my LAN interface IP) When i saved this, my backup automatically picked these settings up and applied the peer IP back to the master node IP.As for the hardware, I have the same setup on both units.
Master Node Interfaces
LAN Interface (lan, lagg0)
IOT Interface (opt1, lagg0.10)
NOT Interface (opt2, lagg0.20)
SECURITYCAM Interface (opt3, lagg0.30)
VPN Interface (opt4, ovpns1)
SYNC Interface (opt5, em0)Backup Node Interfaces
LAN Interface (lan, lagg0)
IOT Interface (opt1, lagg0.10)
NOT Interface (opt2, lagg0.20)
SECURITYCAM Interface (opt3, lagg0.30)
VPN Interface (opt4, ovpns1)
SYNC Interface (opt5, em0)Status -> DHCP Leases on my nodes show My State: normal and Peer State: normal
I checked Leases on my backup node under status - DHCP Leases and i see a bunch that say online active next to them - and these are also online active showing on the master node.
Then i also see a bunch as offline / active on my backup service and online / active on my master
Not sure if this is normal operation for the dhcp leases to be hand on hand with each other.
-
@iptvcld
Your settings look well and should work this way.Then i also see a bunch as offline / active on my backup service and online / active on my master
I don't use any DHCP on CARP set ups at this time, so I cannot verify.
However, "offline" means that the machine holding the lease is actually not present in the pfSense ARP table. This seems normal as the machines doesn't communicate with the backup node in normal usage.
After you try to access the backup from one of the offline machines (e.g. ping its IP) it's state should become online. -
@viragomann
Not a problem Sir! Might be normal operation as right now i just noticed my cell phone showing online/active on both master and backup nodes (same showing) Check my phone network info and i can see it has a dhcp IP of my backup node. Again not sure if this is the way it should be working.But i will post a new thread if i can not find anything on other forums as well.
-
@iptvcld
I think, the important part is the lease state. Both nodes must be aware of all active DHCP leases. So that used IPs cannot be assigned a second time by the other machine. And that's the case as you mentioned.
Then it doesn't matter which node has issued the lease. -
-
@iptvcld
Out of curiosity, which devices are belonging to your NOT subnet?
