pfSense HA LAN Interfaces Only

CloudNode · Sep 20, 2021, 5:14 PM

@viragomann Ran a packet cap on master and i see this request 3 times

13:10:40.370864 IP 192.168.2.81 > 1.1.1.1: ICMP echo request, id 18433, seq 2360, length 9

seems to be coming from the secondary pf LAN IP still and not the WAN IP

Outbound NAT on master is this:

And LAN Rules

CloudNode · Sep 20, 2021, 5:19 PM

@viragomann Ran it again and i see this

13:17:05.527815 IP 76.64.x.x > 1.1.1.1: ICMP echo request, id 55322, seq 0, length 64
13:17:05.542715 IP 1.1.1.1 > 76.64.x.x: ICMP echo reply, id 55322, seq 0, length 64

then

13:17:05.562166 IP 192.168.2.81 > 1.1.1.1: ICMP echo request, id 34782, seq 78, length 9

PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=55 time=15.187 ms

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 1 packets received, 66.7% packet loss
round-trip min/avg/max/stddev = 15.187/15.187/15.187/0.000 ms

CloudNode · Sep 20, 2021, 5:25 PM

@viragomann Does this need to be unchecked under my WAN interface?
Block private networks and loopback addresses

viragomann · Sep 20, 2021, 5:27 PM

@iptvcld said in pfSense HA LAN Interfaces Only:

Ran it again and i see this
13:17:05.527815 IP 76.64.x.x > 1.1.1.1: ICMP echo request, id 55322, seq 0, length 64
13:17:05.542715 IP 1.1.1.1 > 76.64.x.x: ICMP echo reply, id 55322, seq 0, length 64

then
13:17:05.562166 IP 192.168.2.81 > 1.1.1.1: ICMP echo request, id 34782, seq 78, length 9

PING 1.1.1.1 (1.1.1.1): 56 data bytes

Strange!

Check the masters state table and filter for 1.1.1.1 after pinging from backup.

viragomann · Sep 20, 2021, 5:28 PM

@iptvcld said in pfSense HA LAN Interfaces Only:

@viragomann Does this need to be unchecked under my WAN interface?
Block private networks and loopback addresses

No, this is only for incoming packets. There is no need to allow private addresses on WAN.

CloudNode · Sep 20, 2021, 5:31 PM

This post is deleted!

CloudNode · Sep 20, 2021, 5:33 PM

@viragomann This is the state table after pinging from 2nd

CloudNode · Sep 20, 2021, 5:45 PM

@viragomann When try to ping 1.0.0.1 from 2nd; i get all fails

PING 1.0.0.1 (1.0.0.1): 56 data bytes
92 bytes from 127.0.0.1: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 ee77   0 0000  01  01 0000 127.0.0.1  1.0.0.1 

92 bytes from 127.0.0.1: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 4453   0 0000  01  01 0000 127.0.0.1  1.0.0.1 

92 bytes from 127.0.0.1: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 071e   0 0000  01  01 0000 127.0.0.1  1.0.0.1 


--- 1.0.0.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

And when i run this ping while doing packet cap on master LAN; i dont see the entries come over

viragomann · Sep 20, 2021, 6:01 PM

@iptvcld said in pfSense HA LAN Interfaces Only:

This is the state table after pinging from 2nd

The last two lines look well for only one ping packet. One state entry for LAN and one for WAN.
But the other lines are showing faults at all. The packets column shows the packets for each direction (request, respond). So it shows many requests, but few responds.

Not clear, what's going wrong here, now.
Maybe there are some hints in the system log?

viragomann · Sep 20, 2021, 6:02 PM

@iptvcld said in pfSense HA LAN Interfaces Only:

When try to ping 1.0.0.1 from 2nd; i get all fails

This IP might be routed out to WAN, since there is no special route for it. But WAN is not connected, hence it will fail.

CloudNode · Sep 20, 2021, 6:03 PM

@viragomann This makes sense..

Its just so odd that 1 get 1/3 ping request to pass when doing a ping from 2nd pf. It should almost be all or none..

CloudNode · Sep 20, 2021, 6:19 PM

@viragomann So just an over view of what I have done to try to get this going

Master Node: Disabled Static Route configuration under HA Sync setting
Master Node: Outbound NAT changed to Hybrid and added a mapping for source 127.0.0.0/8 to ANY and NAT Address is LAN address (this synced over to Secondary node)
Secondary Node: System-Advanced-Miscellaneous; enabled State Killing on Gateway Failure
Secondary Node: System-Routing; Created a new GW using LAN interface with the IP of LAN interface from Master node as the Gateway and 1.1.1.1 for the monitoring IP.
Secondary Node: Created a GW Group as PPPOE WAN Tier1, Above noted GW as Tier 2 and VPN GW as Tier 3
Secondary Node: The new GW group was set to default under IPv4

viragomann · Sep 20, 2021, 7:39 PM

@iptvcld said in pfSense HA LAN Interfaces Only:

Secondary Node: Created a GW Group as PPPOE WAN Tier1, Above noted GW as Tier 2 and VPN GW as Tier 3

The VPN GW has to be set to "never", so that it is no member of this gateway group.

All other settings should be correct.

When you run a ping with 3 requests to 1.1.1.1 on the secondary the packets should go out to the masters LAN and be forwarded to the internet, since the gw monitoring has set the static route for this IP.

When you sniff the traffic filter for ICMP and 1.1.1.1, you should see 3 ICMP request packets (and 3 responds if it works) on

both masters and secondarys LAN: 192.168.2.81 > 1.1.1.1
on the masters WAN: WAN IP > 1.1.1.1

If you look into the masters state table you should see

one entry for LAN: 192.168.2.81 > 1.1.1.1
and one for WAN: WAN IP > 1.1.1.1

Both should show 3/3 in the packets column.

Not clear, why it doesn't behave this way on your setup.

CloudNode · Sep 20, 2021, 8:15 PM

@viragomann I removed all the settings we added, rebooted 2nd node and re-added all the settings as per above and still no go.. I am not able to see the ping req from master packet cap now even when sending a ping from 2nd node. But on the master wan, i can see the 1/1 request which shows that in the ping log on the 2nd node as well 1/3 goes

CloudNode · Sep 20, 2021, 8:55 PM

@viragomann I think were on to something... I just did this and i was able to ping 1.1.1.1 3/3 and my 2nd node now has internet..

As a test, i change the Outbound NAT rule from Source 127.0.0.0/8 to any..

bolded text

But.. I know this is not the correct way to leave it; what do you think the issue was with 127.0.0.0/8 as the source?

viragomann · Sep 20, 2021, 9:05 PM

@iptvcld said in pfSense HA LAN Interfaces Only:

I think were on to something... I just did this and i was able to ping 1.1.1.1 3/3 and my 2nd node now has internet..

Interestingly.

I know this is not the correct way to leave it; what do you think the issue was with 127.0.0.0/8 as the source?

It's not really a good idea to have any source natted, at least if you have incoming connections.
Imagine you have a port forwarding to your web server. So packets form the IP 1.2.3.4 is forwarded to the internal server. However, since pfSense translated the source into its LAN address, the web server sees the packets coming from pfSense and you're not able to determine the real source address.

Don't know, why 127.0.0.0/8 doesn't work here as source. But as a workaround you can try to set the destination to non-RFC1918 networks (add a proper alias first). So you rule will only be applied to packets which go to the internet.

CloudNode · Sep 20, 2021, 9:13 PM

@viragomann
This is very odd for sure!
Even when i change the source to This Firewall - the internet works..

So for your workaround you suggest to put the source back to 127.0.0.0/8 and set the destination to non-RFC1918 via Alias?

Would this still be a good solution if it works? As in, would it cause the issues you noted above?

viragomann · Sep 20, 2021, 9:16 PM

@iptvcld said in pfSense HA LAN Interfaces Only:

Even when i change the source to This Firewall - the internet works..

Nice! That's all you need at all.
Didn't think of this option.

CloudNode · Sep 20, 2021, 9:49 PM

@viragomann
Amazing Sir... I left it as This Firewall for the source and internet is up and running on backup node!!!

Later tonight, I will try swinging the wan over to make sure the tier 1 GW Group takes over as i currently have the group action trigger set to Link Down.

CloudNode · Sep 21, 2021, 2:16 AM

@viragomann
I just tested this evening and everything works as i visioned!

LAN interfaces are using CARP and what i did was shutdown master node and my backup node became master for the carp LAN interfaces.. Great.. I then saw the GW from master go offline (which is good). I then swang the WAN cable over from master to backup node and right away WAN interface went UP and my GW changed to the PPPOE WAN Tier 1!! which was great.

I then powered on master node and saw my carp LAN interfaces switch back to the master node and once i swang my wan cable back to the master pf node, link went up there and also noticed on my backup node, the GW from master went ONLINE and internet access was flowing!!

One thing i noticed which may be normal after my CARP was active on my backup pf node for a short period before my master taking over again, but I noticed some of devices are showing the DHCP server IP of my backup node when i run ipconfig /all and if i do a release and renew, it will jump back to my master pf node. Is this normal and does this effect anything?

Thank you for all your great work, I have learned ALOT!!!