nat/firewall/ha not working

bryanpedini

Hello,

I'm sorry to post in general questions because I almost certainly know this is not the right place (hence, feel free to move the post to whatever topic it fit the most).
However, I'm currently trying to setup a 3 firewalls ha system in my homelab, configured with firewall 1 having ip .2 on the interfaces, firewall 2 ip .3, firewall 3 ip .4 and carp VIPs being .1. (firewall 1 push configs to firewall 2, which push to firewall 3).
As of now I have the following interfaces:

WAN: 10.10.0.0/24
LAN: 192.168.10.0/24 (needs to change in 10.20, someday)
VPN: 10.22.0.0/24
WIFI: 10.30.0.0/24
IOT: 10.35.0.0/24
VOIP: 10.40.0.0/24
SRV: 10.50.0.0/24
MGMT: 10.60.0.0/24
STOR: 10.90.0.0/24

The carp ips I set up seem to work (ipv4 at least), however I'm not able neither to reach the internet, nor the uplink router for that matter.
I'm currently connected to the MGMT interface on subnet 10.60, static address, using the .1 ip as gateway, dns servers don't matter.
Outbound nat is manual with 2 rules, number 1 being FROM 10.60.0.0/24 TO 10.10.0.0/24 use the VIP 10.10.0.1 as source address, while rule 2 is FROM 10.60.0.0/24 TO any DO NOT NAT.
The firewall has a rule that allows everything from MGMT NET to any.

The WAN side of things works this way:
The WAN interface has a private IP address, 10.10.0.x, and the Cisco router in front routes a /29 to this address, hence I configured the 8 addresses as VIPs of 10.10.0.1 (the CARP ip).
I also have a /32 address on a Loopback interface on the Cisco, because I'm trying to release the /29 as I don't need all the addresses anymore and I'm transitioning towards a more "gateway/bastion" system for port forwarding and internal routing of requests, all on the same IP. (hence why the NO NAT, because NAT is handled by the Cisco - which works correctly).

Capturing packets from the MGMT interface I can see my laptop ping the router 10.10.0.254, however capturing from the WAN interface instead I don't see anything.

I'm at a loss, I can't figure out what's not working...
Any help or suggestion towards the right direction is really, really appreciated!
Thanks in advance.

bryanpedini

Okay,

I think I narrowed it down a little, it seems the NAT settings don't get applied for some reason.
I removed all NAT rules and created only one that matches any source&port and any destination&port, and NATs with source the VIP 10.10.0.1 - however - when trying to reach even 1.1.1.1 for example, capturing packets from the WAN interface I see my laptop (10.60.0.14) reaching cloudflare 1.1.1.1 with my laptop private ip as source, instead of the VIP.

This gets more complicated every passing day.
Thanks in advance.

stephenw10

Sounds like it's still running the 'do not NAT rule'.

You still have the 3 firewalls in HA? Are the CARP VIPs correctly working across those? Technically pfSense only supports 2 devices in HA. 3 nodes is basically untested.

Steve

bryanpedini

Yup, seems so.

I can try rolling rebooting the three firewalls and see if it "gets it" the new conf.

Also, three because I have three Proxmox nodes in HA and I want each VM in a different node, but as of now there are no affinity/antiaffinity rules for Proxmox to prevent two VMs running in the same node, hence why these are not managed by Proxmox HA but doing HA via pfSense in itself having one VM per node.
Also yes, the three firewall work fine with the CARPs and everything, the first is the master, if the first goes down there's the second, then the second is master for the third. It's basically working in an active-passive-passive mode or something.

stephenw10

How do you have the config sync setup? 1 > 2, 2 > 3 ?

bryanpedini

correct, 1 is master for 2, which is master for 3.

stephenw10

As long as 3 isn't also syncing to 1 you should be OK.

Check the actual running rules with: pfctl -sr

Make sure they are updated as expected.

bryanpedini

3 is not syncing back to 1, can confirm
however this is my issue with the rules, I don't see nat or anything listed with pfctl -sr
these are my current rules:

floating none
wan
lan
vpn
wifi
iot
voip
srv
mgmt
and finally outbout nat

the two rules for outbound nat from "this firewall" were an attempt to use the firewall interface ip instead of the VIP to allow the firewall to be NATed by the Cisco router and get the Open-VM-Tools package - didn't work - tried moving the VIP over - didn't work, still got error Another instance of pfSense-upgrade is running. Try again later.
one upside is that on the first firewall it worked, that issue is just on firewalls 2 and 3 - and internet access and NAT from clients behind the firewall still isn't working.

again, I'm at a loss - I still, with much evidence, don't fully understand how these things work.
edit: I can't post my ruleset from output of pfctl -sr, it gets flagged by Akismet

stephenw10

@bryanpedini said in nat/firewall/ha not working:

I don't see nat or anything listed with pfctl -sr

You see no rules loaded?

You need to use pfctl -sn to see the NAT rules. See: https://man.freebsd.org/cgi/man.cgi?query=pfctl

@bryanpedini said in nat/firewall/ha not working:

still got error Another instance of pfSense-upgrade is running. Try again later.

That can be a confusing error message. Try running pkg-static -d update at the console and see what error is actually happening.

You should not NAT traffic from the firewall itself. That should use the WAN IP dircetly. NATing that with an over-matching rule (like the source any rule you have) can break some things.

Steve

bryanpedini

@stephenw10 said in nat/firewall/ha not working:

You see no rules loaded?

no, I see rules loaded, tho if I grep for "nat" or something, or for the NAT IP address I find nothing. (might be me that I can't read those).
I'll try to post the rules as a gist since Akismet forbids me to post them here as a code block: here

@stephenw10 said in nat/firewall/ha not working:

You should not NAT traffic from the firewall itself. That should use the WAN IP dircetly.

first firewall: works fine just fine
other firewalls: meh, nope

@stephenw10 said in nat/firewall/ha not working:

Try running pkg-static -d update at the console

[2.7.0-RELEASE][root@pfsense02.home.bjphoster.cloud]/root: pkg-static -d update
DBG(1)[42222]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[42222]> PkgRepo: verifying update for pfSense-core
DBG(1)[42222]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[42222]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-core/meta.conf
DBG(1)[42222]> opening libfetch fetcher
DBG(1)[42222]> Fetch > libfetch: connecting
DBG(1)[42222]> Fetch: fetching from: https://pkg00-atx.netgate.com/pfSense_v2_7_0_amd64-core/meta.conf with opts "i"
DBG(1)[42222]> Fetch: fetcher chosen: https
DBG(1)[42222]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-core/packagesite.pkg
DBG(1)[42222]> opening libfetch fetcher
DBG(1)[42222]> Fetch > libfetch: connecting
DBG(1)[42222]> Fetch: fetching from: https://pkg00-atx.netgate.com/pfSense_v2_7_0_amd64-core/packagesite.pkg with opts "i"
DBG(1)[42222]> Fetch: fetcher chosen: https
DBG(1)[42222]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-core/packagesite.txz
DBG(1)[42222]> opening libfetch fetcher
DBG(1)[42222]> Fetch > libfetch: connecting
DBG(1)[42222]> Fetch: fetching from: https://pkg00-atx.netgate.com/pfSense_v2_7_0_amd64-core/packagesite.txz with opts "i"
DBG(1)[42222]> Fetch: fetcher chosen: https
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
DBG(1)[42222]> PkgRepo: verifying update for pfSense
DBG(1)[42222]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[42222]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-pfSense_v2_7_0/meta.conf
DBG(1)[42222]> opening libfetch fetcher
DBG(1)[42222]> Fetch > libfetch: connecting
DBG(1)[42222]> Fetch: fetching from: https://pkg01-atx.netgate.com/pfSense_v2_7_0_amd64-pfSense_v2_7_0/meta.conf with opts "i"
DBG(1)[42222]> Fetch: fetcher chosen: https
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
DBG(1)[42222]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-pfSense_v2_7_0/packagesite.pkg
DBG(1)[42222]> opening libfetch fetcher
DBG(1)[42222]> Fetch > libfetch: connecting
DBG(1)[42222]> Fetch: fetching from: https://pkg01-atx.netgate.com/pfSense_v2_7_0_amd64-pfSense_v2_7_0/packagesite.pkg with opts "i"
DBG(1)[42222]> Fetch: fetcher chosen: https
Fetching packagesite.pkg: 100%  156 KiB 159.7kB/s    00:01    
DBG(1)[42222]> PkgRepo: extracting packagesite.yaml of repo pfSense
DBG(1)[42485]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[42222]> Pkgrepo, reading new packagesite.yaml for '/var/db/pkg/repo-pfSense.sqlite'
Processing entries: 100%
pfSense repository update completed. 533 packages processed.
All repositories are up to date.
[2.7.0-RELEASE][root@pfsense02.home.bjphoster.cloud]/root:

it works, in console, in gui installing the package does not, same error.

I also tried pkg-static install from console (found here):

[2.7.0-RELEASE][root@pfsense02.home.bjphoster.cloud]/root: pkg-static install -yf pfSense-pkg-Open-VM-Tools
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 4 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        libdnet: 1.13_3 [pfSense]
        libmspack: 0.11alpha [pfSense]
        open-vm-tools-nox11: 12.2.0,2 [pfSense]
        pfSense-pkg-Open-VM-Tools: 10.1.0_5,1 [pfSense]

Number of packages to be installed: 4

The process will require 4 MiB more space.
753 KiB to be downloaded.
[1/4] Fetching libmspack-0.11alpha.pkg: 100%   80 KiB  81.5kB/s    00:01    
[2/4] Fetching libdnet-1.13_3.pkg: 100%   75 KiB  76.8kB/s    00:01    
[3/4] Fetching open-vm-tools-nox11-12.2.0,2.pkg: 100%  591 KiB 605.5kB/s    00:01    
[4/4] Fetching pfSense-pkg-Open-VM-Tools-10.1.0_5,1.pkg: 100%    7 KiB   6.9kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/4] Installing libmspack-0.11alpha...
[1/4] Extracting libmspack-0.11alpha: 100%
[2/4] Installing libdnet-1.13_3...
[2/4] Extracting libdnet-1.13_3: 100%
[3/4] Installing open-vm-tools-nox11-12.2.0,2...
[3/4] Extracting open-vm-tools-nox11-12.2.0,2: 100%
[4/4] Installing pfSense-pkg-Open-VM-Tools-10.1.0_5,1...
[4/4] Extracting pfSense-pkg-Open-VM-Tools-10.1.0_5,1: 100%
Saving updated package information...
done.
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Services... done.
Writing configuration... done.

seems to work fine just fine (is it only the gui?)

stephenw10

@bryanpedini said in nat/firewall/ha not working:

no, I see rules loaded, tho if I grep for "nat" or something, or for the NAT IP address I find nothing.

Yes you need to use pfctl -sn to 'show nat' rules.

@bryanpedini said in nat/firewall/ha not working:

first firewall: works fine just fine
other firewalls: meh, nope

That's what happens if you NAT the firewalls own traffic to the CARP VIP. Only the master node can connect out. Traffic from the firewall itself should not be NAT'ed

There was a bug that affected pkg operations in the gui only though I thought it was only ever in Plus. Check the installed pkg versions: pkg info -x pfsense

Steve

bryanpedini

@stephenw10 said in nat/firewall/ha not working:

Check the installed pkg versions: pkg info -x pfsense

first node I upgraded to pfSense+ version pfSense-23.05.1 while the other two I haven't been able to upgrade yet, still on pfSense-2.7.0

@stephenw10 said in nat/firewall/ha not working:

Yes you need to use pfctl -sn to 'show nat' rules.

oh, you said pfctl -sr to check the running rules - I'm sorry, I don't know these stuff, got it now.

my nat rules output are pretty simple, there's no shit applied: (same rules for all three nodes)

no nat proto carp all
nat-anchor "natearly/*" all
nat-anchor "natrules/*" all
no rdr proto carp all
rdr-anchor "tftp-proxy/*" all

this does not coincide with my "nat everything using the carp vip" rule, or maybe it's my rule that is wrong... idk at this point.

I tried putting my public ip address instead and now it displays the rule (?) but also doesn't work, neither from the firewall Diagnostics -> Ping, nor from a LAN device (???)

stephenw10

Hmm, indeed there are no outbound NAT rules shown there.

Try going to Status > Filter Reload and reload the ruleset. Make sure no errors are shown.

bryanpedini

this worked, in the sense that I now see the rules with pfctl -sn (or maybe it was me changing the rules from using the CARP VIP 10.10.0.1 to a public ip address of mine).

tho I can't still update (nor register for that matter) firewalls 2 and 3 - and now update fails both from GUI and from console on firewall 1 as well... now for example not even ping works from firewall 1 (which is the master for the CARPs and the public IPs configured as VIP of the WAN CARP VIP).

this gets stranger every time I look at it.

bryanpedini

I'm more and more convinced that either I fed up the installation and initial configuration, or I'm fing up now with all these tries and confs - 'cause I'm an idiot - and still don't know the hell these things work...

stephenw10

Ok, so what NAT rules do you have there? If they are over-matching traffic from the firewall itself and NATing to the CARP VIP then only the master node will be able to connect. Even then it can cause problems for some protocols. pkg functions would not normally be one of them but...

bryanpedini

I NAT everything coming from any host port 500 (ISAKMP) to any destination port 500 (ISAKMP) with static port with the public IP (VIP), and then another rule NATting anything everything coming from any host any port to any destination any port with the public IP (VIP).

the public IP is a VIP of 10.10.0.1 (the CARP VIP), which is currently on firewall 1.
this is done because I have multiple IP addresses I route from the Cisco router to the firewalls via the CARP VIP, and would need to decide which IP address to use outbound for the specific NATted virtual machines.

stephenw10

Ok, so your second rule there should not be source 'any' because that will include the firewall WAN IP which should not be NAT'd.

Either add NAT rules for each of your internal subnets as source or create an alias with that subnets in it and use that as the source in a single rule.
https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html#configure-outbound-nat-for-carp

Steve

bryanpedini

okay now I got a strange one at hands...

this be my alias configuration

this be my nat configuration (I verified, it got propagated to all three machines)

when trying to ping 1.1.1.1 from pfsense01 it works, when trying to ping from either pfsense02 or pfsense03 not only it doesn't work, trying to perform a packet capture from the same firewall it doesn't output anything, like if it doesn't even try (and of course it says I don't have network connectivity to register the firewall to pfSense+ or to update packages or whatever)...

tried moving the CARP VIPs (all of them) to pfsense02 with - again - zero luck

bryanpedini

PS: fuck IPv4 - it doesn't work - IPv6 works fine just fine from all three machines pinging google.com (tho only from the WAN interface, the other IP addresses are routed to the CARP VIP so I suppose it's normal it doesn't work)