Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    nat/firewall/ha not working

    Scheduled Pinned Locked Moved General pfSense Questions
    25 Posts 2 Posters 2.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator @bryanpedini
      last edited by

      @bryanpedini said in nat/firewall/ha not working:

      I don't see nat or anything listed with pfctl -sr

      You see no rules loaded?

      You need to use pfctl -sn to see the NAT rules. See: https://man.freebsd.org/cgi/man.cgi?query=pfctl

      @bryanpedini said in nat/firewall/ha not working:

      still got error Another instance of pfSense-upgrade is running. Try again later.

      That can be a confusing error message. Try running pkg-static -d update at the console and see what error is actually happening.

      You should not NAT traffic from the firewall itself. That should use the WAN IP dircetly. NATing that with an over-matching rule (like the source any rule you have) can break some things.

      Steve

      B 1 Reply Last reply Reply Quote 0
      • B
        bryanpedini @stephenw10
        last edited by bryanpedini

        @stephenw10 said in nat/firewall/ha not working:

        You see no rules loaded?

        no, I see rules loaded, tho if I grep for "nat" or something, or for the NAT IP address I find nothing. (might be me that I can't read those).
        I'll try to post the rules as a gist since Akismet forbids me to post them here as a code block: here

        @stephenw10 said in nat/firewall/ha not working:

        You should not NAT traffic from the firewall itself. That should use the WAN IP dircetly.

        first firewall: works fine just fine
        other firewalls: meh, nope

        @stephenw10 said in nat/firewall/ha not working:

        Try running pkg-static -d update at the console

        [2.7.0-RELEASE][root@pfsense02.home.bjphoster.cloud]/root: pkg-static -d update
        DBG(1)[42222]> pkg initialized
        Updating pfSense-core repository catalogue...
        DBG(1)[42222]> PkgRepo: verifying update for pfSense-core
        DBG(1)[42222]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
        DBG(1)[42222]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-core/meta.conf
        DBG(1)[42222]> opening libfetch fetcher
        DBG(1)[42222]> Fetch > libfetch: connecting
        DBG(1)[42222]> Fetch: fetching from: https://pkg00-atx.netgate.com/pfSense_v2_7_0_amd64-core/meta.conf with opts "i"
        DBG(1)[42222]> Fetch: fetcher chosen: https
        DBG(1)[42222]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-core/packagesite.pkg
        DBG(1)[42222]> opening libfetch fetcher
        DBG(1)[42222]> Fetch > libfetch: connecting
        DBG(1)[42222]> Fetch: fetching from: https://pkg00-atx.netgate.com/pfSense_v2_7_0_amd64-core/packagesite.pkg with opts "i"
        DBG(1)[42222]> Fetch: fetcher chosen: https
        DBG(1)[42222]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-core/packagesite.txz
        DBG(1)[42222]> opening libfetch fetcher
        DBG(1)[42222]> Fetch > libfetch: connecting
        DBG(1)[42222]> Fetch: fetching from: https://pkg00-atx.netgate.com/pfSense_v2_7_0_amd64-core/packagesite.txz with opts "i"
        DBG(1)[42222]> Fetch: fetcher chosen: https
        pfSense-core repository is up to date.
        Updating pfSense repository catalogue...
        DBG(1)[42222]> PkgRepo: verifying update for pfSense
        DBG(1)[42222]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
        DBG(1)[42222]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-pfSense_v2_7_0/meta.conf
        DBG(1)[42222]> opening libfetch fetcher
        DBG(1)[42222]> Fetch > libfetch: connecting
        DBG(1)[42222]> Fetch: fetching from: https://pkg01-atx.netgate.com/pfSense_v2_7_0_amd64-pfSense_v2_7_0/meta.conf with opts "i"
        DBG(1)[42222]> Fetch: fetcher chosen: https
        Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
        DBG(1)[42222]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-pfSense_v2_7_0/packagesite.pkg
        DBG(1)[42222]> opening libfetch fetcher
        DBG(1)[42222]> Fetch > libfetch: connecting
        DBG(1)[42222]> Fetch: fetching from: https://pkg01-atx.netgate.com/pfSense_v2_7_0_amd64-pfSense_v2_7_0/packagesite.pkg with opts "i"
        DBG(1)[42222]> Fetch: fetcher chosen: https
        Fetching packagesite.pkg: 100%  156 KiB 159.7kB/s    00:01    
        DBG(1)[42222]> PkgRepo: extracting packagesite.yaml of repo pfSense
        DBG(1)[42485]> PkgRepo: extracting signature of repo in a sandbox
        DBG(1)[42222]> Pkgrepo, reading new packagesite.yaml for '/var/db/pkg/repo-pfSense.sqlite'
        Processing entries: 100%
        pfSense repository update completed. 533 packages processed.
        All repositories are up to date.
        [2.7.0-RELEASE][root@pfsense02.home.bjphoster.cloud]/root:
        

        it works, in console, in gui installing the package does not, same error.

        I also tried pkg-static install from console (found here):

        [2.7.0-RELEASE][root@pfsense02.home.bjphoster.cloud]/root: pkg-static install -yf pfSense-pkg-Open-VM-Tools
        Updating pfSense-core repository catalogue...
        pfSense-core repository is up to date.
        Updating pfSense repository catalogue...
        pfSense repository is up to date.
        All repositories are up to date.
        The following 4 package(s) will be affected (of 0 checked):
        
        New packages to be INSTALLED:
                libdnet: 1.13_3 [pfSense]
                libmspack: 0.11alpha [pfSense]
                open-vm-tools-nox11: 12.2.0,2 [pfSense]
                pfSense-pkg-Open-VM-Tools: 10.1.0_5,1 [pfSense]
        
        Number of packages to be installed: 4
        
        The process will require 4 MiB more space.
        753 KiB to be downloaded.
        [1/4] Fetching libmspack-0.11alpha.pkg: 100%   80 KiB  81.5kB/s    00:01    
        [2/4] Fetching libdnet-1.13_3.pkg: 100%   75 KiB  76.8kB/s    00:01    
        [3/4] Fetching open-vm-tools-nox11-12.2.0,2.pkg: 100%  591 KiB 605.5kB/s    00:01    
        [4/4] Fetching pfSense-pkg-Open-VM-Tools-10.1.0_5,1.pkg: 100%    7 KiB   6.9kB/s    00:01    
        Checking integrity... done (0 conflicting)
        [1/4] Installing libmspack-0.11alpha...
        [1/4] Extracting libmspack-0.11alpha: 100%
        [2/4] Installing libdnet-1.13_3...
        [2/4] Extracting libdnet-1.13_3: 100%
        [3/4] Installing open-vm-tools-nox11-12.2.0,2...
        [3/4] Extracting open-vm-tools-nox11-12.2.0,2: 100%
        [4/4] Installing pfSense-pkg-Open-VM-Tools-10.1.0_5,1...
        [4/4] Extracting pfSense-pkg-Open-VM-Tools-10.1.0_5,1: 100%
        Saving updated package information...
        done.
        Loading package configuration... done.
        Configuring package components...
        Loading package instructions...
        Custom commands...
        Executing custom_php_install_command()...done.
        Services... done.
        Writing configuration... done.
        

        seems to work fine just fine (is it only the gui?)

        stephenw10S 1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator @bryanpedini
          last edited by

          @bryanpedini said in nat/firewall/ha not working:

          no, I see rules loaded, tho if I grep for "nat" or something, or for the NAT IP address I find nothing.

          Yes you need to use pfctl -sn to 'show nat' rules.

          @bryanpedini said in nat/firewall/ha not working:

          first firewall: works fine just fine
          other firewalls: meh, nope

          That's what happens if you NAT the firewalls own traffic to the CARP VIP. Only the master node can connect out. Traffic from the firewall itself should not be NAT'ed

          There was a bug that affected pkg operations in the gui only though I thought it was only ever in Plus. Check the installed pkg versions: pkg info -x pfsense

          Steve

          B 1 Reply Last reply Reply Quote 0
          • B
            bryanpedini @stephenw10
            last edited by

            @stephenw10 said in nat/firewall/ha not working:

            Check the installed pkg versions: pkg info -x pfsense

            first node I upgraded to pfSense+ version pfSense-23.05.1 while the other two I haven't been able to upgrade yet, still on pfSense-2.7.0

            @stephenw10 said in nat/firewall/ha not working:

            Yes you need to use pfctl -sn to 'show nat' rules.

            oh, you said pfctl -sr to check the running rules - I'm sorry, I don't know these stuff, got it now.

            my nat rules output are pretty simple, there's no shit applied: (same rules for all three nodes)

            no nat proto carp all
            nat-anchor "natearly/*" all
            nat-anchor "natrules/*" all
            no rdr proto carp all
            rdr-anchor "tftp-proxy/*" all
            

            this does not coincide with my "nat everything using the carp vip" rule, or maybe it's my rule that is wrong... idk at this point.

            I tried putting my public ip address instead and now it displays the rule (?) but also doesn't work, neither from the firewall Diagnostics -> Ping, nor from a LAN device (???)

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              Hmm, indeed there are no outbound NAT rules shown there.

              Try going to Status > Filter Reload and reload the ruleset. Make sure no errors are shown.

              B 1 Reply Last reply Reply Quote 0
              • B
                bryanpedini @stephenw10
                last edited by

                this worked, in the sense that I now see the rules with pfctl -sn (or maybe it was me changing the rules from using the CARP VIP 10.10.0.1 to a public ip address of mine).

                tho I can't still update (nor register for that matter) firewalls 2 and 3 - and now update fails both from GUI and from console on firewall 1 as well... now for example not even ping works from firewall 1 (which is the master for the CARPs and the public IPs configured as VIP of the WAN CARP VIP).

                this gets stranger every time I look at it.

                B 1 Reply Last reply Reply Quote 0
                • B
                  bryanpedini @bryanpedini
                  last edited by

                  I'm more and more convinced that either I fed up the installation and initial configuration, or I'm fing up now with all these tries and confs - 'cause I'm an idiot - and still don't know the hell these things work...

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Ok, so what NAT rules do you have there? If they are over-matching traffic from the firewall itself and NATing to the CARP VIP then only the master node will be able to connect. Even then it can cause problems for some protocols. pkg functions would not normally be one of them but...

                    1 Reply Last reply Reply Quote 0
                    • B
                      bryanpedini
                      last edited by

                      I NAT everything coming from any host port 500 (ISAKMP) to any destination port 500 (ISAKMP) with static port with the public IP (VIP), and then another rule NATting anything everything coming from any host any port to any destination any port with the public IP (VIP).

                      the public IP is a VIP of 10.10.0.1 (the CARP VIP), which is currently on firewall 1.
                      this is done because I have multiple IP addresses I route from the Cisco router to the firewalls via the CARP VIP, and would need to decide which IP address to use outbound for the specific NATted virtual machines.

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Ok, so your second rule there should not be source 'any' because that will include the firewall WAN IP which should not be NAT'd.

                        Either add NAT rules for each of your internal subnets as source or create an alias with that subnets in it and use that as the source in a single rule.
                        https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html#configure-outbound-nat-for-carp

                        Steve

                        1 Reply Last reply Reply Quote 0
                        • B
                          bryanpedini
                          last edited by

                          okay now I got a strange one at hands...

                          this be my alias configuration
                          7dc13298-f8b6-493c-b985-00fd2e35b495-image.png

                          this be my nat configuration (I verified, it got propagated to all three machines)
                          d3c0c3f5-4301-43fd-9871-00d8a7ac4b66-image.png

                          when trying to ping 1.1.1.1 from pfsense01 it works, when trying to ping from either pfsense02 or pfsense03 not only it doesn't work, trying to perform a packet capture from the same firewall it doesn't output anything, like if it doesn't even try (and of course it says I don't have network connectivity to register the firewall to pfSense+ or to update packages or whatever)...

                          tried moving the CARP VIPs (all of them) to pfsense02 with - again - zero luck

                          1 Reply Last reply Reply Quote 0
                          • B
                            bryanpedini
                            last edited by

                            PS: fuck IPv4 - it doesn't work - IPv6 works fine just fine from all three machines pinging google.com (tho only from the WAN interface, the other IP addresses are routed to the CARP VIP so I suppose it's normal it doesn't work)

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Hmm, I assume that IP you are NATing to is the CARP VIP?

                              On the primary node that works check the states created when it checks for updates. What source IP is it using? The CARP VIP?

                              We did see issues with the primary IP address on an interface moving to a VIP on it but that should be solved. Also I never saw it affect pkg operations.

                              B 1 Reply Last reply Reply Quote 0
                              • B
                                bryanpedini @stephenw10
                                last edited by

                                @stephenw10 yes, but actually not really...

                                the configuration is like this:
                                10.10.0.1 is the CARP VIP of the interface
                                213.137.54.24/29 (to be exact 8 single /32 IPs) are virtual IPs based on the CARP VIP

                                this is done to more quickly move all the IPs following the CARP VIP and to minimize CARP traffic between the firewalls

                                @stephenw10 said in nat/firewall/ha not working:

                                On the primary node that works check the states created when it checks for updates

                                with three machines and that many interfaces there are too much states to go through to analyze
                                can you share the update servers' subnet? is 208.123.73.212 part of it? (if so, from the states it seems the source IP is 10.10.0.2, the IP of the interface, so not NATted)

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  Ah so the IPs from 213.137.54.24/29 are IPAliases that are on the 10.10.0.1 CARP VIP?

                                  What is NATing the 10.10.0.X subnet then, some upstream router?

                                  If there's nothing NATing that then the nodes would have to use IPs from the public /29 directly. But since those are on the CARP VIP only the master node can use it.

                                  Why not use IPs from the /29 directly on the nodes?

                                  B 1 Reply Last reply Reply Quote 0
                                  • stephenw10S stephenw10 referenced this topic on
                                  • B
                                    bryanpedini @stephenw10
                                    last edited by

                                    @stephenw10 said in nat/firewall/ha not working:

                                    What is NATing the 10.10.0.X subnet then, some upstream router?

                                    correct, the Cisco router

                                    @stephenw10 said in nat/firewall/ha not working:

                                    Why not use IPs from the /29 directly on the nodes?

                                    a. to have all of them available
                                    b. there will be no /29, I'm slowly migrating services to a single /32

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Hmm, then there should be no problem with them using the primary IP in the 10.10.0X subnet as long as it' not the CARP VIP.

                                      Do you not see states at all on the other nodes?

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post
                                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.