Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Growing states table, some leak?

    Plus 23.09 Development Snapshots (Retired)
    7
    41
    5.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • w0wW
      w0w
      last edited by

      Subtotals.

      Primary firewall               Secondary firewall                   Ghost states
      
      23.09                               23.05                                    NO
      23.09                               23.09                                    YES
      23.05                               23.09                                    YES
      23.05                               23.05                                    NO
      
      

      Additional tests:
      I connected a virtual machine as a second firewall through a real interface (SYNC), where everything is configured minimally, the number of states immediately starts to increase in virtual machine too.
      Two virtual machines on 23.09 between each other—no problem, but the test cannot be considered complete, since there is very little real traffic there, perhaps some kind of trigger is missing. Later, there is an idea to replace the main firewall with a virtual machine.
      Also tried to replace SYNC interface on both machines with USB card, this changed nothing so at least NIC driver is not suspected, because it is absolutely different vendors all the way.

      w0wW 1 Reply Last reply Reply Quote 0
      • w0wW
        w0w @w0w
        last edited by

        Virtual machine configured as primary showed this in the CARP maintenance
        94eb4e22-28f2-434c-af0f-40e4d2c4824e-image.png
        Minimal config. Just one WAN, LAN and SYNC.
        When it becomes “master”, not “backup” then states looks normal on both, not growing. 😑

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          I ran this by Kristof and he has a couple different theories about what might be happening, but it's not clear exactly based just on the information in the thread, especially since we can't seem to reproduce it in a lab setup.

          There are a few ways to help gather info:

          • Install a regular kernel from https://www.codepro.be/files/pfSense-kernel-pfSense-23.09.a.20230905.1950.pkg
          • Install a debug kernel from https://www.codepro.be/files/pfSense-kernel-debug-pfSense-23.09.a.20230905.1950.pkg

          There are potentially some issues with error handling in pf_create_state() that could cause states to be allocated and then lost before they’re connected in the state table. That matches the problem description, although it's not clear how or why this would suddenly start manifesting, and doing so on an inconsistent basis.

          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          w0wW 1 Reply Last reply Reply Quote 1
          • w0wW
            w0w @jimp
            last edited by

            @jimp
            Thank you for your time and attention.
            Installed debug kernel. What should I do next?

            1 Reply Last reply Reply Quote 0
            • jimpJ
              jimp Rebel Alliance Developer Netgate
              last edited by

              Once you are booted into the debug kernel, see if you can reproduce the problem.

              If you can, see if anything additional shows up in the system log.

              If you don't see the problem on the debug kernel, then that may also confirm that what Kristof attempted to fix there was the actual problem.

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              w0wW 1 Reply Last reply Reply Quote 0
              • w0wW
                w0w @jimp
                last edited by w0w

                @jimp
                I see it's growing, but not reached its limit. When should I expect anything additional in logs? Can you provide some keywords?

                EDIT: Ahh wait... forgot to select the right kernel :)

                1 Reply Last reply Reply Quote 0
                • w0wW
                  w0w
                  last edited by

                  I regret to inform you that the firewall is not available
                  because of
                  80f9a7a8-6de0-49d6-820f-5ddb41208d94-image.png

                  I don't see anything useful in the log files, nothing new, nothing related to the problem.

                  1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    Just to confirm, in each of your tests these systems have state synchronization (pfsync) configured and enabled, right?

                    Is it enabled with an IP address for the peer filled in or just enabled and left blank?

                    Is the sync interface private between the two HA nodes alone? Or could there be something else on the sync segment also doing pfsync?

                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    w0wW 1 Reply Last reply Reply Quote 0
                    • w0wW
                      w0w @jimp
                      last edited by

                      @jimp said in Growing states table, some leak?:

                      Just to confirm, in each of your tests these systems have state synchronization (pfsync) configured and enabled, right?

                      Yes, exactly

                      @jimp said in Growing states table, some leak?:

                      Is it enabled with an IP address for the peer filled in or just enabled and left blank?

                      'pfsync Synchronize Peer IP', 'Synchronize Config to IP' are filled on the primary node with peer IP address, on the secondary 'pfsync Synchronize Peer IP' only filled.
                      'Set a custom Filter Host ID' is left blank, but I see those generated IDs on both nodes.

                      @jimp said in Growing states table, some leak?:

                      Is the sync interface private between the two HA nodes alone? Or could there be something else on the sync segment also doing pfsync?

                      Direct connection between firewalls, 10.0.88.0 network, not used anywhere else.

                      K jimpJ 2 Replies Last reply Reply Quote 0
                      • K
                        kprovost @w0w
                        last edited by

                        @w0w Can you try turning up pf's debugging? (pfctl -x loud)

                        On a system that's not yet run out of states I suspect you're going to see "pfsync_state_import: unknown route interface: <if name>". That'd confirm my current theory.

                        1 Reply Last reply Reply Quote 1
                        • jimpJ
                          jimp Rebel Alliance Developer Netgate @w0w
                          last edited by

                          @w0w said in Growing states table, some leak?:

                          'pfsync Synchronize Peer IP', 'Synchronize Config to IP' are filled on the primary node with peer IP address, on the secondary 'pfsync Synchronize Peer IP' only filled.
                          'Set a custom Filter Host ID' is left blank, but I see those generated IDs on both nodes.

                          Maybe I'm reading this wrong but this should be matching on both systems for State Synchronization (pfsync). You should have it enabled on both and have both set with the address of the peer (or both blank) -- it's not like XMLRPC, state sync wants to work in both directions.

                          And related to what Kristof asked above, also check the output of ifconfig -l on both systems, it should (ideally) match so they all have the same interfaces in the OS. What he's noting is that it may be tossing an error if there is a state for a certain type of rule on an interface that does not exist on the peer node. (e.g. PPPoE WAN, maybe a VPN interface in certain cases, that sort of thing)

                          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                          Need help fast? Netgate Global Support!

                          Do not Chat/PM for help!

                          w0wW 1 Reply Last reply Reply Quote 0
                          • w0wW
                            w0w @jimp
                            last edited by

                            @jimp
                            No this not your reading, it was me writing early morning 😞.
                            State Synchronization enabled on both and have peer IP filled.

                            @jimp said in Growing states table, some leak?:

                            And related to what Kristof asked above, also check the output of ifconfig -l on both systems, it should (ideally) match so they all have the same interfaces in the OS. What he's noting is that it may be tossing an error if there is a state for a certain type of rule on an interface that does not exist on the peer node. (e.g. PPPoE WAN, maybe a VPN interface in certain cases, that sort of thing)

                            Yes, this can be the cause of the problem. WAN2 have different interfaces.

                            @kprovost said in Growing states table, some leak?:

                            Can you try turning up pf's debugging? (pfctl -x loud)

                            Will do that ASAP. Thanks.

                            1 Reply Last reply Reply Quote 0
                            • w0wW
                              w0w
                              last edited by

                              @jimp, @kprovost said in Growing states table, some leak?:

                              pfctl -x loud

                              pfsync_state_import: unknown route interface: igb0 
                              

                              Hmm… Yes. That's it.

                              K 1 Reply Last reply Reply Quote 0
                              • K
                                kprovost @w0w
                                last edited by

                                @w0w Cool. The fix is under review in https://reviews.freebsd.org/D41779 and should easily make 23.09.

                                The only way I see to work around this is to make sure that every interface used as a route-to target exists on all members of the HA setup. You could create dummy interfaces with ifconfig epair name <ifname> if that's difficult for some reason.

                                w0wW 1 Reply Last reply Reply Quote 1
                                • w0wW
                                  w0w @kprovost
                                  last edited by w0w

                                  @kprovost
                                  Thanks a lot!
                                  Not sure if ifconfig epair name <ifname> can be used on pfSense, I think I am going with VLANs instead...
                                  EDIT: No, VLANs can't be used, for the same reason :)

                                  jimpJ 1 Reply Last reply Reply Quote 0
                                  • jimpJ
                                    jimp Rebel Alliance Developer Netgate @w0w
                                    last edited by

                                    @w0w In the past, users have used a single interface lagg to work around similar problems. Then the interfaces end up as laggX.<vlan> on both systems.

                                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                    Need help fast? Netgate Global Support!

                                    Do not Chat/PM for help!

                                    w0wW 1 Reply Last reply Reply Quote 1
                                    • w0wW
                                      w0w @jimp
                                      last edited by

                                      @jimp said in Growing states table, some leak?:

                                      @w0w In the past, users have used a single interface lagg to work around similar problems. Then the interfaces end up as laggX.<vlan> on both systems.

                                      Thank you, sir! 😳
                                      This is definitely a good idea, I've reconfigured both firewalls to use LAGGs, will see…

                                      What about adding this workaround to https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html ?

                                      1 Reply Last reply Reply Quote 0
                                      • jimpJ
                                        jimp Rebel Alliance Developer Netgate
                                        last edited by

                                        We only claim to support systems with matching hardware and so on officially, so while it may work it's not something we want to document that much and imply it should be relied upon regularly.

                                        That said, I think it may be mentioned somewhere already in there since it used to be a problem with state matching and pfsync if the interfaces didn't line up in the OS. Different problem but similar workarounds.

                                        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                        Need help fast? Netgate Global Support!

                                        Do not Chat/PM for help!

                                        w0wW 1 Reply Last reply Reply Quote 1
                                        • w0wW
                                          w0w @jimp
                                          last edited by

                                          @jimp
                                          Ok, great.
                                          It's funny that this leak didn't make itself known earlier.

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.