Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfTop hangs my GUI in 2.5.2 RC

    Scheduled Pinned Locked Moved 2.5.2 Release Candidate Snapshots (Retired)
    35 Posts 4 Posters 7.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jimpJ
      jimp Rebel Alliance Developer Netgate
      last edited by

      I can't reproduce that here. The page loads and updates the content live as it always does.

      What is your configuration for the page set to when this happens?

      Does using pfTop from the console or ssh work?

      Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      T 1 Reply Last reply Reply Quote 0
      • T
        Trey @jimp
        last edited by

        @jimp

        When starting pftop by ssh it directly hangs without any output.

        running pftops:

        ps -ax | grep pftop
        11621 - L 11:49.66 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        13020 - L 13:16.32 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        17041 - L 12:59.58 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        39212 - L 14:19.03 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        39821 - R 11:33.73 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        40097 - L 15:23.33 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        40726 - L 15:38.67 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        40967 - L 13:05.67 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        68675 - L 7:31.30 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        71010 - L 13:15.83 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        79096 - L 11:04.39 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        85169 - L 11:26.45 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        87241 - L 13:15.88 /usr/local/sbin/pftop -b -o bytes -w 135 -v default 100
        29040 0 S+ 0:00.00 grep pftop
        77243 1- L+ 0:22.80 pftop
        10237 2- L+ 0:40.86 /usr/local/sbin/pftop

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          Curious. That isn't happening at all here.

          If I run the command at the CLI it dumps data and immediately exits.

          Are you certain your entire system (kernel and base OS) are running the correct and matching versions of 2.5.2-RC?

          : pkg info -x pfSense | grep 2.5.2
          pfSense-2.5.2.r.20210613.1712
          pfSense-base-2.5.2.r.20210613.1712
          pfSense-default-config-2.5.2.r.20210613.1712
          pfSense-kernel-pfSense-2.5.2.r.20210613.1712
          pfSense-rc-2.5.2.r.20210613.1712
          pfSense-repo-2.5.2.r.20210613.1712
          
          : uname -a
          FreeBSD pfsense.home.arpa 12.2-STABLE FreeBSD 12.2-STABLE RELENG_2_5_2-n226655-2ce2a4a2f71 pfSense  amd64
          
          : ls -l `which pftop`
          -r-xr-xr-x  1 root  wheel  192352 Jun 10 17:54 /usr/local/sbin/pftop
          
          : sha256 `which pftop`
          SHA256 (/usr/local/sbin/pftop) = 0c525409788804d8d578068cb79408cd4f209dc384ddf65a8865eefe481e2a52
          
          : ldd `which pftop`
          /usr/local/sbin/pftop:
          	libncurses.so.8 => /lib/libncurses.so.8 (0x800279000)
          	libc.so.7 => /lib/libc.so.7 (0x8002d2000)
          

          Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          T 1 Reply Last reply Reply Quote 0
          • T
            Trey @jimp
            last edited by

            @jimp

            All the same, see:

            pkg info -x pfSense | grep 2.5.2
            pfSense-2.5.2.r.20210613.1712
            pfSense-base-2.5.2.r.20210613.1712
            pfSense-default-config-2.5.2.r.20210613.1712
            pfSense-kernel-pfSense-2.5.2.r.20210613.1712
            pfSense-rc-2.5.2.r.20210613.1712
            pfSense-repo-2.5.2.r.20210613.1712

            uname -a
            FreeBSD pfsense2.... 12.2-STABLE FreeBSD 12.2-STABLE RELENG_2_5_2-n226655-2ce2a4a2f71 pfSense amd64

            ls -l which pftop
            -r-xr-xr-x 1 root wheel 192352 Jun 10 23:54 /usr/local/sbin/pftop

            sha256 which pftop
            SHA256 (/usr/local/sbin/pftop) = 0c525409788804d8d578068cb79408cd4f209dc384ddf65a8865eefe481e2a52

            ldd which pftop
            /usr/local/sbin/pftop:
            libncurses.so.8 => /lib/libncurses.so.8 (0x800279000)
            libc.so.7 => /lib/libc.so.7 (0x8002d2000)

            M 1 Reply Last reply Reply Quote 0
            • M
              maverick_slo @Trey
              last edited by

              Maybe connected:
              2b9d0ca1-e950-4f12-b1d7-fd1189895f5e-image.png

              It hangs my FW, when there are 4 proccesses like this...

              T M 2 Replies Last reply Reply Quote 0
              • T
                Trey @maverick_slo
                last edited by

                @maverick_slo

                Maybe:

                ps -ax | grep pfctl
                8530 - L 42:21.31 /sbin/pfctl -i ovpnc3 -Fs
                17907 - I 0:00.00 sh -c /sbin/pfctl -vvss | /usr/bin/grep creator | /usr/bin/cut -d" " -f7 | /usr/bin/sort -u
                18015 - R 337:49.40 /sbin/pfctl -vvss
                38322 - RN 365:07.87 /sbin/pfctl -ss
                61701 - I 0:00.00 sh -c /sbin/pfctl -vvss | /usr/bin/grep creator | /usr/bin/cut -d" " -f7 | /usr/bin/sort -u
                61787 - L 17:56.93 /sbin/pfctl -vvss
                82409 - I 0:00.00 sh -c /sbin/pfctl -vvss | /usr/bin/grep creator | /usr/bin/cut -d" " -f7 | /usr/bin/sort -u
                82711 - L 198:37.49 /sbin/pfctl -vvss
                84187 - RN 385:23.46 /sbin/pfctl -ss
                95032 - I 0:00.00 sh -c /sbin/pfctl -vvss | /usr/bin/grep creator | /usr/bin/cut -d" " -f7 | /usr/bin/sort -u
                95080 - R 281:20.27 /sbin/pfctl -vvss
                43823 0 S+ 0:00.00 grep pfctl

                T 1 Reply Last reply Reply Quote 0
                • T
                  Trey @Trey
                  last edited by Trey

                  Indeed pfctl and pftop are not very happy, when looking at cpu usage...

                  last pid: 46705;  load averages:  4.63,  5.26,  5.75                                                                                                                                                                 up 0+07:02:25  19:51:31
                  119 processes: 9 running, 106 sleeping, 1 zombie, 3 lock
                  CPU 0:  0.7% user,  0.0% nice, 63.3% system,  0.0% interrupt, 36.0% idle
                  CPU 1:  0.0% user,  0.0% nice, 87.4% system,  0.0% interrupt, 12.6% idle
                  CPU 2:  0.0% user,  0.0% nice, 51.1% system,  0.0% interrupt, 48.9% idle
                  CPU 3:  0.0% user,  0.0% nice, 74.8% system,  0.0% interrupt, 25.2% idle
                  CPU 4:  0.0% user,  0.0% nice, 79.7% system,  0.0% interrupt, 20.3% idle
                  CPU 5:  0.0% user,  0.0% nice, 63.9% system,  0.0% interrupt, 36.1% idle
                  CPU 6:  0.0% user,  0.0% nice, 39.5% system,  0.0% interrupt, 60.5% idle
                  CPU 7:  0.0% user,  0.0% nice, 51.9% system,  0.0% interrupt, 48.1% idle
                  Mem: 189M Active, 650M Inact, 4241M Wired, 165M Buf, 10G Free
                  Swap: 3656M Total, 3656M Free
                  
                    PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
                  18015 root          1  20    0   503M    61M CPU7     7 345:55  99.96% pfctl
                  38322 root          1  20   20   134M    18M CPU4     4 373:08  99.95% pfctl
                  84187 root          1  20   20   134M    18M CPU3     3 393:25  99.91% pfctl
                  95080 root          1  80    0   503M    61M RUN      5 289:22  99.90% pfctl
                  39212 root          1  20    0   112M    91M CPU6     6  18:23  62.80% pftop
                  39821 root          1  80    0   112M    92M *pf_id   4  15:57  38.04% pftop
                  30353 root          1  72    0   146M  3636K RUN      7   2:32   2.41% pftop
                  11621 root          1  20    0   113M    98M *pf_id   5  15:43   2.41% pftop
                  40726 root          1  75    0   112M  3264K CPU1     1  18:52   1.00% pftop
                  82711 root          1  81    0   503M    61M CPU0     0 199:33   0.95% pfctl
                  18221 root          1  20    0    17M  7284K select   4   1:21   0.37% openvpn
                  45830 root          1  20    0    13M  3740K CPU5     5   0:00   0.07% top
                  93616 unbound       8  20    0   232M    50M kqread   7   0:06   0.06% unbound
                  
                  
                  1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    Can you share your /tmp/rules.debug?

                    Do you have any packages installed?

                    Any large pf tables/aliases? If so, how large?

                    Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    M T 2 Replies Last reply Reply Quote 0
                    • M
                      maverick_slo @jimp
                      last edited by

                      @jimp
                      Packages
                      acme security 0.6.9_3 Automated Certificate Management Environment, for automated use of LetsEncrypt certificates.

                      Package Dependencies:
                      pecl-ssh2-1.3.1โ€ƒ socat-1.7.4.1_1โ€ƒ php74-7.4.20โ€ƒ php74-ftp-7.4.20โ€ƒ
                      Cron sysutils 0.3.7_5 The cron utility is used to manage commands on a schedule.
                      haproxy-devel net 0.62_3 The Reliable, High Performance TCP/HTTP(S) Load Balancer.
                      This package implements the TCP, HTTP and HTTPS balancing features from haproxy.
                      Supports ACLs for smart backend switching.

                      Package Dependencies:
                      haproxy-2.2.14โ€ƒ
                      iperf benchmarks 3.0.2_5 Iperf is a tool for testing network throughput, loss, and jitter.

                      Package Dependencies:
                      iperf3-3.10.1โ€ƒ
                      openvpn-client-export security 1.6_1 Allows a pre-configured OpenVPN Windows Client or Mac OS X's Viscosity configuration bundle to be exported directly from pfSense.

                      Package Dependencies:
                      openvpn-client-export-2.5.2โ€ƒ openvpn-2.5.2_2โ€ƒ zip-3.0_1โ€ƒ p7zip-16.02_3โ€ƒ
                      snort

                      No large tables at all.. largest has like 5 entries

                      1 Reply Last reply Reply Quote 0
                      • T
                        Trey @jimp
                        last edited by

                        @jimp

                        I can share files and screen, but not in this forum. To many IPs in there.

                        Packages:
                        FRR, FreeRadius, OpenVPN Client Export

                        Aliases:

                        Couple of our networks, like 12x 192.168.0.0/24 networks, couple of dns aliases,
                        did have bogus network on in the interface config, and disabled it now, but i think you have to reboot...

                        1 Reply Last reply Reply Quote 0
                        • jimpJ
                          jimp Rebel Alliance Developer Netgate
                          last edited by

                          Is this bare metal hardware or a VM?

                          How many CPUs/cores?

                          Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                          Need help fast? Netgate Global Support!

                          Do not Chat/PM for help!

                          T 1 Reply Last reply Reply Quote 0
                          • T
                            Trey @jimp
                            last edited by

                            @jimp

                            Bare metal SuperMicro:
                            Intel(R) Atom(TM) CPU C3758 @ 2.20GHz
                            8 CPUs: 1 package(s) x 8 core(s)
                            AES-NI CPU Crypto: Yes (active)
                            QAT Crypto: Yes (inactive)

                            1 Reply Last reply Reply Quote 0
                            • jimpJ
                              jimp Rebel Alliance Developer Netgate
                              last edited by jimp

                              I thought maybe https://redmine.pfsense.org/issues/10414 had popped back up, but I can't replicate it using the commands that used to trigger it before. It's worth trying on your hardware though to see what happens.

                              Do you have bogons enabled?

                              If so, try the following commands:

                              $ /etc/rc.update_bogons.sh 0
                              $ time pfctl -t bogonsv6 -T flush
                              $ time pfctl -t bogonsv6 -T add -f /etc/bogonsv6
                              $ time pfctl -t bogonsv6 -T add -f /etc/bogonsv6
                              

                              (yes, that last one is done twice)

                              You may have to reboot before doing that if it's already in the bad state it was in before.

                              Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                              Need help fast? Netgate Global Support!

                              Do not Chat/PM for help!

                              T 1 Reply Last reply Reply Quote 0
                              • T
                                Trey @jimp
                                last edited by Trey

                                @jimp

                                time pfctl -t bogonsv6 -T flush
                                122609 addresses deleted.
                                0.000u 0.048s 0:00.04 100.0%    240+210k 0+0io 0pf+0w
                                
                                 time pfctl -t bogonsv6 -T add -f /etc/bogonsv6
                                122609/122609 addresses added.
                                0.226u 0.218s 0:00.44 97.7%     212+185k 0+0io 0pf+0w
                                
                                 time pfctl -t bogonsv6 -T add -f /etc/bogonsv6
                                0/122609 addresses added.
                                0.231u 0.079s 0:00.31 96.7%     208+182k 0+0io 0pf+0w
                                
                                

                                PS: It was still in this bad state but the commands still worked...

                                M 1 Reply Last reply Reply Quote 0
                                • M
                                  maverick_slo @Trey
                                  last edited by

                                  I had similar result.

                                  Hyperv, 4 cores.

                                  Before the upgrade not a single issue wigh beta...

                                  1 Reply Last reply Reply Quote 0
                                  • jimpJ
                                    jimp Rebel Alliance Developer Netgate
                                    last edited by

                                    OK, so at least we know it isn't the same bug coming back to haunt us, though that would have likely made tracking down the cause and resolution much easier.

                                    From your earlier output, the pfctl commands getting hung up are dumping the state table contents.

                                    Try each of the following, but no need to post the individual state output, just the timing:

                                    $ pfctl -si | grep -A4 State
                                    $ time pfctl -ss
                                    $ time pfctl -vvss
                                    

                                    Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                    Need help fast? Netgate Global Support!

                                    Do not Chat/PM for help!

                                    M T 2 Replies Last reply Reply Quote 0
                                    • M
                                      maverick_slo @jimp
                                      last edited by

                                      State Table Total Rate
                                      current entries 1434
                                      searches 3714325 359.4/s
                                      inserts 138111 13.4/s
                                      removals 136676 13.2/s

                                      4.50 real 0.01 user 4.48 sys

                                      1.32 real 0.01 user 1.30 sys

                                      1 Reply Last reply Reply Quote 0
                                      • T
                                        Trey @jimp
                                        last edited by

                                        Okay, after a reboot the pftop page works again.

                                        CPU usage was back to none. Then a pfctl -ss was running long....

                                        After it ended I run:

                                        pfctl -si | grep -A4 State
                                        State Table                          Total             Rate
                                          current entries                     2407
                                          searches                          404293          546.3/s
                                          inserts                            20698           28.0/s
                                          removals                           18291           24.7/s
                                        
                                        time pfctl -ss
                                        0.141u 140.191s 2:20.52 99.8%   203+177k 0+0io 0pf+0w
                                        
                                        time pfctl -vvss
                                        0.157u 87.638s 1:28.08 99.6%    203+177k 0+0io 0pf+0w
                                        
                                        
                                        

                                        And perhaps these aliases add a lot of ipv4 and ipv6:

                                        Jun 15 20:21:31 filterdns 68338 Adding Action: pf table: Office365Server host: outlook.office365.com
                                        Jun 15 20:21:31 filterdns 68338 Adding Action: pf table: Office365Server host: outlook.office.com
                                        Jun 15 20:21:31 filterdns 68338 Adding Action: pf table: MailExternalServerIPs host: outlook.office365.com
                                        Jun 15 20:21:31 filterdns 68338 Adding Action: pf table: IcingaExternClients host: outlook.office365.com

                                        1 Reply Last reply Reply Quote 0
                                        • jimpJ
                                          jimp Rebel Alliance Developer Netgate
                                          last edited by

                                          Hmm, it definitely shouldn't be taking that long to print out only 2400 states.

                                          Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                          Need help fast? Netgate Global Support!

                                          Do not Chat/PM for help!

                                          T 1 Reply Last reply Reply Quote 0
                                          • T
                                            Trey @jimp
                                            last edited by

                                            Block bogon networks was now on before reboot and is still...

                                            pfctl shows up in top every now and then and uses a lot of cpu...

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.