Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    High CPU Usage and Delay Calling ifconfig -v ix0/ix1; Delays Dashboard Load

    Scheduled Pinned Locked Moved General pfSense Questions
    19 Posts 3 Posters 1.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • L
      ltctech
      last edited by

      pfSense Plus 22.05
      HP DL320e Gen8
      Intel X540-T2

      Seems that the Dashboard calls ifconfig -v for each interface. For whatever reason, it seems to take a while on ix0 and ix1, but not bge0 nor bge1. Calling ifconfig -v on ix1.[vlan] doesn't result in a delay either. It prints output in the terminal and then sits there for a second spinning the CPU. Calling ifconfig without a verbose flag is quick.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        So you can replicate that delay at the command line? On just one interface?

        How long is the delay? Try for example:

        [23.01-RC][root@6100.stevew.lan]/root: time ifconfig -v ix0
        ix0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                description: WAN3
                options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
                ether 00:08:a2:12:17:7e
                inet6 fe80::208:a2ff:fe12:177e%ix0 prefixlen 64 scopeid 0x5
                media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
                status: active
                nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                plugged: SFP/SFP+/SFP28 1X Copper Active (Copper pigtail)
                vendor: BROCADE PN: 58-1000027-01 SN: CBMB110100409D5 DATE: 2011-01-10
        0.000u 0.191s 0:00.19 100.0%    166+202k 0+0io 0pf+0w
        

        Steve

        L 3 Replies Last reply Reply Quote 0
        • L
          ltctech @stephenw10
          last edited by

          @stephenw10

          The card in question is a 10Gtek Intel X540-T2:
          https://www.amazon.com/dp/B01HMGWOU8

          Is the card at fault here?

          It prints out everything immediately but sits at the end for a second eating cycles:

          [22.05-RELEASE][root@redacted.redacted.com]/root: time ifconfig -v ix0
          ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
                  description: WAN
                  options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
                  ether 80:61:5f:XX:XX:XX
                  inet6 y:y:y:y:y:y:y:y prefixlen 64 scopeid 0x1
                  inet X.X.X.254 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.194 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.195 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.196 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.197 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.198 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.199 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.200 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.201 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.202 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.203 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.204 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.205 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.206 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.207 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.208 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.209 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.210 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.211 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.212 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.213 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.214 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.215 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.216 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.217 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.218 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.219 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.220 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.221 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.222 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.223 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.224 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.225 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.226 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.227 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.228 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.229 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.230 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.231 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.232 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.233 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.234 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.235 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.236 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.237 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.238 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.239 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.240 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.241 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.242 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.243 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.244 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.245 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.246 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.247 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.248 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.249 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.250 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.251 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.252 netmask 0xffffffc0 broadcast X.X.X.255
                  inet X.X.X.253 netmask 0xffffffc0 broadcast X.X.X.255
                  media: Ethernet autoselect (1000baseT <full-duplex>)
                  status: active
                  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          0.000u 1.368s 0:01.36 100.0%    148+209k 1+0io 0pf+0w
          
          [22.05-RELEASE][root@redacted.redacted.com]/root: time ifconfig -v ix1
          ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                  description: LAN
                  options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
                  ether 80:61:5f:XX:XX:XX
                  inet6 y:y:y:y:y:y:y:y prefixlen 64 scopeid 0x2
                  media: Ethernet autoselect (1000baseT <full-duplex>)
                  status: active
                  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          0.000u 1.367s 0:01.37 99.2%     149+210k 1+0io 0pf+0w
          
          [22.05-RELEASE][root@redacted.redacted.com]/root: time ifconfig -v bge0
          bge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
                  options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
                  ether a4:5d:36:XX:XX:XX
                  media: Ethernet autoselect
                  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
          0.000u 0.003s 0:00.00 0.0%      0+0k 1+0io 0pf+0w
          
          
          1 Reply Last reply Reply Quote 0
          • L
            ltctech @stephenw10
            last edited by

            @stephenw10

            Seems it gets stuck on:

            1.408296106 ioctl(3,SIOCGI2C,0x235240)           = 0 (0x0)
            

            I2C seems to be related to SFP detection.

            Could have the FreeBSD driver inherited a Linux bug:
            https://access.redhat.com/solutions/1274823

            The card in question only has two RJ-45 ports, no SFP.

            Here is tail end output of truss:

            truss -d ifconfig -v ix1
            
            0.038939836 ioctl(3,SIOCGIFXMEDIA,0x7fffffffe230) = 0 (0x0)
                    media: Ethernet autoselect (1000baseT <full-duplex>)
            0.039000609 write(1,"\tmedia: Ethernet autoselect (10"...,54) = 54 (0x36)
                    status: active
            0.039051235 write(1,"\tstatus: active\n",16)     = 16 (0x10)
            0.039093662 issetugid()                          = 0 (0x0)
            0.039150202 open("/etc/mac.conf",O_RDONLY|O_CLOEXEC,0666) = 4 (0x4)
            0.039196161 fstat(4,{ mode=-rw-r--r-- ,inode=65570,size=507,blksize=4096 }) = 0 (0x0)
            0.039264207 read(4,"#\n# $FreeBSD$\n#\n# TrustedBSD "...,4096) = 507 (0x1fb)
            0.039348365 mmap(0x0,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367033344 (0x8006f5000)
            0.039433003 read(4,0x8006e3000,4096)             = 0 (0x0)
            0.039505842 close(4)                             = 0 (0x0)
            0.039564397 mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367045632 (0x8006f8000)
            0.039659059 mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367053824 (0x8006fa000)
            0.039722913 mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367062016 (0x8006fc000)
            0.039776749 mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367070208 (0x8006fe000)
            0.039831632 mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367078400 (0x800700000)
            0.039889118 mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367086592 (0x800702000)
            0.039948891 mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367094784 (0x800704000)
            0.040000932 mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367102976 (0x800706000)
            0.040056116 mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367111168 (0x800708000)
            0.040114264 mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34367119360 (0x80070a000)
            0.040178666 ioctl(3,SIOCGIFMAC,0x7fffffffe260)   ERR#22 'Invalid argument'
            0.040228753 socket(PF_INET6,SOCK_DGRAM,0)        = 4 (0x4)
            0.040274436 ioctl(4,SIOCGIFINFO_IN6,0x7fffffffe200) = 0 (0x0)
            0.040328191 socket(PF_INET,SOCK_DGRAM|SOCK_CLOEXEC,0) = 5 (0x5)
            0.040384041 ioctl(5,SIOCGIFINDEX,0x7fffffffe1a0) = 0 (0x0)
            0.040431480 close(5)                             = 0 (0x0)
            0.040480004 ioctl(4,SIOCGDEFIFACE_IN6,0x7fffffffe250) = 0 (0x0)
            0.040526779 close(4)                             = 0 (0x0)
                    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
            0.040600918 write(1,"\tnd6 options=21<PERFORMNUD,AUTO"...,43) = 43 (0x2b)
            0.040681234 ioctl(3,SIOCGIFSTATUS,0x7fffffffe6a0) ERR#22 'Invalid argument'
            1.408296106 ioctl(3,SIOCGI2C,0x235240)           = 0 (0x0)
            1.408381260 close(3)                             = 0 (0x0)
            1.408462493 sigprocmask(SIG_BLOCK,{ SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0)
            1.408523846 sigprocmask(SIG_SETMASK,{ },0x0)     = 0 (0x0)
            1.408552732 sigprocmask(SIG_BLOCK,{ SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0)
            1.408617949 sigprocmask(SIG_SETMASK,{ },0x0)     = 0 (0x0)
            1.408660380 sigprocmask(SIG_BLOCK,{ SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0)
            1.408689782 sigprocmask(SIG_SETMASK,{ },0x0)     = 0 (0x0)
            1.408727500 sigprocmask(SIG_BLOCK,{ SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0)
            1.408794199 sigprocmask(SIG_SETMASK,{ },0x0)     = 0 (0x0)
            1.408869163 sigprocmask(SIG_BLOCK,{ SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0)
            1.408924575 sigprocmask(SIG_SETMASK,{ },0x0)     = 0 (0x0)
            1.408958432 sigprocmask(SIG_BLOCK,{ SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0)
            1.408983991 sigprocmask(SIG_SETMASK,{ },0x0)     = 0 (0x0)
            1.409022363 sigprocmask(SIG_BLOCK,{ SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2 },{ }) = 0 (0x0)
            1.409052587 sigprocmask(SIG_SETMASK,{ },0x0)     = 0 (0x0)
            1.409242793 exit(0x0)
            1.409262149 process exit, rval = 0
            
            1 Reply Last reply Reply Quote 0
            • L
              ltctech @stephenw10
              last edited by

              @stephenw10

              It's a driver bug...

              https://github.com/opnsense/core/issues/5349
              https://forum.opnsense.org/index.php?topic=25440.msg127981#msg127981
              https://github.com/opnsense/src/commit/938a0467e8cbf9e47629aff57e04250e7f6be251

              They fixed it a year ago in OPNsense. Is it fixed in pfSense Plus 23.01 and/or FreeBSD 14-Current?

              Seems that it still tries to detect it 10 times:
              https://github.com/freebsd/freebsd-src/blob/main/sys/dev/ixgbe/ixgbe_phy.h#L150
              https://github.com/freebsd/freebsd-src/blob/main/sys/dev/ixgbe/ixgbe_phy.c#L1996

              Here is why dashboard is slow:
              https://github.com/pfsense/pfsense/blob/master/src/etc/inc/pfsense-utils.inc#L1695
              https://github.com/pfsense/pfsense/search?q=get_interface_info

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                It looks like they never upstreamed that fix so, yeah, I don't see it FreeBSD or pfSense.

                However that NIC doesn't have SFP modules. I wouldn't expect it to apply there. I do see that here with NICs that are SFP though:

                [22.05.1-RELEASE][admin@8200-2.stevew.lan]/root: time ifconfig -vvvm ix0
                ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                	description: WAN3
                	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
                	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
                	ether 90:ec:77:47:5c:e6
                	inet6 fe80::92ec:77ff:fe47:5ce6%ix0 prefixlen 64 scopeid 0x5
                	media: Ethernet autoselect
                	status: no carrier
                	supported media:
                		media autoselect
                	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                0.000u 1.130s 0:01.13 100.0%	147+208k 1+0io 0pf+0w
                [22.05.1-RELEASE][admin@8200-2.stevew.lan]/root: time ifconfig -vvvm ix2
                ix2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                	description: WAN2
                	options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
                	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
                	ether 90:ec:77:47:5c:e5
                	inet6 fe80::92ec:77ff:fe47:5ce5%ix2 prefixlen 64 scopeid 0x7
                	media: Ethernet autoselect
                	status: no carrier
                	supported media:
                		media autoselect
                		media 10baseT/UTP
                		media 100baseTX
                		media 1000baseT
                	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                0.000u 0.413s 0:00.41 100.0%	149+212k 1+0io 0pf+0w
                

                Does that 10GTek NIC report that it's SFP capable?

                Steve

                L 1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  What delay do you actually see on the dashboard? Just 1s? 1s for each port?

                  L 1 Reply Last reply Reply Quote 0
                  • L
                    ltctech @stephenw10
                    last edited by

                    @stephenw10

                    How would I check that the card is reporting it's SFP capable?

                    The dashboard takes roughly 7 seconds to load/refresh. That's without the widgets that take longer. It also tends to keep a core busy as its constantly trying to update.

                    1 Reply Last reply Reply Quote 0
                    • L
                      ltctech @stephenw10
                      last edited by ltctech

                      Output of same command that you ran:

                      [22.05-RELEASE][root@redacted.redacted.com]/root: time ifconfig -vvvm ix1
                      ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                              description: LAN
                              options=8138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER>
                              capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
                              ether 80:61:5f:XX:XX:XX
                              inet6 [redacted] prefixlen 64 scopeid 0x2
                              media: Ethernet autoselect (1000baseT <full-duplex>)
                              status: active
                              supported media:
                                      media autoselect
                                      media 100baseTX
                                      media 1000baseT
                                      media 10Gbase-T
                              nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                      0.000u 1.365s 0:01.37 99.2%     145+205k 1+0io 0pf+0w
                      

                      This one shows supported media but nothing that I wouldn't expect for the card.

                      1 Reply Last reply Reply Quote 0
                      • L
                        ltctech
                        last edited by

                        @stephenw10

                        More info about card:

                        sysctl dev.ix.1
                        
                        
                        dev.ix.1.fw_version: eTrack 0x80000528 PHY FW V286
                        dev.ix.1.advertise_speed: 7
                        dev.ix.1.fc: 0
                        dev.ix.1.mac_stats.tx_frames_1024_1522: 1206525516
                        dev.ix.1.mac_stats.tx_frames_512_1023: 8917549
                        dev.ix.1.mac_stats.tx_frames_256_511: 8111570
                        dev.ix.1.mac_stats.tx_frames_128_255: 79421629
                        dev.ix.1.mac_stats.tx_frames_65_127: 65310030
                        dev.ix.1.mac_stats.tx_frames_64: 265654722
                        dev.ix.1.mac_stats.management_pkts_txd: 0
                        dev.ix.1.mac_stats.mcast_pkts_txd: 3
                        dev.ix.1.mac_stats.bcast_pkts_txd: 50373
                        dev.ix.1.mac_stats.good_pkts_txd: 1633941016
                        dev.ix.1.mac_stats.total_pkts_txd: 1633941016
                        dev.ix.1.mac_stats.good_octets_txd: 1813044533668
                        dev.ix.1.mac_stats.checksum_errs: 0
                        dev.ix.1.mac_stats.management_pkts_drpd: 0
                        dev.ix.1.mac_stats.management_pkts_rcvd: 0
                        dev.ix.1.mac_stats.recv_jabberd: 0
                        dev.ix.1.mac_stats.recv_oversized: 0
                        dev.ix.1.mac_stats.recv_fragmented: 0
                        dev.ix.1.mac_stats.recv_undersized: 0
                        dev.ix.1.mac_stats.rx_frames_1024_1522: 677354061
                        dev.ix.1.mac_stats.rx_frames_512_1023: 9601513
                        dev.ix.1.mac_stats.rx_frames_256_511: 10145596
                        dev.ix.1.mac_stats.rx_frames_128_255: 82799180
                        dev.ix.1.mac_stats.rx_frames_65_127: 820440758
                        dev.ix.1.mac_stats.rx_frames_64: 0
                        dev.ix.1.mac_stats.bcast_pkts_rcvd: 608287
                        dev.ix.1.mac_stats.mcast_pkts_rcvd: 3
                        dev.ix.1.mac_stats.good_pkts_rcvd: 1600341108
                        dev.ix.1.mac_stats.total_pkts_rcvd: 1600598184
                        dev.ix.1.mac_stats.good_octets_rcvd: 1110241698944
                        dev.ix.1.mac_stats.total_octets_rcvd: 1110277666432
                        dev.ix.1.mac_stats.xoff_recvd: 0
                        dev.ix.1.mac_stats.xoff_txd: 0
                        dev.ix.1.mac_stats.xon_recvd: 0
                        dev.ix.1.mac_stats.xon_txd: 0
                        dev.ix.1.mac_stats.rx_missed_packets: 0
                        dev.ix.1.mac_stats.rec_len_errs: 0
                        dev.ix.1.mac_stats.remote_faults: 0
                        dev.ix.1.mac_stats.local_faults: 7
                        dev.ix.1.mac_stats.short_discards: 0
                        dev.ix.1.mac_stats.byte_errs: 0
                        dev.ix.1.mac_stats.ill_errs: 0
                        dev.ix.1.mac_stats.crc_errs: 0
                        dev.ix.1.mac_stats.rx_errs: 0
                        dev.ix.1.queue3.rx_discarded: 0
                        dev.ix.1.queue3.rx_copies: 0
                        dev.ix.1.queue3.rx_bytes: 342057574713
                        dev.ix.1.queue3.rx_packets: 593135439
                        dev.ix.1.queue3.rxd_tail: 1870
                        dev.ix.1.queue3.rxd_head: 1871
                        dev.ix.1.queue3.irqs: 499456640
                        dev.ix.1.queue3.interrupt_rate: 31250
                        dev.ix.1.queue3.tx_packets: 0
                        dev.ix.1.queue3.tso_tx: 0
                        dev.ix.1.queue3.txd_tail: 0
                        dev.ix.1.queue3.txd_head: 0
                        dev.ix.1.queue2.rx_discarded: 0
                        dev.ix.1.queue2.rx_copies: 0
                        dev.ix.1.queue2.rx_bytes: 214014283409
                        dev.ix.1.queue2.rx_packets: 172466257
                        dev.ix.1.queue2.rxd_tail: 80
                        dev.ix.1.queue2.rxd_head: 81
                        dev.ix.1.queue2.irqs: 111247329
                        dev.ix.1.queue2.interrupt_rate: 31250
                        dev.ix.1.queue2.tx_packets: 0
                        dev.ix.1.queue2.tso_tx: 0
                        dev.ix.1.queue2.txd_tail: 0
                        dev.ix.1.queue2.txd_head: 0
                        dev.ix.1.queue1.rx_discarded: 0
                        dev.ix.1.queue1.rx_copies: 0
                        dev.ix.1.queue1.rx_bytes: 265381420690
                        dev.ix.1.queue1.rx_packets: 230347674
                        dev.ix.1.queue1.rxd_tail: 921
                        dev.ix.1.queue1.rxd_head: 922
                        dev.ix.1.queue1.irqs: 156052235
                        dev.ix.1.queue1.interrupt_rate: 31250
                        dev.ix.1.queue1.tx_packets: 0
                        dev.ix.1.queue1.tso_tx: 0
                        dev.ix.1.queue1.txd_tail: 0
                        dev.ix.1.queue1.txd_head: 0
                        dev.ix.1.queue0.rx_discarded: 0
                        dev.ix.1.queue0.rx_copies: 0
                        dev.ix.1.queue0.rx_bytes: 275985711464
                        dev.ix.1.queue0.rx_packets: 604391817
                        dev.ix.1.queue0.rxd_tail: 392
                        dev.ix.1.queue0.rxd_head: 393
                        dev.ix.1.queue0.irqs: 1182566786
                        dev.ix.1.queue0.interrupt_rate: 31250
                        dev.ix.1.queue0.tx_packets: 1633941113
                        dev.ix.1.queue0.tso_tx: 0
                        dev.ix.1.queue0.txd_tail: 655
                        dev.ix.1.queue0.txd_head: 655
                        dev.ix.1.link_irq: 2
                        dev.ix.1.watchdog_events: 0
                        dev.ix.1.dropped: 0
                        dev.ix.1.iflib.rxq3.rxq_fl0.buf_size: 2048
                        dev.ix.1.iflib.rxq3.rxq_fl0.credits: 2047
                        dev.ix.1.iflib.rxq3.rxq_fl0.cidx: 1871
                        dev.ix.1.iflib.rxq3.rxq_fl0.pidx: 1870
                        dev.ix.1.iflib.rxq3.cpu: 3
                        dev.ix.1.iflib.rxq2.rxq_fl0.buf_size: 2048
                        dev.ix.1.iflib.rxq2.rxq_fl0.credits: 2047
                        dev.ix.1.iflib.rxq2.rxq_fl0.cidx: 81
                        dev.ix.1.iflib.rxq2.rxq_fl0.pidx: 80
                        dev.ix.1.iflib.rxq2.cpu: 2
                        dev.ix.1.iflib.rxq1.rxq_fl0.buf_size: 2048
                        dev.ix.1.iflib.rxq1.rxq_fl0.credits: 2047
                        dev.ix.1.iflib.rxq1.rxq_fl0.cidx: 923
                        dev.ix.1.iflib.rxq1.rxq_fl0.pidx: 922
                        dev.ix.1.iflib.rxq1.cpu: 1
                        dev.ix.1.iflib.rxq0.rxq_fl0.buf_size: 2048
                        dev.ix.1.iflib.rxq0.rxq_fl0.credits: 2047
                        dev.ix.1.iflib.rxq0.rxq_fl0.cidx: 393
                        dev.ix.1.iflib.rxq0.rxq_fl0.pidx: 392
                        dev.ix.1.iflib.rxq0.cpu: 0
                        dev.ix.1.iflib.txq3.r_abdications: 0
                        dev.ix.1.iflib.txq3.r_restarts: 0
                        dev.ix.1.iflib.txq3.r_stalls: 0
                        dev.ix.1.iflib.txq3.r_starts: 0
                        dev.ix.1.iflib.txq3.r_drops: 0
                        dev.ix.1.iflib.txq3.r_enqueues: 0
                        dev.ix.1.iflib.txq3.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
                        dev.ix.1.iflib.txq3.txq_cleaned: 0
                        dev.ix.1.iflib.txq3.txq_processed: 0
                        dev.ix.1.iflib.txq3.txq_in_use: 0
                        dev.ix.1.iflib.txq3.txq_cidx_processed: 0
                        dev.ix.1.iflib.txq3.txq_cidx: 0
                        dev.ix.1.iflib.txq3.txq_pidx: 0
                        dev.ix.1.iflib.txq3.no_tx_dma_setup: 0
                        dev.ix.1.iflib.txq3.txd_encap_efbig: 0
                        dev.ix.1.iflib.txq3.tx_map_failed: 0
                        dev.ix.1.iflib.txq3.no_desc_avail: 0
                        dev.ix.1.iflib.txq3.mbuf_defrag_failed: 0
                        dev.ix.1.iflib.txq3.m_pullups: 0
                        dev.ix.1.iflib.txq3.mbuf_defrag: 0
                        dev.ix.1.iflib.txq3.cpu: 3
                        dev.ix.1.iflib.txq2.r_abdications: 0
                        dev.ix.1.iflib.txq2.r_restarts: 0
                        dev.ix.1.iflib.txq2.r_stalls: 0
                        dev.ix.1.iflib.txq2.r_starts: 0
                        dev.ix.1.iflib.txq2.r_drops: 0
                        dev.ix.1.iflib.txq2.r_enqueues: 0
                        dev.ix.1.iflib.txq2.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
                        dev.ix.1.iflib.txq2.txq_cleaned: 0
                        dev.ix.1.iflib.txq2.txq_processed: 0
                        dev.ix.1.iflib.txq2.txq_in_use: 0
                        dev.ix.1.iflib.txq2.txq_cidx_processed: 0
                        dev.ix.1.iflib.txq2.txq_cidx: 0
                        dev.ix.1.iflib.txq2.txq_pidx: 0
                        dev.ix.1.iflib.txq2.no_tx_dma_setup: 0
                        dev.ix.1.iflib.txq2.txd_encap_efbig: 0
                        dev.ix.1.iflib.txq2.tx_map_failed: 0
                        dev.ix.1.iflib.txq2.no_desc_avail: 0
                        dev.ix.1.iflib.txq2.mbuf_defrag_failed: 0
                        dev.ix.1.iflib.txq2.m_pullups: 0
                        dev.ix.1.iflib.txq2.mbuf_defrag: 0
                        dev.ix.1.iflib.txq2.cpu: 2
                        dev.ix.1.iflib.txq1.r_abdications: 0
                        dev.ix.1.iflib.txq1.r_restarts: 0
                        dev.ix.1.iflib.txq1.r_stalls: 0
                        dev.ix.1.iflib.txq1.r_starts: 0
                        dev.ix.1.iflib.txq1.r_drops: 0
                        dev.ix.1.iflib.txq1.r_enqueues: 0
                        dev.ix.1.iflib.txq1.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE
                        dev.ix.1.iflib.txq1.txq_cleaned: 0
                        dev.ix.1.iflib.txq1.txq_processed: 0
                        dev.ix.1.iflib.txq1.txq_in_use: 0
                        dev.ix.1.iflib.txq1.txq_cidx_processed: 0
                        dev.ix.1.iflib.txq1.txq_cidx: 0
                        dev.ix.1.iflib.txq1.txq_pidx: 0
                        dev.ix.1.iflib.txq1.no_tx_dma_setup: 0
                        dev.ix.1.iflib.txq1.txd_encap_efbig: 0
                        dev.ix.1.iflib.txq1.tx_map_failed: 0
                        dev.ix.1.iflib.txq1.no_desc_avail: 0
                        dev.ix.1.iflib.txq1.mbuf_defrag_failed: 0
                        dev.ix.1.iflib.txq1.m_pullups: 0
                        dev.ix.1.iflib.txq1.mbuf_defrag: 0
                        dev.ix.1.iflib.txq1.cpu: 1
                        dev.ix.1.iflib.txq0.r_abdications: 0
                        dev.ix.1.iflib.txq0.r_restarts: 0
                        dev.ix.1.iflib.txq0.r_stalls: 0
                        dev.ix.1.iflib.txq0.r_starts: 1633941118
                        dev.ix.1.iflib.txq0.r_drops: 0
                        dev.ix.1.iflib.txq0.r_enqueues: 1633941121
                        dev.ix.1.iflib.txq0.ring_state: pidx_head: 1665 pidx_tail: 1665 cidx: 1665 state: IDLE
                        dev.ix.1.iflib.txq0.txq_cleaned: 3267887733
                        dev.ix.1.iflib.txq0.txq_processed: 3267887765
                        dev.ix.1.iflib.txq0.txq_in_use: 34
                        dev.ix.1.iflib.txq0.txq_cidx_processed: 661
                        dev.ix.1.iflib.txq0.txq_cidx: 629
                        dev.ix.1.iflib.txq0.txq_pidx: 663
                        dev.ix.1.iflib.txq0.no_tx_dma_setup: 0
                        dev.ix.1.iflib.txq0.txd_encap_efbig: 0
                        dev.ix.1.iflib.txq0.tx_map_failed: 0
                        dev.ix.1.iflib.txq0.no_desc_avail: 0
                        dev.ix.1.iflib.txq0.mbuf_defrag_failed: 0
                        dev.ix.1.iflib.txq0.m_pullups: 50
                        dev.ix.1.iflib.txq0.mbuf_defrag: 0
                        dev.ix.1.iflib.txq0.cpu: 0
                        dev.ix.1.iflib.override_nrxds: 0
                        dev.ix.1.iflib.override_ntxds: 0
                        dev.ix.1.iflib.use_logical_cores: 0
                        dev.ix.1.iflib.separate_txrx: 0
                        dev.ix.1.iflib.core_offset: 0
                        dev.ix.1.iflib.tx_abdicate: 0
                        dev.ix.1.iflib.rx_budget: 0
                        dev.ix.1.iflib.disable_msix: 0
                        dev.ix.1.iflib.override_qs_enable: 0
                        dev.ix.1.iflib.override_nrxqs: 0
                        dev.ix.1.iflib.override_ntxqs: 0
                        dev.ix.1.iflib.driver_version: 4.0.1-k
                        dev.ix.1.%parent: pci3
                        dev.ix.1.%pnpinfo: vendor=0x8086 device=0x1528 subvendor=0x1dcf subdevice=0x031b class=0x020000
                        dev.ix.1.%location: slot=0 function=1 dbsf=pci0:8:0:1 handle=\_SB_.PCI0.IPT1.SLT1
                        dev.ix.1.%driver: ix
                        dev.ix.1.%desc: Intel(R) X540-AT2
                        
                        
                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Hmm, odd. I don't expect it to hit that on that NIC.
                          Looks like it's 1s per NIC then, which I guess is expected. Not 10x 1s which would be..painful.

                          I raised this internally though, I'll see what our developers think.

                          Steve

                          L 1 Reply Last reply Reply Quote 0
                          • L
                            ltctech @stephenw10
                            last edited by

                            @stephenw10

                            The firewall in question has TAC Pro support on it. Should we open a ticket for this?

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              You can do but TAC won't be able to tell you anything further until we hear back from the developers. If you do be sure to reference this thread.

                              I opened a bug to track this: https://redmine.pfsense.org/issues/13911

                              Steve

                              L 1 Reply Last reply Reply Quote 0
                              • L
                                ltctech @stephenw10
                                last edited by

                                @stephenw10

                                Thanks, will keep checking the bug tracker.

                                1 Reply Last reply Reply Quote 0
                                • L
                                  ltctech
                                  last edited by

                                  Looks like it may not be fixed anytime soon. We have moved from 22.05 to 23.01 and it's still happening. I compared the output of ifconfig with and without the verbose flag and found it to be the same for our NICs. Thus I made the following temporary patch to use with the System_Patches package. With this patch Dashboard is quick to load and no longer causes high CPU usage.

                                  --- a/src/etc/inc/pfsense-utils.inc
                                  +++ b/src/etc/inc/pfsense-utils.inc
                                  @@ -1790,7 +1790,7 @@
                                   	if ($ifinfo['status'] == "up" && $swifinfo == NULL) {
                                   		/* try to determine media with ifconfig */
                                   		$ifconfiginfo = [];
                                  -		exec("/sbin/ifconfig -v " . $ifinfo['if'], $ifconfiginfo);
                                  +		exec("/sbin/ifconfig " . $ifinfo['if'], $ifconfiginfo);
                                   		$wifconfiginfo = [];
                                   		if (is_interface_wireless($ifdescr)) {
                                   			exec("/sbin/ifconfig {$ifinfo['if']} list sta", $wifconfiginfo);
                                  

                                  Do you foresee any issues with this patch?

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    It likely wouldn't make any difference for a NIC that doesn't have a module inserted, which is where the problem exists.
                                    Without that though you won't get any module info in the status page etc. for an interface that is linked with SFP.

                                    Steve

                                    L 1 Reply Last reply Reply Quote 0
                                    • M
                                      mvikman
                                      last edited by

                                      I have and dual-RJ45 X540 card (ix0/ix1) that also shows same kind of delay running ifconfig -v command
                                      On the dual-SFP+ X710 card (ixl0/ixl1), the result prints instantly.

                                      X540 has cable plugged in the ix0 and X710 has an RJ45 SFP plugged&connected in the ixl0.
                                      Second port on both cards is unplugged.

                                      [23.01-RELEASE][admin@shodan.phobos.home.arpa]/root: time ifconfig -v ix0
                                      ix0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                                              description: WAN
                                              options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
                                              ether 48:df:37:68:54:08
                                              inet6 fe80::4adf:37ff:fe68:5408%ix0 prefixlen 64 scopeid 0x3
                                              inet [redacted] netmask 0xfffffc00 broadcast [redacted]
                                              media: Ethernet autoselect (1000baseT <full-duplex>)
                                              status: active
                                              nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                                      0.000u 1.314s 0:01.31 100.0%    170+206k 0+0io 0pf+0w
                                      [23.01-RELEASE][admin@shodan.phobos.home.arpa]/root: time ifconfig -v ix1
                                      ix1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
                                              options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
                                              ether 48:df:37:68:54:09
                                              media: Ethernet autoselect
                                              status: no carrier
                                              nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                                      0.000u 1.291s 0:01.29 100.0%    167+203k 0+0io 0pf+0w
                                      [23.01-RELEASE][admin@shodan.phobos.home.arpa]/root: time ifconfig -v ixl0
                                      ixl0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                                              options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
                                              ether 48:df:37:74:bb:40
                                              inet6 [redacted] prefixlen 64 scopeid 0x1
                                              media: Ethernet autoselect (1000baseT <full-duplex>)
                                              status: active
                                              nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                                              plugged: SFP/SFP+/SFP28 1000BASE-T (RJ45)
                                              vendor: DELTA PN: LCP-1250RJ3SR-HV SN: CN8BKK22X0 DATE: 2018-12-07
                                      0.000u 0.045s 0:00.04 100.0%    198+240k 0+0io 0pf+0w
                                      [23.01-RELEASE][admin@shodan.phobos.home.arpa]/root: time ifconfig -v ixl1
                                      ixl1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
                                              options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
                                              ether 48:df:37:74:bb:48
                                              media: Ethernet autoselect
                                              status: no carrier
                                              nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                                      0.000u 0.006s 0:00.00 0.0%      0+0k 0+0io 0pf+0w
                                      [23.01-RELEASE][admin@shodan.phobos.home.arpa]/root:
                                      

                                      pfSense Plus 24.11-RELEASE (amd64)
                                      Dell Optiplex 7040 SFF
                                      Core i5-6500, 8GB RAM, 2x 240GB SSD (ZFS Mirror)
                                      HPE 561T (X540-AT2), 2-port 10Gb RJ45
                                      HPE 562SFP+ (X710-DA2), 2-port 10Gb SFP+

                                      1 Reply Last reply Reply Quote 0
                                      • L
                                        ltctech @stephenw10
                                        last edited by

                                        @stephenw10

                                        We do not have a module capable card so it's not an issue at least for us.

                                        If the underlying delay and CPU usage with module detection in the driver cannot be fixed could Netgate add a system tunable to turn off module detection on the dashboard?

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Better to add comments to the bug so all developers can see them.

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.