Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Reporting Lock Order Reversal (LOR) Backtraces

    CE 2.8.0 Development Snapshots
    9
    10
    2.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jimpJ
      jimp Rebel Alliance Developer Netgate
      last edited by

      Lock Order Reversals are non-fatal errors that indicate a potential problem in OS code. They are only visible because we have specific debugging options enabled on current development snapshot kernels. These options are disabled before release. Though they are not fatal and are not typically a cause for concern, they are still worth reporting. FreeBSD developers on our staff can look them over and either fix them or report them upstream. In many cases these are already known issues in FreeBSD that have not yet been fixed.

      If you encounter one, please post the entire LOR backtrace from the console output or system message buffer. If possible, please include some detail about when the LOR happened (e.g. when configuring a CARP VIP, during a CARP transition, etc).

      Because these are non-fatal and relatively common, there is no need to make separate threads for them. Feel free to add LORs here in this thread.

      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      1 Reply Last reply Reply Quote 1
      • jimpJ jimp pinned this topic on
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by stephenw10

        In a VM. I was doing nothing I was aware of at the time. Only noticed it later.

        Sep 22 13:00:53 	kernel 		lock order reversal:
        Sep 22 13:00:53 	kernel 		1st 0xffffffff8364f118 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/bpf.c:1518
        Sep 22 13:00:53 	kernel 		2nd 0xfffff80005d4fd80 iflib ctx lock (iflib ctx lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/iflib.c:4373
        Sep 22 13:00:53 	kernel 		lock order iflib ctx lock -> bpf global lock established at:
        Sep 22 13:00:53 	kernel 		#0 0xffffffff80de8a0d at witness_checkorder+0x32d
        Sep 22 13:00:53 	kernel 		#1 0xffffffff80d83043 at _sx_xlock+0x63
        Sep 22 13:00:53 	kernel 		#2 0xffffffff80ea2158 at bpfattach2+0xd8
        Sep 22 13:00:53 	kernel 		#3 0xffffffff80ec063b at ether_ifattach+0xeb
        Sep 22 13:00:53 	kernel 		#4 0xffffffff80ed5dbb at iflib_device_register+0xceb
        Sep 22 13:00:53 	kernel 		#5 0xffffffff80eda4d5 at iflib_device_attach+0xb5
        Sep 22 13:00:53 	kernel 		#6 0xffffffff80db4701 at device_attach+0x3c1
        Sep 22 13:00:53 	kernel 		#7 0xffffffff80db42b0 at device_probe_and_attach+0x70
        Sep 22 13:00:53 	kernel 		#8 0xffffffff80db58f8 at bus_generic_attach+0x18
        Sep 22 13:00:53 	kernel 		#9 0xffffffff8093f2c0 at pci_attach+0xe0
        Sep 22 13:00:53 	kernel 		#10 0xffffffff8120ea49 at acpi_pci_attach+0x19
        Sep 22 13:00:53 	kernel 		#11 0xffffffff80db4701 at device_attach+0x3c1
        Sep 22 13:00:53 	kernel 		#12 0xffffffff80db42b0 at device_probe_and_attach+0x70
        Sep 22 13:00:53 	kernel 		#13 0xffffffff80db58f8 at bus_generic_attach+0x18
        Sep 22 13:00:53 	kernel 		#14 0xffffffff81211ced at acpi_pcib_acpi_attach+0x42d
        Sep 22 13:00:53 	kernel 		#15 0xffffffff80db4701 at device_attach+0x3c1
        Sep 22 13:00:53 	kernel 		#16 0xffffffff80db42b0 at device_probe_and_attach+0x70
        Sep 22 13:00:53 	kernel 		#17 0xffffffff80db58f8 at bus_generic_attach+0x18
        Sep 22 13:00:53 	kernel 		lock order bpf global lock -> iflib ctx lock attempted at:
        Sep 22 13:00:53 	kernel 		#0 0xffffffff80de92dd at witness_checkorder+0xbfd
        Sep 22 13:00:53 	kernel 		#1 0xffffffff80d83043 at _sx_xlock+0x63
        Sep 22 13:00:53 	kernel 		#2 0xffffffff80ee1c8f at iflib_if_ioctl+0x2df
        Sep 22 13:00:53 	kernel 		#3 0xffffffff80eb527e at if_setflag+0xde
        Sep 22 13:00:53 	kernel 		#4 0xffffffff80eb514a at ifpromisc+0x2a
        Sep 22 13:00:53 	kernel 		#5 0xffffffff80ea46bd at bpfioctl+0xe9d
        Sep 22 13:00:53 	kernel 		#6 0xffffffff80bea23d at devfs_ioctl+0xcd
        Sep 22 13:00:53 	kernel 		#7 0xffffffff80e7dee1 at vn_ioctl+0x131
        Sep 22 13:00:53 	kernel 		#8 0xffffffff80bea8ee at devfs_ioctl_f+0x1e
        Sep 22 13:00:53 	kernel 		#9 0xffffffff80deef02 at kern_ioctl+0x202
        Sep 22 13:00:53 	kernel 		#10 0xffffffff80deec4a at sys_ioctl+0x12a
        Sep 22 13:00:53 	kernel 		#11 0xffffffff8133540e at amd64_syscall+0x12e
        Sep 22 13:00:53 	kernel 		#12 0xffffffff8130821b at fast_syscall_common+0xf8 
        

        Looks identical to this though: https://redmine.netgate.com/issues/8868
        That was on a 4100.

        1 Reply Last reply Reply Quote 0
        • A
          Anaerin
          last edited by

          On startup, I got the following:

          Oct 13 16:11:45 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
          Oct 13 16:11:46 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
          Oct 13 16:11:46 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
          Oct 13 16:11:46 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
          Oct 13 16:11:46 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
          Oct 13 16:11:46 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
          Oct 13 16:11:46 router vnstatd[74726]: Error: pidfile "/var/run/vnstat/vnstat.pid" lock failed (Resource temporarily unavailable), exiting.
          Oct 13 16:11:46 router kernel: gif0: promiscuous mode enabled
          Oct 13 16:11:46 router kernel: lock order reversal:
          Oct 13 16:11:46 router kernel:  1st 0xffffffff8365b698 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sy
          s/net/bpf.c:1518
          Oct 13 16:11:46 router kernel:  2nd 0xffffffff8365ca10 vlan_sx (vlan_sx, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/if_vlan.c:
          2144
          Oct 13 16:11:46 router kernel: lock order bpf global lock -> vlan_sx attempted at:
          Oct 13 16:11:46 router kernel: #0 0xffffffff80deb68d at witness_checkorder+0xbfd
          Oct 13 16:11:46 router kernel: #1 0xffffffff80d84783 at _sx_xlock+0x63
          Oct 13 16:11:46 router kernel: #2 0xffffffff80ed6113 at vlan_ioctl+0x203
          Oct 13 16:11:46 router kernel: #3 0xffffffff80eb83ae at if_setflag+0xde
          Oct 13 16:11:46 router kernel: #4 0xffffffff80eb827a at ifpromisc+0x2a
          Oct 13 16:11:46 router kernel: #5 0xffffffff80ea77ad at bpfioctl+0xe9d
          Oct 13 16:11:46 router kernel: #6 0xffffffff80beb6bd at devfs_ioctl+0xcd
          Oct 13 16:11:46 router kernel: #7 0xffffffff80e80d21 at vn_ioctl+0x131
          Oct 13 16:11:46 router kernel: #8 0xffffffff80bebd6e at devfs_ioctl_f+0x1e
          Oct 13 16:11:46 router kernel: #9 0xffffffff80df12b2 at kern_ioctl+0x202
          Oct 13 16:11:46 router kernel: #10 0xffffffff80df0ffa at sys_ioctl+0x12a
          Oct 13 16:11:46 router kernel: #11 0xffffffff8133a3be at amd64_syscall+0x12e
          Oct 13 16:11:46 router kernel: #12 0xffffffff8130dc8b at fast_syscall_common+0xf8
          Oct 13 16:11:46 router kernel: em0: promiscuous mode enabled
          Oct 13 16:11:46 router kernel: em0.3000: promiscuous mode enabled
          Oct 13 22:11:46 router softflowd[74739]: softflowd v1.0.0 starting data collection
          Oct 13 22:11:46 router softflowd[74739]: Exporting flows to [10.2.0.22]:2055
          
          1 Reply Last reply Reply Quote 0
          • bmeeksB bmeeks referenced this topic on
          • bmeeksB bmeeks referenced this topic on
          • L
            LandRocket
            last edited by marcosm

            Got one for you:

            Dec 11 10:39:09 router kernel: lock order reversal:
            Dec 11 10:39:09 router kernel:  1st 0xffffffff8365ae98 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-s
            rc-devel-main/sys/net/bpf.c:1518
            Dec 11 10:39:09 router kernel:  2nd 0xfffff800a2448628 if_lagg sx (if_lagg sx, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-m
            ain/sys/net/if_lagg.c:1669
            Dec 11 10:39:09 router kernel: lock order if_lagg sx -> bpf global lock established at:
            Dec 11 10:39:09 router kernel: #0 0xffffffff80debf1d at witness_checkorder+0x32d
            Dec 11 10:39:09 router kernel: #1 0xffffffff80d855c3 at _sx_xlock+0x63
            Dec 11 10:39:09 router kernel: #2 0xffffffff80ea66d8 at bpfattach2+0xd8
            Dec 11 10:39:09 router kernel: #3 0xffffffff80ec50eb at ether_ifattach+0xeb
            Dec 11 10:39:09 router kernel: #4 0xffffffff80eca37a at lagg_clone_create+0x29a
            Dec 11 10:39:09 router kernel: #5 0xffffffff80ec20ac at if_clone_createif+0x17c
            Dec 11 10:39:09 router kernel: #6 0xffffffff80ec1efa at ifc_create_ifp+0xea
            Dec 11 10:39:09 router kernel: #7 0xffffffff80ec21d4 at if_clone_create+0x34
            Dec 11 10:39:09 router kernel: #8 0xffffffff80eb907b at ifioctl+0x92b
            Dec 11 10:39:09 router kernel: #9 0xffffffff80df2412 at kern_ioctl+0x202
            Dec 11 10:39:09 router kernel: #10 0xffffffff80df215a at sys_ioctl+0x12a
            Dec 11 10:39:09 router kernel: #11 0xffffffff813333be at amd64_syscall+0x12e
            Dec 11 10:39:09 router kernel: #12 0xffffffff81306b8b at fast_syscall_common+0xf8
            Dec 11 10:39:09 router kernel: lock order bpf global lock -> if_lagg sx attempted at:
            Dec 11 10:39:09 router kernel: #0 0xffffffff80dec7ed at witness_checkorder+0xbfd
            Dec 11 10:39:09 router kernel: #1 0xffffffff80d855c3 at _sx_xlock+0x63
            Dec 11 10:39:09 router kernel: #2 0xffffffff80ecb382 at lagg_ioctl+0x742
            Dec 11 10:39:09 router kernel: #3 0xffffffff80eb983e at if_setflag+0xde
            Dec 11 10:39:09 router kernel: #4 0xffffffff80eb970a at ifpromisc+0x2a
            Dec 11 10:39:09 router kernel: #5 0xffffffff80ea8c3d at bpfioctl+0xe9d
            Dec 11 10:39:09 router kernel: #6 0xffffffff80bebd1d at devfs_ioctl+0xcd
            Dec 11 10:39:09 router kernel: #7 0xffffffff80e820d1 at vn_ioctl+0x131
            Dec 11 10:39:09 router kernel: #8 0xffffffff80bec3ce at devfs_ioctl_f+0x1e
            Dec 11 10:39:09 router kernel: #9 0xffffffff80df2412 at kern_ioctl+0x202
            Dec 11 10:39:09 router kernel: #10 0xffffffff80df215a at sys_ioctl+0x12a
            Dec 11 10:39:09 router kernel: #11 0xffffffff813333be at amd64_syscall+0x12e
            Dec 11 10:39:09 router kernel: #12 0xffffffff81306b8b at fast_syscall_common+0xf8
            Dec 11 10:39:09 router kernel: igb0: promiscuous mode enabled
            Dec 11 10:39:09 router kernel: igb1: promiscuous mode enabled
            Dec 11 10:39:09 router kernel: lagg0: promiscuous mode enabled
            

            this usually happens shortly after bootup, and as it suggests, I have 2 interfaces (igb0,igb1) configured as a LAN LACP (lagg0)

            It's a 4-port card (igb0-igb3):

            Dec 11 10:38:41 router kernel: igb0: <Intel(R) PRO/1000 ET 82576 (Quad Copper)> port 0xd020-0xd03f mem 0xf7c20000-0xf7c3ffff,0xf7800000-0xf7bfffff,0xf7c44000-0xf7c47fff
            irq 18 at device 0.0 on pci3

            Not sure if it's related but other messages that are in the log that may be relevant:
            Dec 11 10:38:41 router kernel: debugnet_any_ifnet_update: Bad dn_init result from igb0 (ifp 0xfffff80001ce8800), ignoring.

            omitted duplicate message pertaining to igb1

            The message doesn't appear for igb2 (currently assigned to WAN1). igb3 not currently in use

            R 1 Reply Last reply Reply Quote 1
            • R
              renesdk @LandRocket
              last edited by

              I get the almost the same as LandRocket
              I'm running Intel Ethernet Controller I226-V (4 ports).

              The only difference is mine is missing:
              Dec 11 10:39:09 router kernel: igb0: promiscuous mode enabled
              Dec 11 10:39:09 router kernel: igb1: promiscuous mode enabled
              Dec 11 10:39:09 router kernel: lagg0: promiscuous mode enabled

              1 Reply Last reply Reply Quote 0
              • M
                marcosm Netgate
                last edited by

                Logs reversed:

                Jan 2 17:55:03 	kernel 		#17 0xffffffff80ef020f at netisr_dispatch_src+0xaf
                Jan 2 17:55:03 	kernel 		#16 0xffffffff80ecc0d6 at ether_nh_input+0x3f6
                Jan 2 17:55:03 	kernel 		#15 0xffffffff80eca94b at ether_demux+0x6b
                Jan 2 17:55:03 	kernel 		#14 0xffffffff80ef3ca7 at pfil_run_hooks+0xb7
                Jan 2 17:55:03 	kernel 		#13 0xffffffff8106f3b5 at pf_eth_check_in+0x25
                Jan 2 17:55:03 	kernel 		#12 0xffffffff81049e75 at pf_test_eth+0x14a5
                Jan 2 17:55:03 	kernel 		#11 0xffffffff84c645ba at dummynet_io+0x41a
                Jan 2 17:55:03 	kernel 		#10 0xffffffff84c640bd at dummynet_send+0x17d
                Jan 2 17:55:03 	kernel 		#9 0xffffffff80ecaa5c at ether_demux+0x17c
                Jan 2 17:55:03 	kernel 		#8 0xffffffff80ef0380 at netisr_dispatch_src+0x220
                Jan 2 17:55:03 	kernel 		#7 0xffffffff80f99b28 at ip_input+0x298
                Jan 2 17:55:03 	kernel 		#6 0xffffffff80fb1acb at tcp_input+0xb
                Jan 2 17:55:03 	kernel 		#5 0xffffffff80fb0585 at tcp_input_with_port+0x5e5
                Jan 2 17:55:03 	kernel 		#4 0xffffffff80f93bc4 at in_pcblookup_mbuf+0x24
                Jan 2 17:55:03 	kernel 		#3 0xffffffff80f93ab3 at in_pcblookup_hash+0x83
                Jan 2 17:55:03 	kernel 		#2 0xffffffff80f927d6 at inp_smr_lock+0x136
                Jan 2 17:55:03 	kernel 		#1 0xffffffff80d7a597 at _rw_wlock_cookie+0x67
                Jan 2 17:55:03 	kernel 		#0 0xffffffff80df23cd at witness_checkorder+0xbfd
                Jan 2 17:55:03 	kernel 		lock order pf rulesets -> tcpinp attempted at:
                Jan 2 17:55:03 	kernel 		#13 0xffffffff8130c72b at fast_syscall_common+0xf8
                Jan 2 17:55:03 	kernel 		#12 0xffffffff813393be at amd64_syscall+0x12e
                Jan 2 17:55:03 	kernel 		#11 0xffffffff80e34485 at sys_connect+0x75
                Jan 2 17:55:03 	kernel 		#10 0xffffffff80e34574 at kern_connectat+0xc4
                Jan 2 17:55:03 	kernel 		#9 0xffffffff80e2d84f at soconnectat+0xaf
                Jan 2 17:55:03 	kernel 		#8 0xffffffff80fd443a at tcp_usr_connect+0x13a
                Jan 2 17:55:03 	kernel 		#7 0xffffffff80fd8001 at tcp_output+0x31
                Jan 2 17:55:03 	kernel 		#6 0xffffffff80fbe3c4 at tcp_default_output+0x2094
                Jan 2 17:55:03 	kernel 		#5 0xffffffff80fa3216 at ip_output+0xd26
                Jan 2 17:55:03 	kernel 		#4 0xffffffff80ef3ca7 at pfil_run_hooks+0xb7
                Jan 2 17:55:03 	kernel 		#3 0xffffffff8106f12f at pf_check_out+0x1f
                Jan 2 17:55:03 	kernel 		#2 0xffffffff8104a175 at pf_test+0xc5
                Jan 2 17:55:03 	kernel 		#1 0xffffffff80d79150 at _rm_rlock_debug+0x130
                Jan 2 17:55:03 	kernel 		#0 0xffffffff80df1afd at witness_checkorder+0x32d
                Jan 2 17:55:03 	kernel 		lock order tcpinp -> pf rulesets established at:
                Jan 2 17:55:03 	kernel 		2nd 0xfffff801b3b49a90 tcpinp (tcpinp, rw) @ /var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/sys/netinet/in_pcb.c:1554
                Jan 2 17:55:03 	kernel 		1st 0xffffffff836726d8 pf rulesets (pf rulesets, rm) @ /var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/sys/netpfil/pf/pf.c:7130
                Jan 2 17:55:03 	kernel 		lock order reversal: 
                
                1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks
                  last edited by bmeeks

                  Lock order reversal logged by Suricata when starting up using netmap Inline IPS Mode. This appears to be a problem within the iflib library when netmap operation is enabled. This is captured from a pfSense 2.7.0 CE snapshot current as of January 6, 2023. The system log setting is configured to display the most recent events first, so the order of the logged messages is reversed.

                  Jan 6 11:07:51	kernel		#12 0xffffffff81306b8b at fast_syscall_common+0xf8
                  Jan 6 11:07:51	kernel		#11 0xffffffff813333be at amd64_syscall+0x12e
                  Jan 6 11:07:51	kernel		#10 0xffffffff80df215a at sys_ioctl+0x12a
                  Jan 6 11:07:51	kernel		#9 0xffffffff80df2412 at kern_ioctl+0x202
                  Jan 6 11:07:51	kernel		#8 0xffffffff80bec3ce at devfs_ioctl_f+0x1e
                  Jan 6 11:07:51	kernel		#7 0xffffffff80e820d1 at vn_ioctl+0x131
                  Jan 6 11:07:51	kernel		#6 0xffffffff80bebd1d at devfs_ioctl+0xcd
                  Jan 6 11:07:51	kernel		#5 0xffffffff80ea8c3d at bpfioctl+0xe9d
                  Jan 6 11:07:51	kernel		#4 0xffffffff80eb970a at ifpromisc+0x2a
                  Jan 6 11:07:51	kernel		#3 0xffffffff80eb983e at if_setflag+0xde
                  Jan 6 11:07:51	kernel		#2 0xffffffff80ee687f at iflib_if_ioctl+0x2df
                  Jan 6 11:07:51	kernel		#1 0xffffffff80d855c3 at _sx_xlock+0x63
                  Jan 6 11:07:51	kernel		#0 0xffffffff80dec7ed at witness_checkorder+0xbfd
                  Jan 6 11:07:51	kernel		lock order bpf global lock -> iflib ctx lock attempted at:
                  Jan 6 11:07:51	kernel		#17 0xffffffff80db8dd8 at bus_generic_attach+0x18
                  Jan 6 11:07:51	kernel		#16 0xffffffff80db7790 at device_probe_and_attach+0x70
                  Jan 6 11:07:51	kernel		#15 0xffffffff80db7be1 at device_attach+0x3c1
                  Jan 6 11:07:51	kernel		#14 0xffffffff81210be0 at acpi_pcib_pci_attach+0xa0
                  Jan 6 11:07:51	kernel		#13 0xffffffff80db8dd8 at bus_generic_attach+0x18
                  Jan 6 11:07:51	kernel		#12 0xffffffff80db7790 at device_probe_and_attach+0x70
                  Jan 6 11:07:51	kernel		#11 0xffffffff80db7be1 at device_attach+0x3c1
                  Jan 6 11:07:51	kernel		#10 0xffffffff8120cea9 at acpi_pci_attach+0x19
                  Jan 6 11:07:51	kernel		#9 0xffffffff80940670 at pci_attach+0xe0
                  Jan 6 11:07:51	kernel		#8 0xffffffff80db8dd8 at bus_generic_attach+0x18
                  Jan 6 11:07:51	kernel		#7 0xffffffff80db7790 at device_probe_and_attach+0x70
                  Jan 6 11:07:51	kernel		#6 0xffffffff80db7be1 at device_attach+0x3c1
                  Jan 6 11:07:51	kernel		#5 0xffffffff80edf0b5 at iflib_device_attach+0xb5
                  Jan 6 11:07:51	kernel		#4 0xffffffff80eda99b at iflib_device_register+0xceb
                  Jan 6 11:07:51	kernel		#3 0xffffffff80ec50eb at ether_ifattach+0xeb
                  Jan 6 11:07:51	kernel		#2 0xffffffff80ea66d8 at bpfattach2+0xd8
                  Jan 6 11:07:51	kernel		#1 0xffffffff80d855c3 at _sx_xlock+0x63
                  Jan 6 11:07:51	kernel		#0 0xffffffff80debf1d at witness_checkorder+0x32d
                  Jan 6 11:07:51	kernel		lock order iflib ctx lock -> bpf global lock established at:
                  Jan 6 11:07:51	kernel		2nd 0xfffff80005c80d80 iflib ctx lock (iflib ctx lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/iflib.c:4523
                  Jan 6 11:07:51	kernel		1st 0xffffffff8365ae98 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/bpf.c:1518
                  Jan 6 11:07:51	kernel		lock order reversal:
                  Jan 6 11:07:48	kernel		em0: permanently promiscuous mode enabled
                  Jan 6 11:07:48	kernel		268.102300 [ 851] iflib_netmap_config txr 1 rxr 1 txd 1024 rxd 1024 rbufsz 2048
                  Jan 6 11:07:48	kernel		268.102281 [ 851] iflib_netmap_config txr 1 rxr 1 txd 1024 rxd 1024 rbufsz 2048
                  Jan 6 11:07:48	kernel		268.102248 [ 851] iflib_netmap_config txr 1 rxr 1 txd 1024 rxd 1024 rbufsz 2048
                  
                  
                  1 Reply Last reply Reply Quote 0
                  • B
                    bud4life
                    last edited by bud4life

                    I have interfaces locking up having to reboot, ever since I upgraded 2 weeks ago

                    Jan 14 14:23:57	kernel		lagg0.401: promiscuous mode enabled
                    Jan 14 14:23:57	kernel		lagg0.500: promiscuous mode enabled
                    Jan 14 14:23:56	kernel		lagg0.3: promiscuous mode enabled
                    Jan 14 14:23:56	kernel		lagg0.2: promiscuous mode enabled
                    Jan 14 14:23:56	kernel		#12 0xffffffff81306b8b at fast_syscall_common+0xf8
                    Jan 14 14:23:56	kernel		#11 0xffffffff813333be at amd64_syscall+0x12e
                    Jan 14 14:23:56	kernel		#10 0xffffffff80df215a at sys_ioctl+0x12a
                    Jan 14 14:23:56	kernel		#9 0xffffffff80df2412 at kern_ioctl+0x202
                    Jan 14 14:23:56	kernel		#8 0xffffffff80bec3ce at devfs_ioctl_f+0x1e
                    Jan 14 14:23:56	kernel		#7 0xffffffff80e820d1 at vn_ioctl+0x131
                    Jan 14 14:23:56	kernel		#6 0xffffffff80bebd1d at devfs_ioctl+0xcd
                    Jan 14 14:23:56	kernel		#5 0xffffffff80ea8c3d at bpfioctl+0xe9d
                    Jan 14 14:23:56	kernel		#4 0xffffffff80eb970a at ifpromisc+0x2a
                    Jan 14 14:23:56	kernel		#3 0xffffffff80eb983e at if_setflag+0xde
                    Jan 14 14:23:56	kernel		#2 0xffffffff80ed75a3 at vlan_ioctl+0x203
                    Jan 14 14:23:56	kernel		#1 0xffffffff80d855c3 at _sx_xlock+0x63
                    Jan 14 14:23:56	kernel		#0 0xffffffff80dec7ed at witness_checkorder+0xbfd
                    Jan 14 14:23:56	kernel		lock order bpf global lock -> vlan_sx attempted at:
                    Jan 14 14:23:56	kernel		2nd 0xffffffff8365c210 vlan_sx (vlan_sx, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/if_vlan.c:2144
                    Jan 14 14:23:56	kernel		1st 0xffffffff8365ae98 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/bpf.c:1518
                    Jan 14 14:23:56	kernel		lock order reversal:
                    Jan 14 14:23:56	kernel		ix5: promiscuous mode enabled
                    Jan 14 14:23:56	kernel		#12 0xffffffff81306b8b at fast_syscall_common+0xf8
                    Jan 14 14:23:56	kernel		#11 0xffffffff813333be at amd64_syscall+0x12e
                    Jan 14 14:23:56	kernel		#10 0xffffffff80df215a at sys_ioctl+0x12a
                    Jan 14 14:23:56	kernel		#9 0xffffffff80df2412 at kern_ioctl+0x202
                    Jan 14 14:23:56	kernel		#8 0xffffffff80bec3ce at devfs_ioctl_f+0x1e
                    Jan 14 14:23:56	kernel		#7 0xffffffff80e820d1 at vn_ioctl+0x131
                    Jan 14 14:23:56	kernel		#6 0xffffffff80bebd1d at devfs_ioctl+0xcd
                    Jan 14 14:23:56	kernel		#5 0xffffffff80ea8c3d at bpfioctl+0xe9d
                    Jan 14 14:23:56	kernel		#4 0xffffffff80eb970a at ifpromisc+0x2a
                    Jan 14 14:23:56	kernel		#3 0xffffffff80eb983e at if_setflag+0xde
                    Jan 14 14:23:56	kernel		#2 0xffffffff80ee687f at iflib_if_ioctl+0x2df
                    Jan 14 14:23:56	kernel		#1 0xffffffff80d855c3 at _sx_xlock+0x63
                    Jan 14 14:23:56	kernel		#0 0xffffffff80dec7ed at witness_checkorder+0xbfd
                    Jan 14 14:23:56	kernel		lock order bpf global lock -> iflib ctx lock attempted at:
                    Jan 14 14:23:56	kernel		#17 0xffffffff80db7be1 at device_attach+0x3c1
                    Jan 14 14:23:56	kernel		#16 0xffffffff80940670 at pci_attach+0xe0
                    Jan 14 14:23:56	kernel		#15 0xffffffff80db8dd8 at bus_generic_attach+0x18
                    Jan 14 14:23:56	kernel		#14 0xffffffff80db7790 at device_probe_and_attach+0x70
                    Jan 14 14:23:56	kernel		#13 0xffffffff80db7be1 at device_attach+0x3c1
                    Jan 14 14:23:56	kernel		#12 0xffffffff80db8dd8 at bus_generic_attach+0x18
                    Jan 14 14:23:56	kernel		#11 0xffffffff80db7790 at device_probe_and_attach+0x70
                    Jan 14 14:23:56	kernel		#10 0xffffffff80db7be1 at device_attach+0x3c1
                    Jan 14 14:23:56	kernel		#9 0xffffffff80940670 at pci_attach+0xe0
                    Jan 14 14:23:56	kernel		#8 0xffffffff80db8dd8 at bus_generic_attach+0x18
                    Jan 14 14:23:56	kernel		#7 0xffffffff80db7790 at device_probe_and_attach+0x70
                    Jan 14 14:23:56	kernel		#6 0xffffffff80db7be1 at device_attach+0x3c1
                    Jan 14 14:23:56	kernel		#5 0xffffffff80edf0b5 at iflib_device_attach+0xb5
                    Jan 14 14:23:56	kernel		#4 0xffffffff80eda99b at iflib_device_register+0xceb
                    Jan 14 14:23:56	kernel		#3 0xffffffff80ec50eb at ether_ifattach+0xeb
                    Jan 14 14:23:56	kernel		#2 0xffffffff80ea66d8 at bpfattach2+0xd8
                    Jan 14 14:23:56	kernel		#1 0xffffffff80d855c3 at _sx_xlock+0x63
                    Jan 14 14:23:56	kernel		#0 0xffffffff80debf1d at witness_checkorder+0x32d
                    Jan 14 14:23:56	kernel		lock order iflib ctx lock -> bpf global lock established at:
                    Jan 14 14:23:56	kernel		2nd 0xfffff80005fe4580 iflib ctx lock (iflib ctx lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/iflib.c:4523
                    Jan 14 14:23:56	kernel		1st 0xffffffff8365ae98 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/bpf.c:1518
                    Jan 14 14:23:56	kernel		lock order reversal:
                    
                    1 Reply Last reply Reply Quote 0
                    • M
                      marcosm Netgate
                      last edited by

                      See https://redmine.pfsense.org/issues/13896

                      lock order reversal:
                       1st 0xfffff8031035c590 lle (lle, rw) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/netinet6/in6.c:2401
                       2nd 0xfffffe0021f30270 nd6 list (nd6 list, rw) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/netinet6/nd6_rtr.c:863
                      lock order nd6 list -> lle established at:
                      #0 0xffffffff80debf1d at witness_checkorder+0x32d
                      #1 0xffffffff80d74c97 at _rw_wlock_cookie+0x67
                      #2 0xffffffff810077ed at nd6_llinfo_timer+0x9d
                      #3 0xffffffff80d98ed1 at softclock_call_cc+0x151
                      #4 0xffffffff80d9a726 at softclock_thread+0xc6
                      #5 0xffffffff80d31dc0 at fork_exit+0x80
                      #6 0xffffffff813072de at fork_trampoline+0xe
                      lock order lle -> nd6 list attempted at:
                      #0 0xffffffff80dec7ed at witness_checkorder+0xbfd
                      #1 0xffffffff80d74c97 at _rw_wlock_cookie+0x67
                      #2 0xffffffff8100dce1 at defrouter_remove+0x41
                      #3 0xffffffff8100ab0d at nd6_na_input+0x97d
                      #4 0xffffffff80fdb7e4 at icmp6_input+0x8a4
                      #5 0xffffffff80ff53d3 at ip6_input+0xbc3
                      #6 0xffffffff80eea7a0 at netisr_dispatch_src+0x220
                      #7 0xffffffff80ec4e7c at ether_demux+0x17c
                      #8 0xffffffff80ec64f6 at ether_nh_input+0x3f6
                      #9 0xffffffff80eea62f at netisr_dispatch_src+0xaf
                      #10 0xffffffff80ec5339 at ether_input+0x99
                      #11 0xffffffff80ec4dcd at ether_demux+0xcd
                      #12 0xffffffff80ec64f6 at ether_nh_input+0x3f6
                      #13 0xffffffff80eea62f at netisr_dispatch_src+0xaf
                      #14 0xffffffff80ec5339 at ether_input+0x99
                      #15 0xffffffff80ee5ed4 at iflib_rxeof+0xdf4
                      #16 0xffffffff80edfeea at _task_fn_rx+0x7a
                      #17 0xffffffff80dc7917 at gtaskqueue_run_locked+0xa7
                      
                      1 Reply Last reply Reply Quote 0
                      • stephenw10S stephenw10 referenced this topic on
                      • stephenw10S stephenw10 referenced this topic on
                      • P
                        pfsvrb
                        last edited by

                        FYI, seeing these on the latest 2.7.0-BETA snapshot build from today (06/21/2023).

                        This is a google fiber system with no VLANs set using genuine Intel i210 NICs. WAN interface is DHCP/DHCP6 and the LAN interface is tracking DHCP6 on WAN. IPv6 still works great after the message, just wanted to post here as an FYI in case it helps the devs. This same lock order message pops up randomly between 20 minutes and 4 hours after the system has booted.

                        Jun 21 10:46:41	kernel		#16 0xffffffff8129035e at fork_trampoline+0xe
                        Jun 21 10:46:41	kernel		#15 0xffffffff80ca6cf0 at fork_exit+0x80
                        Jun 21 10:46:41	kernel		#14 0xffffffff80d3c7a2 at gtaskqueue_thread_loop+0xc2
                        Jun 21 10:46:41	kernel		#13 0xffffffff80d3c977 at gtaskqueue_run_locked+0xa7
                        Jun 21 10:46:41	kernel		#12 0xffffffff80e55bda at _task_fn_rx+0x7a
                        Jun 21 10:46:41	kernel		#11 0xffffffff80e5bbb9 at iflib_rxeof+0xe59
                        Jun 21 10:46:41	kernel		#10 0xffffffff80e3b1c9 at ether_input+0x99
                        Jun 21 10:46:41	kernel		#9 0xffffffff80e6022d at netisr_dispatch_src+0xad
                        Jun 21 10:46:41	kernel		#8 0xffffffff80e3c359 at ether_nh_input+0x349
                        Jun 21 10:46:41	kernel		#7 0xffffffff80e3adba at ether_demux+0x17a
                        Jun 21 10:46:41	kernel		#6 0xffffffff80e6039d at netisr_dispatch_src+0x21d
                        Jun 21 10:46:41	kernel		#5 0xffffffff80f6c227 at ip6_input+0xc77
                        Jun 21 10:46:41	kernel		#4 0xffffffff80f521b6 at icmp6_input+0x9e6
                        Jun 21 10:46:41	kernel		#3 0xffffffff80f81bf1 at nd6_na_input+0x981
                        Jun 21 10:46:41	kernel		#2 0xffffffff80f84fc1 at defrouter_remove+0x41
                        Jun 21 10:46:41	kernel		#1 0xffffffff80cea087 at _rw_wlock_cookie+0x67
                        Jun 21 10:46:41	kernel		#0 0xffffffff80d615e3 at witness_checkorder+0xbb3
                        Jun 21 10:46:41	kernel		lock order lle -> nd6 list attempted at:
                        Jun 21 10:46:41	kernel		#17 0xffffffff80d3c7a2 at gtaskqueue_thread_loop+0xc2
                        Jun 21 10:46:41	kernel		#16 0xffffffff80d3c977 at gtaskqueue_run_locked+0xa7
                        Jun 21 10:46:41	kernel		#15 0xffffffff80e55bda at _task_fn_rx+0x7a
                        Jun 21 10:46:41	kernel		#14 0xffffffff80e5bbb9 at iflib_rxeof+0xe59
                        Jun 21 10:46:41	kernel		#13 0xffffffff80e3b1c9 at ether_input+0x99
                        Jun 21 10:46:41	kernel		#12 0xffffffff80e6022d at netisr_dispatch_src+0xad
                        Jun 21 10:46:41	kernel		#11 0xffffffff80e3c359 at ether_nh_input+0x349
                        Jun 21 10:46:41	kernel		#10 0xffffffff80e3adba at ether_demux+0x17a
                        Jun 21 10:46:41	kernel		#9 0xffffffff80e6039d at netisr_dispatch_src+0x21d
                        Jun 21 10:46:41	kernel		#8 0xffffffff80f6c227 at ip6_input+0xc77
                        Jun 21 10:46:41	kernel		#7 0xffffffff80f52068 at icmp6_input+0x898
                        Jun 21 10:46:41	kernel		#6 0xffffffff80f84445 at nd6_ra_input+0x1335
                        Jun 21 10:46:41	kernel		#5 0xffffffff80f7d858 at nd6_cache_lladdr+0x858
                        Jun 21 10:46:41	kernel		#4 0xffffffff80f854ac at defrouter_select_fib+0xdc
                        Jun 21 10:46:41	kernel		#3 0xffffffff80f7c023 at nd6_lookup+0x83
                        Jun 21 10:46:41	kernel		#2 0xffffffff80f5a29b at in6_lltable_lookup+0x10b
                        Jun 21 10:46:41	kernel		#1 0xffffffff80cead92 at __rw_rlock_int+0x82
                        Jun 21 10:46:41	kernel		#0 0xffffffff80d60d2b at witness_checkorder+0x2fb
                        Jun 21 10:46:41	kernel		lock order nd6 list -> lle established at:
                        Jun 21 10:46:41	kernel		2nd 0xfffffe0021de8460 nd6 list (nd6 list, rw) @ /var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-RELENG_2_7_0/sys/netinet6/nd6_rtr.c:864
                        Jun 21 10:46:41	kernel		1st 0xfffff8000218c410 lle (lle, rw) @ /var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-RELENG_2_7_0/sys/net/if_llatbl.c:370
                        Jun 21 10:46:41	kernel		lock order reversal:
                        
                        1 Reply Last reply Reply Quote 1
                        • jimpJ jimp moved this topic from CE 2.7.0 Development Snapshots (Retired) on
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.