Reporting Lock Order Reversal (LOR) Backtraces

jimp · Sep 20, 2022, 8:33 PM

Lock Order Reversals are non-fatal errors that indicate a potential problem in OS code. They are only visible because we have specific debugging options enabled on current development snapshot kernels. These options are disabled before release. Though they are not fatal and are not typically a cause for concern, they are still worth reporting. FreeBSD developers on our staff can look them over and either fix them or report them upstream. In many cases these are already known issues in FreeBSD that have not yet been fixed.

If you encounter one, please post the entire LOR backtrace from the console output or system message buffer. If possible, please include some detail about when the LOR happened (e.g. when configuring a CARP VIP, during a CARP transition, etc).

Because these are non-fatal and relatively common, there is no need to make separate threads for them. Feel free to add LORs here in this thread.

stephenw10 · Sep 22, 2022, 8:07 PM

In a VM. I was doing nothing I was aware of at the time. Only noticed it later.

Sep 22 13:00:53 	kernel 		lock order reversal:
Sep 22 13:00:53 	kernel 		1st 0xffffffff8364f118 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/bpf.c:1518
Sep 22 13:00:53 	kernel 		2nd 0xfffff80005d4fd80 iflib ctx lock (iflib ctx lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/iflib.c:4373
Sep 22 13:00:53 	kernel 		lock order iflib ctx lock -> bpf global lock established at:
Sep 22 13:00:53 	kernel 		#0 0xffffffff80de8a0d at witness_checkorder+0x32d
Sep 22 13:00:53 	kernel 		#1 0xffffffff80d83043 at _sx_xlock+0x63
Sep 22 13:00:53 	kernel 		#2 0xffffffff80ea2158 at bpfattach2+0xd8
Sep 22 13:00:53 	kernel 		#3 0xffffffff80ec063b at ether_ifattach+0xeb
Sep 22 13:00:53 	kernel 		#4 0xffffffff80ed5dbb at iflib_device_register+0xceb
Sep 22 13:00:53 	kernel 		#5 0xffffffff80eda4d5 at iflib_device_attach+0xb5
Sep 22 13:00:53 	kernel 		#6 0xffffffff80db4701 at device_attach+0x3c1
Sep 22 13:00:53 	kernel 		#7 0xffffffff80db42b0 at device_probe_and_attach+0x70
Sep 22 13:00:53 	kernel 		#8 0xffffffff80db58f8 at bus_generic_attach+0x18
Sep 22 13:00:53 	kernel 		#9 0xffffffff8093f2c0 at pci_attach+0xe0
Sep 22 13:00:53 	kernel 		#10 0xffffffff8120ea49 at acpi_pci_attach+0x19
Sep 22 13:00:53 	kernel 		#11 0xffffffff80db4701 at device_attach+0x3c1
Sep 22 13:00:53 	kernel 		#12 0xffffffff80db42b0 at device_probe_and_attach+0x70
Sep 22 13:00:53 	kernel 		#13 0xffffffff80db58f8 at bus_generic_attach+0x18
Sep 22 13:00:53 	kernel 		#14 0xffffffff81211ced at acpi_pcib_acpi_attach+0x42d
Sep 22 13:00:53 	kernel 		#15 0xffffffff80db4701 at device_attach+0x3c1
Sep 22 13:00:53 	kernel 		#16 0xffffffff80db42b0 at device_probe_and_attach+0x70
Sep 22 13:00:53 	kernel 		#17 0xffffffff80db58f8 at bus_generic_attach+0x18
Sep 22 13:00:53 	kernel 		lock order bpf global lock -> iflib ctx lock attempted at:
Sep 22 13:00:53 	kernel 		#0 0xffffffff80de92dd at witness_checkorder+0xbfd
Sep 22 13:00:53 	kernel 		#1 0xffffffff80d83043 at _sx_xlock+0x63
Sep 22 13:00:53 	kernel 		#2 0xffffffff80ee1c8f at iflib_if_ioctl+0x2df
Sep 22 13:00:53 	kernel 		#3 0xffffffff80eb527e at if_setflag+0xde
Sep 22 13:00:53 	kernel 		#4 0xffffffff80eb514a at ifpromisc+0x2a
Sep 22 13:00:53 	kernel 		#5 0xffffffff80ea46bd at bpfioctl+0xe9d
Sep 22 13:00:53 	kernel 		#6 0xffffffff80bea23d at devfs_ioctl+0xcd
Sep 22 13:00:53 	kernel 		#7 0xffffffff80e7dee1 at vn_ioctl+0x131
Sep 22 13:00:53 	kernel 		#8 0xffffffff80bea8ee at devfs_ioctl_f+0x1e
Sep 22 13:00:53 	kernel 		#9 0xffffffff80deef02 at kern_ioctl+0x202
Sep 22 13:00:53 	kernel 		#10 0xffffffff80deec4a at sys_ioctl+0x12a
Sep 22 13:00:53 	kernel 		#11 0xffffffff8133540e at amd64_syscall+0x12e
Sep 22 13:00:53 	kernel 		#12 0xffffffff8130821b at fast_syscall_common+0xf8

Looks identical to this though: https://redmine.netgate.com/issues/8868
That was on a 4100.

Anaerin · Oct 13, 2022, 10:33 PM

On startup, I got the following:

Oct 13 16:11:45 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
Oct 13 16:11:46 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
Oct 13 16:11:46 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
Oct 13 16:11:46 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
Oct 13 16:11:46 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
Oct 13 16:11:46 router php-fpm[343]: /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.
Oct 13 16:11:46 router vnstatd[74726]: Error: pidfile "/var/run/vnstat/vnstat.pid" lock failed (Resource temporarily unavailable), exiting.
Oct 13 16:11:46 router kernel: gif0: promiscuous mode enabled
Oct 13 16:11:46 router kernel: lock order reversal:
Oct 13 16:11:46 router kernel:  1st 0xffffffff8365b698 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sy
s/net/bpf.c:1518
Oct 13 16:11:46 router kernel:  2nd 0xffffffff8365ca10 vlan_sx (vlan_sx, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/if_vlan.c:
2144
Oct 13 16:11:46 router kernel: lock order bpf global lock -> vlan_sx attempted at:
Oct 13 16:11:46 router kernel: #0 0xffffffff80deb68d at witness_checkorder+0xbfd
Oct 13 16:11:46 router kernel: #1 0xffffffff80d84783 at _sx_xlock+0x63
Oct 13 16:11:46 router kernel: #2 0xffffffff80ed6113 at vlan_ioctl+0x203
Oct 13 16:11:46 router kernel: #3 0xffffffff80eb83ae at if_setflag+0xde
Oct 13 16:11:46 router kernel: #4 0xffffffff80eb827a at ifpromisc+0x2a
Oct 13 16:11:46 router kernel: #5 0xffffffff80ea77ad at bpfioctl+0xe9d
Oct 13 16:11:46 router kernel: #6 0xffffffff80beb6bd at devfs_ioctl+0xcd
Oct 13 16:11:46 router kernel: #7 0xffffffff80e80d21 at vn_ioctl+0x131
Oct 13 16:11:46 router kernel: #8 0xffffffff80bebd6e at devfs_ioctl_f+0x1e
Oct 13 16:11:46 router kernel: #9 0xffffffff80df12b2 at kern_ioctl+0x202
Oct 13 16:11:46 router kernel: #10 0xffffffff80df0ffa at sys_ioctl+0x12a
Oct 13 16:11:46 router kernel: #11 0xffffffff8133a3be at amd64_syscall+0x12e
Oct 13 16:11:46 router kernel: #12 0xffffffff8130dc8b at fast_syscall_common+0xf8
Oct 13 16:11:46 router kernel: em0: promiscuous mode enabled
Oct 13 16:11:46 router kernel: em0.3000: promiscuous mode enabled
Oct 13 22:11:46 router softflowd[74739]: softflowd v1.0.0 starting data collection
Oct 13 22:11:46 router softflowd[74739]: Exporting flows to [10.2.0.22]:2055

LandRocket · Jan 24, 2023, 7:06 PM

Got one for you:

Dec 11 10:39:09 router kernel: lock order reversal:
Dec 11 10:39:09 router kernel:  1st 0xffffffff8365ae98 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-s
rc-devel-main/sys/net/bpf.c:1518
Dec 11 10:39:09 router kernel:  2nd 0xfffff800a2448628 if_lagg sx (if_lagg sx, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-m
ain/sys/net/if_lagg.c:1669
Dec 11 10:39:09 router kernel: lock order if_lagg sx -> bpf global lock established at:
Dec 11 10:39:09 router kernel: #0 0xffffffff80debf1d at witness_checkorder+0x32d
Dec 11 10:39:09 router kernel: #1 0xffffffff80d855c3 at _sx_xlock+0x63
Dec 11 10:39:09 router kernel: #2 0xffffffff80ea66d8 at bpfattach2+0xd8
Dec 11 10:39:09 router kernel: #3 0xffffffff80ec50eb at ether_ifattach+0xeb
Dec 11 10:39:09 router kernel: #4 0xffffffff80eca37a at lagg_clone_create+0x29a
Dec 11 10:39:09 router kernel: #5 0xffffffff80ec20ac at if_clone_createif+0x17c
Dec 11 10:39:09 router kernel: #6 0xffffffff80ec1efa at ifc_create_ifp+0xea
Dec 11 10:39:09 router kernel: #7 0xffffffff80ec21d4 at if_clone_create+0x34
Dec 11 10:39:09 router kernel: #8 0xffffffff80eb907b at ifioctl+0x92b
Dec 11 10:39:09 router kernel: #9 0xffffffff80df2412 at kern_ioctl+0x202
Dec 11 10:39:09 router kernel: #10 0xffffffff80df215a at sys_ioctl+0x12a
Dec 11 10:39:09 router kernel: #11 0xffffffff813333be at amd64_syscall+0x12e
Dec 11 10:39:09 router kernel: #12 0xffffffff81306b8b at fast_syscall_common+0xf8
Dec 11 10:39:09 router kernel: lock order bpf global lock -> if_lagg sx attempted at:
Dec 11 10:39:09 router kernel: #0 0xffffffff80dec7ed at witness_checkorder+0xbfd
Dec 11 10:39:09 router kernel: #1 0xffffffff80d855c3 at _sx_xlock+0x63
Dec 11 10:39:09 router kernel: #2 0xffffffff80ecb382 at lagg_ioctl+0x742
Dec 11 10:39:09 router kernel: #3 0xffffffff80eb983e at if_setflag+0xde
Dec 11 10:39:09 router kernel: #4 0xffffffff80eb970a at ifpromisc+0x2a
Dec 11 10:39:09 router kernel: #5 0xffffffff80ea8c3d at bpfioctl+0xe9d
Dec 11 10:39:09 router kernel: #6 0xffffffff80bebd1d at devfs_ioctl+0xcd
Dec 11 10:39:09 router kernel: #7 0xffffffff80e820d1 at vn_ioctl+0x131
Dec 11 10:39:09 router kernel: #8 0xffffffff80bec3ce at devfs_ioctl_f+0x1e
Dec 11 10:39:09 router kernel: #9 0xffffffff80df2412 at kern_ioctl+0x202
Dec 11 10:39:09 router kernel: #10 0xffffffff80df215a at sys_ioctl+0x12a
Dec 11 10:39:09 router kernel: #11 0xffffffff813333be at amd64_syscall+0x12e
Dec 11 10:39:09 router kernel: #12 0xffffffff81306b8b at fast_syscall_common+0xf8
Dec 11 10:39:09 router kernel: igb0: promiscuous mode enabled
Dec 11 10:39:09 router kernel: igb1: promiscuous mode enabled
Dec 11 10:39:09 router kernel: lagg0: promiscuous mode enabled

this usually happens shortly after bootup, and as it suggests, I have 2 interfaces (igb0,igb1) configured as a LAN LACP (lagg0)

It's a 4-port card (igb0-igb3):

Dec 11 10:38:41 router kernel: igb0: <Intel(R) PRO/1000 ET 82576 (Quad Copper)> port 0xd020-0xd03f mem 0xf7c20000-0xf7c3ffff,0xf7800000-0xf7bfffff,0xf7c44000-0xf7c47fff
irq 18 at device 0.0 on pci3

Not sure if it's related but other messages that are in the log that may be relevant:
Dec 11 10:38:41 router kernel: debugnet_any_ifnet_update: Bad dn_init result from igb0 (ifp 0xfffff80001ce8800), ignoring.

omitted duplicate message pertaining to igb1

The message doesn't appear for igb2 (currently assigned to WAN1). igb3 not currently in use

renesdk · Dec 30, 2022, 11:27 PM

I get the almost the same as LandRocket
I'm running Intel Ethernet Controller I226-V (4 ports).

The only difference is mine is missing:
Dec 11 10:39:09 router kernel: igb0: promiscuous mode enabled
Dec 11 10:39:09 router kernel: igb1: promiscuous mode enabled
Dec 11 10:39:09 router kernel: lagg0: promiscuous mode enabled

marcosm · Jan 3, 2023, 12:20 AM

Logs reversed:

Jan 2 17:55:03 	kernel 		#17 0xffffffff80ef020f at netisr_dispatch_src+0xaf
Jan 2 17:55:03 	kernel 		#16 0xffffffff80ecc0d6 at ether_nh_input+0x3f6
Jan 2 17:55:03 	kernel 		#15 0xffffffff80eca94b at ether_demux+0x6b
Jan 2 17:55:03 	kernel 		#14 0xffffffff80ef3ca7 at pfil_run_hooks+0xb7
Jan 2 17:55:03 	kernel 		#13 0xffffffff8106f3b5 at pf_eth_check_in+0x25
Jan 2 17:55:03 	kernel 		#12 0xffffffff81049e75 at pf_test_eth+0x14a5
Jan 2 17:55:03 	kernel 		#11 0xffffffff84c645ba at dummynet_io+0x41a
Jan 2 17:55:03 	kernel 		#10 0xffffffff84c640bd at dummynet_send+0x17d
Jan 2 17:55:03 	kernel 		#9 0xffffffff80ecaa5c at ether_demux+0x17c
Jan 2 17:55:03 	kernel 		#8 0xffffffff80ef0380 at netisr_dispatch_src+0x220
Jan 2 17:55:03 	kernel 		#7 0xffffffff80f99b28 at ip_input+0x298
Jan 2 17:55:03 	kernel 		#6 0xffffffff80fb1acb at tcp_input+0xb
Jan 2 17:55:03 	kernel 		#5 0xffffffff80fb0585 at tcp_input_with_port+0x5e5
Jan 2 17:55:03 	kernel 		#4 0xffffffff80f93bc4 at in_pcblookup_mbuf+0x24
Jan 2 17:55:03 	kernel 		#3 0xffffffff80f93ab3 at in_pcblookup_hash+0x83
Jan 2 17:55:03 	kernel 		#2 0xffffffff80f927d6 at inp_smr_lock+0x136
Jan 2 17:55:03 	kernel 		#1 0xffffffff80d7a597 at _rw_wlock_cookie+0x67
Jan 2 17:55:03 	kernel 		#0 0xffffffff80df23cd at witness_checkorder+0xbfd
Jan 2 17:55:03 	kernel 		lock order pf rulesets -> tcpinp attempted at:
Jan 2 17:55:03 	kernel 		#13 0xffffffff8130c72b at fast_syscall_common+0xf8
Jan 2 17:55:03 	kernel 		#12 0xffffffff813393be at amd64_syscall+0x12e
Jan 2 17:55:03 	kernel 		#11 0xffffffff80e34485 at sys_connect+0x75
Jan 2 17:55:03 	kernel 		#10 0xffffffff80e34574 at kern_connectat+0xc4
Jan 2 17:55:03 	kernel 		#9 0xffffffff80e2d84f at soconnectat+0xaf
Jan 2 17:55:03 	kernel 		#8 0xffffffff80fd443a at tcp_usr_connect+0x13a
Jan 2 17:55:03 	kernel 		#7 0xffffffff80fd8001 at tcp_output+0x31
Jan 2 17:55:03 	kernel 		#6 0xffffffff80fbe3c4 at tcp_default_output+0x2094
Jan 2 17:55:03 	kernel 		#5 0xffffffff80fa3216 at ip_output+0xd26
Jan 2 17:55:03 	kernel 		#4 0xffffffff80ef3ca7 at pfil_run_hooks+0xb7
Jan 2 17:55:03 	kernel 		#3 0xffffffff8106f12f at pf_check_out+0x1f
Jan 2 17:55:03 	kernel 		#2 0xffffffff8104a175 at pf_test+0xc5
Jan 2 17:55:03 	kernel 		#1 0xffffffff80d79150 at _rm_rlock_debug+0x130
Jan 2 17:55:03 	kernel 		#0 0xffffffff80df1afd at witness_checkorder+0x32d
Jan 2 17:55:03 	kernel 		lock order tcpinp -> pf rulesets established at:
Jan 2 17:55:03 	kernel 		2nd 0xfffff801b3b49a90 tcpinp (tcpinp, rw) @ /var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/sys/netinet/in_pcb.c:1554
Jan 2 17:55:03 	kernel 		1st 0xffffffff836726d8 pf rulesets (pf rulesets, rm) @ /var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/sys/netpfil/pf/pf.c:7130
Jan 2 17:55:03 	kernel 		lock order reversal:

bmeeks · Jan 6, 2023, 4:19 PM

Lock order reversal logged by Suricata when starting up using netmap Inline IPS Mode. This appears to be a problem within the iflib library when netmap operation is enabled. This is captured from a pfSense 2.7.0 CE snapshot current as of January 6, 2023. The system log setting is configured to display the most recent events first, so the order of the logged messages is reversed.

Jan 6 11:07:51	kernel		#12 0xffffffff81306b8b at fast_syscall_common+0xf8
Jan 6 11:07:51	kernel		#11 0xffffffff813333be at amd64_syscall+0x12e
Jan 6 11:07:51	kernel		#10 0xffffffff80df215a at sys_ioctl+0x12a
Jan 6 11:07:51	kernel		#9 0xffffffff80df2412 at kern_ioctl+0x202
Jan 6 11:07:51	kernel		#8 0xffffffff80bec3ce at devfs_ioctl_f+0x1e
Jan 6 11:07:51	kernel		#7 0xffffffff80e820d1 at vn_ioctl+0x131
Jan 6 11:07:51	kernel		#6 0xffffffff80bebd1d at devfs_ioctl+0xcd
Jan 6 11:07:51	kernel		#5 0xffffffff80ea8c3d at bpfioctl+0xe9d
Jan 6 11:07:51	kernel		#4 0xffffffff80eb970a at ifpromisc+0x2a
Jan 6 11:07:51	kernel		#3 0xffffffff80eb983e at if_setflag+0xde
Jan 6 11:07:51	kernel		#2 0xffffffff80ee687f at iflib_if_ioctl+0x2df
Jan 6 11:07:51	kernel		#1 0xffffffff80d855c3 at _sx_xlock+0x63
Jan 6 11:07:51	kernel		#0 0xffffffff80dec7ed at witness_checkorder+0xbfd
Jan 6 11:07:51	kernel		lock order bpf global lock -> iflib ctx lock attempted at:
Jan 6 11:07:51	kernel		#17 0xffffffff80db8dd8 at bus_generic_attach+0x18
Jan 6 11:07:51	kernel		#16 0xffffffff80db7790 at device_probe_and_attach+0x70
Jan 6 11:07:51	kernel		#15 0xffffffff80db7be1 at device_attach+0x3c1
Jan 6 11:07:51	kernel		#14 0xffffffff81210be0 at acpi_pcib_pci_attach+0xa0
Jan 6 11:07:51	kernel		#13 0xffffffff80db8dd8 at bus_generic_attach+0x18
Jan 6 11:07:51	kernel		#12 0xffffffff80db7790 at device_probe_and_attach+0x70
Jan 6 11:07:51	kernel		#11 0xffffffff80db7be1 at device_attach+0x3c1
Jan 6 11:07:51	kernel		#10 0xffffffff8120cea9 at acpi_pci_attach+0x19
Jan 6 11:07:51	kernel		#9 0xffffffff80940670 at pci_attach+0xe0
Jan 6 11:07:51	kernel		#8 0xffffffff80db8dd8 at bus_generic_attach+0x18
Jan 6 11:07:51	kernel		#7 0xffffffff80db7790 at device_probe_and_attach+0x70
Jan 6 11:07:51	kernel		#6 0xffffffff80db7be1 at device_attach+0x3c1
Jan 6 11:07:51	kernel		#5 0xffffffff80edf0b5 at iflib_device_attach+0xb5
Jan 6 11:07:51	kernel		#4 0xffffffff80eda99b at iflib_device_register+0xceb
Jan 6 11:07:51	kernel		#3 0xffffffff80ec50eb at ether_ifattach+0xeb
Jan 6 11:07:51	kernel		#2 0xffffffff80ea66d8 at bpfattach2+0xd8
Jan 6 11:07:51	kernel		#1 0xffffffff80d855c3 at _sx_xlock+0x63
Jan 6 11:07:51	kernel		#0 0xffffffff80debf1d at witness_checkorder+0x32d
Jan 6 11:07:51	kernel		lock order iflib ctx lock -> bpf global lock established at:
Jan 6 11:07:51	kernel		2nd 0xfffff80005c80d80 iflib ctx lock (iflib ctx lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/iflib.c:4523
Jan 6 11:07:51	kernel		1st 0xffffffff8365ae98 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/bpf.c:1518
Jan 6 11:07:51	kernel		lock order reversal:
Jan 6 11:07:48	kernel		em0: permanently promiscuous mode enabled
Jan 6 11:07:48	kernel		268.102300 [ 851] iflib_netmap_config txr 1 rxr 1 txd 1024 rxd 1024 rbufsz 2048
Jan 6 11:07:48	kernel		268.102281 [ 851] iflib_netmap_config txr 1 rxr 1 txd 1024 rxd 1024 rbufsz 2048
Jan 6 11:07:48	kernel		268.102248 [ 851] iflib_netmap_config txr 1 rxr 1 txd 1024 rxd 1024 rbufsz 2048

bud4life · Jan 14, 2023, 8:19 PM

I have interfaces locking up having to reboot, ever since I upgraded 2 weeks ago

Jan 14 14:23:57	kernel		lagg0.401: promiscuous mode enabled
Jan 14 14:23:57	kernel		lagg0.500: promiscuous mode enabled
Jan 14 14:23:56	kernel		lagg0.3: promiscuous mode enabled
Jan 14 14:23:56	kernel		lagg0.2: promiscuous mode enabled
Jan 14 14:23:56	kernel		#12 0xffffffff81306b8b at fast_syscall_common+0xf8
Jan 14 14:23:56	kernel		#11 0xffffffff813333be at amd64_syscall+0x12e
Jan 14 14:23:56	kernel		#10 0xffffffff80df215a at sys_ioctl+0x12a
Jan 14 14:23:56	kernel		#9 0xffffffff80df2412 at kern_ioctl+0x202
Jan 14 14:23:56	kernel		#8 0xffffffff80bec3ce at devfs_ioctl_f+0x1e
Jan 14 14:23:56	kernel		#7 0xffffffff80e820d1 at vn_ioctl+0x131
Jan 14 14:23:56	kernel		#6 0xffffffff80bebd1d at devfs_ioctl+0xcd
Jan 14 14:23:56	kernel		#5 0xffffffff80ea8c3d at bpfioctl+0xe9d
Jan 14 14:23:56	kernel		#4 0xffffffff80eb970a at ifpromisc+0x2a
Jan 14 14:23:56	kernel		#3 0xffffffff80eb983e at if_setflag+0xde
Jan 14 14:23:56	kernel		#2 0xffffffff80ed75a3 at vlan_ioctl+0x203
Jan 14 14:23:56	kernel		#1 0xffffffff80d855c3 at _sx_xlock+0x63
Jan 14 14:23:56	kernel		#0 0xffffffff80dec7ed at witness_checkorder+0xbfd
Jan 14 14:23:56	kernel		lock order bpf global lock -> vlan_sx attempted at:
Jan 14 14:23:56	kernel		2nd 0xffffffff8365c210 vlan_sx (vlan_sx, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/if_vlan.c:2144
Jan 14 14:23:56	kernel		1st 0xffffffff8365ae98 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/bpf.c:1518
Jan 14 14:23:56	kernel		lock order reversal:
Jan 14 14:23:56	kernel		ix5: promiscuous mode enabled
Jan 14 14:23:56	kernel		#12 0xffffffff81306b8b at fast_syscall_common+0xf8
Jan 14 14:23:56	kernel		#11 0xffffffff813333be at amd64_syscall+0x12e
Jan 14 14:23:56	kernel		#10 0xffffffff80df215a at sys_ioctl+0x12a
Jan 14 14:23:56	kernel		#9 0xffffffff80df2412 at kern_ioctl+0x202
Jan 14 14:23:56	kernel		#8 0xffffffff80bec3ce at devfs_ioctl_f+0x1e
Jan 14 14:23:56	kernel		#7 0xffffffff80e820d1 at vn_ioctl+0x131
Jan 14 14:23:56	kernel		#6 0xffffffff80bebd1d at devfs_ioctl+0xcd
Jan 14 14:23:56	kernel		#5 0xffffffff80ea8c3d at bpfioctl+0xe9d
Jan 14 14:23:56	kernel		#4 0xffffffff80eb970a at ifpromisc+0x2a
Jan 14 14:23:56	kernel		#3 0xffffffff80eb983e at if_setflag+0xde
Jan 14 14:23:56	kernel		#2 0xffffffff80ee687f at iflib_if_ioctl+0x2df
Jan 14 14:23:56	kernel		#1 0xffffffff80d855c3 at _sx_xlock+0x63
Jan 14 14:23:56	kernel		#0 0xffffffff80dec7ed at witness_checkorder+0xbfd
Jan 14 14:23:56	kernel		lock order bpf global lock -> iflib ctx lock attempted at:
Jan 14 14:23:56	kernel		#17 0xffffffff80db7be1 at device_attach+0x3c1
Jan 14 14:23:56	kernel		#16 0xffffffff80940670 at pci_attach+0xe0
Jan 14 14:23:56	kernel		#15 0xffffffff80db8dd8 at bus_generic_attach+0x18
Jan 14 14:23:56	kernel		#14 0xffffffff80db7790 at device_probe_and_attach+0x70
Jan 14 14:23:56	kernel		#13 0xffffffff80db7be1 at device_attach+0x3c1
Jan 14 14:23:56	kernel		#12 0xffffffff80db8dd8 at bus_generic_attach+0x18
Jan 14 14:23:56	kernel		#11 0xffffffff80db7790 at device_probe_and_attach+0x70
Jan 14 14:23:56	kernel		#10 0xffffffff80db7be1 at device_attach+0x3c1
Jan 14 14:23:56	kernel		#9 0xffffffff80940670 at pci_attach+0xe0
Jan 14 14:23:56	kernel		#8 0xffffffff80db8dd8 at bus_generic_attach+0x18
Jan 14 14:23:56	kernel		#7 0xffffffff80db7790 at device_probe_and_attach+0x70
Jan 14 14:23:56	kernel		#6 0xffffffff80db7be1 at device_attach+0x3c1
Jan 14 14:23:56	kernel		#5 0xffffffff80edf0b5 at iflib_device_attach+0xb5
Jan 14 14:23:56	kernel		#4 0xffffffff80eda99b at iflib_device_register+0xceb
Jan 14 14:23:56	kernel		#3 0xffffffff80ec50eb at ether_ifattach+0xeb
Jan 14 14:23:56	kernel		#2 0xffffffff80ea66d8 at bpfattach2+0xd8
Jan 14 14:23:56	kernel		#1 0xffffffff80d855c3 at _sx_xlock+0x63
Jan 14 14:23:56	kernel		#0 0xffffffff80debf1d at witness_checkorder+0x32d
Jan 14 14:23:56	kernel		lock order iflib ctx lock -> bpf global lock established at:
Jan 14 14:23:56	kernel		2nd 0xfffff80005fe4580 iflib ctx lock (iflib ctx lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/iflib.c:4523
Jan 14 14:23:56	kernel		1st 0xffffffff8365ae98 bpf global lock (bpf global lock, sx) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/net/bpf.c:1518
Jan 14 14:23:56	kernel		lock order reversal:

marcosm · Jan 24, 2023, 7:05 PM

See https://redmine.pfsense.org/issues/13896

lock order reversal:
 1st 0xfffff8031035c590 lle (lle, rw) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/netinet6/in6.c:2401
 2nd 0xfffffe0021f30270 nd6 list (nd6 list, rw) @ /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/netinet6/nd6_rtr.c:863
lock order nd6 list -> lle established at:
#0 0xffffffff80debf1d at witness_checkorder+0x32d
#1 0xffffffff80d74c97 at _rw_wlock_cookie+0x67
#2 0xffffffff810077ed at nd6_llinfo_timer+0x9d
#3 0xffffffff80d98ed1 at softclock_call_cc+0x151
#4 0xffffffff80d9a726 at softclock_thread+0xc6
#5 0xffffffff80d31dc0 at fork_exit+0x80
#6 0xffffffff813072de at fork_trampoline+0xe
lock order lle -> nd6 list attempted at:
#0 0xffffffff80dec7ed at witness_checkorder+0xbfd
#1 0xffffffff80d74c97 at _rw_wlock_cookie+0x67
#2 0xffffffff8100dce1 at defrouter_remove+0x41
#3 0xffffffff8100ab0d at nd6_na_input+0x97d
#4 0xffffffff80fdb7e4 at icmp6_input+0x8a4
#5 0xffffffff80ff53d3 at ip6_input+0xbc3
#6 0xffffffff80eea7a0 at netisr_dispatch_src+0x220
#7 0xffffffff80ec4e7c at ether_demux+0x17c
#8 0xffffffff80ec64f6 at ether_nh_input+0x3f6
#9 0xffffffff80eea62f at netisr_dispatch_src+0xaf
#10 0xffffffff80ec5339 at ether_input+0x99
#11 0xffffffff80ec4dcd at ether_demux+0xcd
#12 0xffffffff80ec64f6 at ether_nh_input+0x3f6
#13 0xffffffff80eea62f at netisr_dispatch_src+0xaf
#14 0xffffffff80ec5339 at ether_input+0x99
#15 0xffffffff80ee5ed4 at iflib_rxeof+0xdf4
#16 0xffffffff80edfeea at _task_fn_rx+0x7a
#17 0xffffffff80dc7917 at gtaskqueue_run_locked+0xa7

pfsvrb · Jun 21, 2023, 4:47 PM

FYI, seeing these on the latest 2.7.0-BETA snapshot build from today (06/21/2023).

This is a google fiber system with no VLANs set using genuine Intel i210 NICs. WAN interface is DHCP/DHCP6 and the LAN interface is tracking DHCP6 on WAN. IPv6 still works great after the message, just wanted to post here as an FYI in case it helps the devs. This same lock order message pops up randomly between 20 minutes and 4 hours after the system has booted.

Jun 21 10:46:41	kernel		#16 0xffffffff8129035e at fork_trampoline+0xe
Jun 21 10:46:41	kernel		#15 0xffffffff80ca6cf0 at fork_exit+0x80
Jun 21 10:46:41	kernel		#14 0xffffffff80d3c7a2 at gtaskqueue_thread_loop+0xc2
Jun 21 10:46:41	kernel		#13 0xffffffff80d3c977 at gtaskqueue_run_locked+0xa7
Jun 21 10:46:41	kernel		#12 0xffffffff80e55bda at _task_fn_rx+0x7a
Jun 21 10:46:41	kernel		#11 0xffffffff80e5bbb9 at iflib_rxeof+0xe59
Jun 21 10:46:41	kernel		#10 0xffffffff80e3b1c9 at ether_input+0x99
Jun 21 10:46:41	kernel		#9 0xffffffff80e6022d at netisr_dispatch_src+0xad
Jun 21 10:46:41	kernel		#8 0xffffffff80e3c359 at ether_nh_input+0x349
Jun 21 10:46:41	kernel		#7 0xffffffff80e3adba at ether_demux+0x17a
Jun 21 10:46:41	kernel		#6 0xffffffff80e6039d at netisr_dispatch_src+0x21d
Jun 21 10:46:41	kernel		#5 0xffffffff80f6c227 at ip6_input+0xc77
Jun 21 10:46:41	kernel		#4 0xffffffff80f521b6 at icmp6_input+0x9e6
Jun 21 10:46:41	kernel		#3 0xffffffff80f81bf1 at nd6_na_input+0x981
Jun 21 10:46:41	kernel		#2 0xffffffff80f84fc1 at defrouter_remove+0x41
Jun 21 10:46:41	kernel		#1 0xffffffff80cea087 at _rw_wlock_cookie+0x67
Jun 21 10:46:41	kernel		#0 0xffffffff80d615e3 at witness_checkorder+0xbb3
Jun 21 10:46:41	kernel		lock order lle -> nd6 list attempted at:
Jun 21 10:46:41	kernel		#17 0xffffffff80d3c7a2 at gtaskqueue_thread_loop+0xc2
Jun 21 10:46:41	kernel		#16 0xffffffff80d3c977 at gtaskqueue_run_locked+0xa7
Jun 21 10:46:41	kernel		#15 0xffffffff80e55bda at _task_fn_rx+0x7a
Jun 21 10:46:41	kernel		#14 0xffffffff80e5bbb9 at iflib_rxeof+0xe59
Jun 21 10:46:41	kernel		#13 0xffffffff80e3b1c9 at ether_input+0x99
Jun 21 10:46:41	kernel		#12 0xffffffff80e6022d at netisr_dispatch_src+0xad
Jun 21 10:46:41	kernel		#11 0xffffffff80e3c359 at ether_nh_input+0x349
Jun 21 10:46:41	kernel		#10 0xffffffff80e3adba at ether_demux+0x17a
Jun 21 10:46:41	kernel		#9 0xffffffff80e6039d at netisr_dispatch_src+0x21d
Jun 21 10:46:41	kernel		#8 0xffffffff80f6c227 at ip6_input+0xc77
Jun 21 10:46:41	kernel		#7 0xffffffff80f52068 at icmp6_input+0x898
Jun 21 10:46:41	kernel		#6 0xffffffff80f84445 at nd6_ra_input+0x1335
Jun 21 10:46:41	kernel		#5 0xffffffff80f7d858 at nd6_cache_lladdr+0x858
Jun 21 10:46:41	kernel		#4 0xffffffff80f854ac at defrouter_select_fib+0xdc
Jun 21 10:46:41	kernel		#3 0xffffffff80f7c023 at nd6_lookup+0x83
Jun 21 10:46:41	kernel		#2 0xffffffff80f5a29b at in6_lltable_lookup+0x10b
Jun 21 10:46:41	kernel		#1 0xffffffff80cead92 at __rw_rlock_int+0x82
Jun 21 10:46:41	kernel		#0 0xffffffff80d60d2b at witness_checkorder+0x2fb
Jun 21 10:46:41	kernel		lock order nd6 list -> lle established at:
Jun 21 10:46:41	kernel		2nd 0xfffffe0021de8460 nd6 list (nd6 list, rw) @ /var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-RELENG_2_7_0/sys/netinet6/nd6_rtr.c:864
Jun 21 10:46:41	kernel		1st 0xfffff8000218c410 lle (lle, rw) @ /var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-RELENG_2_7_0/sys/net/if_llatbl.c:370
Jun 21 10:46:41	kernel		lock order reversal: