PF Sense - disconnected NIC - ELINK EVENT LOG
-
Hi all, has anyone seen this before?
My system disconnected all NICs this morning and required a hard reboot to resolve.
errors:
Mar 8 10:48:49 kernel bxe1: ELINK EVENT LOG (3)
Mar 8 10:48:50 kernel bxe1: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 8 10:48:50 kernel bxe1: ELINK EVENT LOG (3)
Mar 8 10:48:50 kernel bxe1: ELINK EVENT LOG (3)
Mar 8 10:48:51 kernel bxe1: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 8 10:48:51 kernel bxe1: ELINK EVENT LOG (3)
Mar 8 10:48:51 kernel bxe1: ELINK EVENT LOG (3)
Mar 8 10:48:52 kernel bxe1: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 8 10:48:52 kernel bxe1: ELINK EVENT LOG (3)
Mar 8 10:48:52 kernel bxe1: ELINK EVENT LOG (3)
Mar 8 10:48:53 kernel bxe1: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 8 10:48:53 kernel bxe1: ELINK EVENT LOG (3)
Mar 8 10:48:53 kernel bxe1: ELINK EVENT LOG (3)
Mar 8 10:48:54 syslogd exiting on signal 15
Mar 8 10:51:31 syslogd kernel boot file is /boot/kernel/kernel -
Hmm, that looks like something from the driver itself rather than a kernel link event that looks more like:
Mar 7 15:24:22 kernel igc2: link state changed to UP
You could try setting the loader variable:
hw.bxe.debug=1
See if that gives you more info.
Not much to go on there.
Steve
-
@stephenw10 said in PF Sense - disconnected NIC - ELINK EVENT LOG:
loader variable
I've added the variable, and this is what It spat out on booting, but its not crashed/disconnected all nics yet again yet. Will see what it does over night.
I've also added - https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#broadcom-bce-4-cards
I know these are bxe but thought i'd give it a try.
kern.ipc.nmbclusters="1000000"
hw.bce.tso_enable="0"
hw.pci.enable_msix="0"bxe0: bxe_setup_queue(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:11503) init complete
Mar 9 19:34:18 kernel bxe0: bxe_set_eth_mac(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:11673) Adding Ethernet MAC
Mar 9 19:34:18 kernel ecore_get_credit_mac (/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/ecore_sp.c,356)
Mar 9 19:34:18 kernel ecore_exe_queue_step (/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/ecore_sp.c,209)
Mar 9 19:34:18 kernel ecore_execute_vlan_mac (/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/ecore_sp.c,1974) -
On a side note I've flattened the box and reinstalled pfsense 2.6 from website and restored cfg before making loader.conf changes.
It still crashed this afternoon.
Thanks.
-
it crashed again and this is the event log.
the box is still up but both of the bxe interfaces are down.
-
ok caught the error log when it went down this morning.
Mar 10 09:25:00 sshguard 44212 Exiting on signal.
Mar 10 09:25:00 sshguard 76684 Now monitoring attacks.
Mar 10 09:29:59 kernel bxe1: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:29:59 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:29:59 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:29:59 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (1)
Mar 10 09:29:59 kernel bxe1: bxe_hw_stats_update(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe_stats.c:929) ERROR: invalid NIG timer max (4294967295)
Mar 10 09:30:00 kernel bxe0: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:30:00 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:00 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:00 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (1)
Mar 10 09:30:00 kernel bxe0: bxe_hw_stats_update(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe_stats.c:929) ERROR: invalid NIG timer max (4294967295)
Mar 10 09:30:00 kernel bxe1: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:30:00 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:00 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:01 kernel bxe0: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:30:01 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:01 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:01 kernel bxe1: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:30:01 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:01 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:02 kernel bxe0: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:30:02 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:02 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:02 kernel carp: 19@bxe1.50: BACKUP -> MASTER (master timed out)
Mar 10 09:30:02 kernel carp: 2@bxe0.998: BACKUP -> MASTER (master timed out)
Mar 10 09:30:02 kernel carp: 17@bxe1.102: BACKUP -> MASTER (master timed out)
Mar 10 09:30:02 check_reload_status 444 Carp master event
Mar 10 09:30:02 check_reload_status 444 Carp master event
Mar 10 09:30:02 check_reload_status 444 Carp master event
Mar 10 09:30:02 kernel bxe1: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:30:02 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:02 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:03 php-fpm 370 /rc.carpmaster: HA cluster member "(172.16.100.254@bxe1.50): (VPN)" has resumed CARP state "MASTER" for vhid 19
Mar 10 09:30:03 kernel bxe0: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:30:03 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:03 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:03 php-fpm 29802 /rc.carpmaster: HA cluster member "(149.11.42.4@bxe0.998): (WANCOGENT01)" has resumed CARP state "MASTER" for vhid 2
Mar 10 09:30:03 php-fpm 20201 /rc.carpmaster: HA cluster member "(154.51.67.254@bxe1.102): (PUBLICIPS)" has resumed CARP state "MASTER" for vhid 17
Mar 10 09:30:03 kernel bxe1: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:30:03 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:03 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:04 kernel bxe0: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:30:04 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:04 kernel bxe0: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:04 kernel bxe1: bxe_watchdog(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:3492) ERROR: TX watchdog timeout on fp[02], resetting!
Mar 10 09:30:04 kernel bxe1: bxe_acquire_hw_lock(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1045) ERROR: resource (0x0) in use (status 0xffffffff bit 0x1)
Mar 10 09:30:04 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:04 kernel bxe1: elink_cb_event_log(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:1865) ELINK EVENT LOG (3)
Mar 10 09:30:04 check_reload_status 444 Linkup starting bxe1
Mar 10 09:30:04 kernel bxe1: bxe_igu_int_disable(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:10488) ERROR: proper val not read from IGU!
Mar 10 09:30:04 kernel bxe1: bxe_handle_error(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/
Mar 10 09:30:04 kernel bxe1: link state changed to DOWN
Mar 10 09:30:04 kernel FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:12634) bxe1: Recovery started errors 0x1 recovery state 0x2
Mar 10 09:30:04 kernel carp: 17@bxe1.102: MASTER -> INIT (hardware interface down)
Mar 10 09:30:04 kernel carp: demoted by 240 to 240 (interface down)
Mar 10 09:30:04 kernel bxe1.102: link state changed to DOWN
Mar 10 09:30:04 kernel carp: 19@bxe1.50: MASTER -> INIT (hardware interface down)
Mar 10 09:30:04 kernel bxe1:
Mar 10 09:30:04 kernel carp: demoted by 240 to 480 (interface down)
Mar 10 09:30:04 kernel bxe_parity_attn(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/F
Mar 10 09:30:04 kernel bxe1.50: link state changed to DOWN
Mar 10 09:30:04 kernel bxe1.101: link state changed to DOWN
Mar 10 09:30:04 kernel reeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:7547) ERROR: Parity error: HW block parity attention:
Mar 10 09:30:04 kernel [0]:0x55540000 [1]:0x55555555 [2]:0x00005555 [3]:0xf0000000 [4]:0x00000028
Mar 10 09:30:04 kernel bxe1: bxe_nic_unload(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:4258) Starting NIC unload...
Mar 10 09:30:04 check_reload_status 444 Carp backup event
Mar 10 09:30:04 check_reload_status 444 Linkup starting bxe1.102
Mar 10 09:30:04 check_reload_status 444 Carp backup event
Mar 10 09:30:04 check_reload_status 444 Linkup starting bxe1.50
Mar 10 09:30:04 check_reload_status 444 Linkup starting bxe1.101 -
@dave10x said in PF Sense - disconnected NIC - ELINK EVENT LOG:
Mar 10 09:30:04 kernel reeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:7547) ERROR: Parity error: HW block parity attention:
Mar 10 09:30:04 kernel [0]:0x55540000 [1]:0x55555555 [2]:0x00005555 [3]:0xf0000000 [4]:0x00000028
Mar 10 09:30:04 kernel bxe1: bxe_nic_unload(/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/sys/dev/bxe/bxe.c:4258) Starting NIC unload...Hmm, that looks like some low level driver error or even an actual hardware issue. I'm not sure any amount of 'tuning' is going to help there.
At a practical level I would look at just swapping out the NIC with something Intel based.Steve
-
I'm going with hardware, we have an identical box in HA with this one as the failover and it hasn't had any issues.
Replacements are on the way. Thanks for the help.