LAN interface disconnects randomly



  • Hi all,

    this is my first post here and we are experiencing the following issue periodically on our Decisio PF Firewall A10 Quad Core SSD Desktop (https://www.applianceshop.eu/security-appliances/security-appliances-desktop-and-wallmountable/pfsense-based-desktop-44/sense-a10-quad-core-ssd-desktop.html):

    Every every three to four weeks or so, our LAN interface (em1) seems to change its state to "DOWN", the log looks like this:

    Jul 11 05:40:10 check_reload_status Reloading filter
    Jul 11 05:40:10 php-fpm 94651 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Jul 11 05:39:57 check_reload_status Reloading filter
    Jul 11 05:39:57 check_reload_status updating dyndns lan
    Jul 11 05:39:54 check_reload_status Restarting ipsec tunnels
    Jul 11 05:39:54 php-fpm /rc.linkup: HOTPLUG: Configuring interface lan
    Jul 11 05:39:54 php-fpm /rc.linkup: DEVD Ethernet attached event for lan
    Jul 11 05:39:53 kernel em1: link state changed to UP
    Jul 11 05:39:53 check_reload_status Linkup starting em1
    Jul 11 05:39:50 check_reload_status Reloading filter
    Jul 11 05:39:50 php-fpm /rc.linkup: DEVD Ethernet detached event for lan
    Jul 11 05:39:49 check_reload_status Linkup starting em1
    Jul 11 05:39:49 kernel em1: link state changed to DOWN
    Jul 11 05:39:49 kernel em1: RX Next to Refresh = 0
    Jul 11 05:39:49 kernel em1: RX Next to Check = 0
    Jul 11 05:39:49 kernel em1: RX discarded packets = 0
    Jul 11 05:39:49 kernel em1: hw rdh = 0, hw rdt = 1023
    Jul 11 05:39:49 kernel em1: RX Queue 0 –----
    Jul 11 05:39:49 kernel em1: Tx Descriptors avail failure = 36
    Jul 11 05:39:49 kernel em1: TX descriptors avail = 128
    Jul 11 05:39:49 kernel em1: Tx Queue Status = -2147483648
    Jul 11 05:39:49 kernel em1: hw tdh = 0, hw tdt = 896
    Jul 11 05:39:49 kernel em1: TX Queue 0 ------
    Jul 11 05:39:49 kernel Interface is RUNNING and ACTIVE
    Jul 11 05:39:49 kernel em1: Watchdog timeout Queue[0]– resetting

    Jul 11 04:48:08 check_reload_status Reloading filter
    Jul 11 04:48:08 php-fpm 36415 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Jul 11 04:47:55 check_reload_status Reloading filter
    Jul 11 04:47:55 check_reload_status updating dyndns lan
    Jul 11 04:47:52 check_reload_status Restarting ipsec tunnels
    Jul 11 04:47:52 php-fpm /rc.linkup: HOTPLUG: Configuring interface lan
    Jul 11 04:47:52 php-fpm /rc.linkup: DEVD Ethernet attached event for lan
    Jul 11 04:47:51 kernel em1: link state changed to UP
    Jul 11 04:47:51 check_reload_status Linkup starting em1
    Jul 11 04:47:48 check_reload_status Reloading filter
    Jul 11 04:47:48 php-fpm /rc.linkup: DEVD Ethernet detached event for lan
    Jul 11 04:47:47 check_reload_status Linkup starting em1
    Jul 11 04:47:47 kernel em1: link state changed to DOWN
    Jul 11 04:47:47 kernel em1: RX Next to Refresh = 0
    Jul 11 04:47:47 kernel em1: RX Next to Check = 0
    Jul 11 04:47:47 kernel em1: RX discarded packets = 0
    Jul 11 04:47:47 kernel em1: hw rdh = 0, hw rdt = 1023
    Jul 11 04:47:47 kernel em1: RX Queue 0 ------
    Jul 11 04:47:47 kernel em1: Tx Descriptors avail failure = 36
    Jul 11 04:47:47 kernel em1: TX descriptors avail = 127
    Jul 11 04:47:47 kernel em1: Tx Queue Status = -2147483648
    Jul 11 04:47:47 kernel em1: hw tdh = 0, hw tdt = 897
    Jul 11 04:47:47 kernel em1: TX Queue 0 ------
    Jul 11 04:47:47 kernel Interface is RUNNING and ACTIVE
    Jul 11 04:47:47 kernel em1: Watchdog timeout Queue[0]– resetting

    Jul 11 03:55:56 check_reload_status Reloading filter
    Jul 11 03:55:56 php-fpm 79945 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Jul 11 03:55:43 check_reload_status Reloading filter
    Jul 11 03:55:43 check_reload_status updating dyndns lan
    Jul 11 03:55:40 check_reload_status Restarting ipsec tunnels
    Jul 11 03:55:40 php-fpm /rc.linkup: HOTPLUG: Configuring interface lan
    Jul 11 03:55:40 php-fpm /rc.linkup: DEVD Ethernet attached event for lan
    Jul 11 03:55:39 kernel em1: link state changed to UP
    Jul 11 03:55:39 check_reload_status Linkup starting em1
    Jul 11 03:55:36 check_reload_status Reloading filter
    Jul 11 03:55:36 php-fpm /rc.linkup: DEVD Ethernet detached event for lan
    Jul 11 03:55:35 check_reload_status Linkup starting em1
    Jul 11 03:55:35 kernel em1: link state changed to DOWN
    Jul 11 03:55:35 kernel em1: RX Next to Refresh = 0
    Jul 11 03:55:35 kernel em1: RX Next to Check = 0
    Jul 11 03:55:35 kernel em1: RX discarded packets = 0
    Jul 11 03:55:35 kernel em1: hw rdh = 0, hw rdt = 1023
    Jul 11 03:55:35 kernel em1: RX Queue 0 ------
    Jul 11 03:55:35 kernel em1: Tx Descriptors avail failure = 36
    Jul 11 03:55:35 kernel em1: TX descriptors avail = 127
    Jul 11 03:55:35 kernel em1: Tx Queue Status = -2147483648
    Jul 11 03:55:35 kernel em1: hw tdh = 0, hw tdt = 897
    Jul 11 03:55:35 kernel em1: TX Queue 0 ------
    Jul 11 03:55:35 kernel Interface is RUNNING and ACTIVE
    Jul 11 03:55:35 kernel em1: Watchdog timeout Queue[0]– resetting

    Jul 11 03:03:39 check_reload_status Reloading filter
    Jul 11 03:03:39 php-fpm 52235 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Jul 11 03:03:26 check_reload_status Reloading filter
    Jul 11 03:03:26 check_reload_status updating dyndns lan
    Jul 11 03:03:23 check_reload_status Restarting ipsec tunnels
    Jul 11 03:03:23 php-fpm /rc.linkup: HOTPLUG: Configuring interface lan
    Jul 11 03:03:23 php-fpm /rc.linkup: DEVD Ethernet attached event for lan
    Jul 11 03:03:22 kernel em1: link state changed to UP
    Jul 11 03:03:22 check_reload_status Linkup starting em1
    Jul 11 03:03:19 check_reload_status Reloading filter
    Jul 11 03:03:19 php-fpm /rc.linkup: DEVD Ethernet detached event for lan
    Jul 11 03:03:18 check_reload_status Linkup starting em1
    Jul 11 03:03:18 kernel em1: link state changed to DOWN
    Jul 11 03:03:18 kernel em1: RX Next to Refresh = 0
    Jul 11 03:03:18 kernel em1: RX Next to Check = 0
    Jul 11 03:03:18 kernel em1: RX discarded packets = 0
    Jul 11 03:03:18 kernel em1: hw rdh = 0, hw rdt = 1023
    Jul 11 03:03:18 kernel em1: RX Queue 0 ------
    Jul 11 03:03:18 kernel em1: Tx Descriptors avail failure = 36
    Jul 11 03:03:18 kernel em1: TX descriptors avail = 127
    Jul 11 03:03:18 kernel em1: Tx Queue Status = -2147483648
    Jul 11 03:03:18 kernel em1: hw tdh = 0, hw tdt = 897
    Jul 11 03:03:18 kernel em1: TX Queue 0 ------
    Jul 11 03:03:18 kernel Interface is RUNNING and ACTIVE
    Jul 11 03:03:18 kernel em1: Watchdog timeout Queue[0]– resetting

    Jul 11 02:11:46 check_reload_status Reloading filter
    Jul 11 02:11:46 php-fpm 97862 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Jul 11 02:11:33 check_reload_status Reloading filter
    Jul 11 02:11:33 check_reload_status updating dyndns lan
    Jul 11 02:11:29 check_reload_status Restarting ipsec tunnels
    Jul 11 02:11:29 php-fpm /rc.linkup: HOTPLUG: Configuring interface lan
    Jul 11 02:11:29 php-fpm /rc.linkup: DEVD Ethernet attached event for lan
    Jul 11 02:11:28 kernel em1: link state changed to UP
    Jul 11 02:11:28 check_reload_status Linkup starting em1
    Jul 11 02:11:25 check_reload_status Reloading filter
    Jul 11 02:11:25 php-fpm /rc.linkup: DEVD Ethernet detached event for lan
    Jul 11 02:11:24 check_reload_status Linkup starting em1
    Jul 11 02:11:24 kernel em1: link state changed to DOWN
    Jul 11 02:11:24 kernel em1: RX Next to Refresh = 0
    Jul 11 02:11:24 kernel em1: RX Next to Check = 0
    Jul 11 02:11:24 kernel em1: RX discarded packets = 0
    Jul 11 02:11:24 kernel em1: hw rdh = 0, hw rdt = 1023
    Jul 11 02:11:24 kernel em1: RX Queue 0 ------
    Jul 11 02:11:24 kernel em1: Tx Descriptors avail failure = 36
    Jul 11 02:11:24 kernel em1: TX descriptors avail = 128
    Jul 11 02:11:24 kernel em1: Tx Queue Status = -2147483648
    Jul 11 02:11:24 kernel em1: hw tdh = 0, hw tdt = 896
    Jul 11 02:11:24 kernel em1: TX Queue 0 ------
    Jul 11 02:11:24 kernel Interface is RUNNING and ACTIVE
    Jul 11 02:11:24 kernel em1: Watchdog timeout Queue[0]– resetting

    Jul 11 01:19:37 check_reload_status Reloading filter
    Jul 11 01:19:37 php-fpm 60235 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Jul 11 01:19:24 check_reload_status Reloading filter
    Jul 11 01:19:24 check_reload_status updating dyndns lan
    Jul 11 01:19:21 check_reload_status Restarting ipsec tunnels
    Jul 11 01:19:21 php-fpm /rc.linkup: HOTPLUG: Configuring interface lan
    Jul 11 01:19:21 php-fpm /rc.linkup: DEVD Ethernet attached event for lan
    Jul 11 01:19:20 kernel em1: link state changed to UP
    Jul 11 01:19:20 check_reload_status Linkup starting em1
    Jul 11 01:19:17 check_reload_status Reloading filter
    Jul 11 01:19:17 php-fpm /rc.linkup: DEVD Ethernet detached event for lan
    Jul 11 01:19:16 kernel em1: link state changed to DOWN
    Jul 11 01:19:16 kernel em1: RX Next to Refresh = 0
    Jul 11 01:19:16 kernel em1: RX Next to Check = 0
    Jul 11 01:19:16 kernel em1: RX discarded packets = 0
    Jul 11 01:19:16 kernel em1: hw rdh = 0, hw rdt = 1023
    Jul 11 01:19:16 kernel em1: RX Queue 0 ------
    Jul 11 01:19:16 kernel em1: Tx Descriptors avail failure = 36
    Jul 11 01:19:16 kernel em1: TX descriptors avail = 128
    Jul 11 01:19:16 kernel em1: Tx Queue Status = -2147483648
    Jul 11 01:19:16 kernel em1: hw tdh = 0, hw tdt = 896
    Jul 11 01:19:16 kernel em1: TX Queue 0 ------
    Jul 11 01:19:16 kernel Interface is RUNNING and ACTIVE
    Jul 11 01:19:16 kernel em1: Watchdog timeout Queue[0]– resetting
    Jul 11 01:19:16 check_reload_status Linkup starting em1

    Jul 11 00:27:30 check_reload_status Reloading filter
    Jul 11 00:27:30 php-fpm 77073 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Jul 11 00:27:17 check_reload_status Reloading filter
    Jul 11 00:27:17 check_reload_status updating dyndns lan
    Jul 11 00:27:14 check_reload_status Restarting ipsec tunnels
    Jul 11 00:27:14 php-fpm 75057 /rc.linkup: HOTPLUG: Configuring interface lan
    Jul 11 00:27:14 php-fpm 75057 /rc.linkup: DEVD Ethernet attached event for lan
    Jul 11 00:27:13 kernel em1: link state changed to UP
    Jul 11 00:27:13 check_reload_status Linkup starting em1
    Jul 11 00:27:10 check_reload_status Reloading filter
    Jul 11 00:27:10 php-fpm /rc.linkup: DEVD Ethernet detached event for lan
    Jul 11 00:27:09 check_reload_status Linkup starting em1
    Jul 11 00:27:09 kernel em1: link state changed to DOWN
    Jul 11 00:27:09 kernel em1: RX Next to Refresh = 0
    Jul 11 00:27:09 kernel em1: RX Next to Check = 0
    Jul 11 00:27:09 kernel em1: RX discarded packets = 0
    Jul 11 00:27:09 kernel em1: hw rdh = 0, hw rdt = 1023
    Jul 11 00:27:09 kernel em1: RX Queue 0 ------
    Jul 11 00:27:09 kernel em1: Tx Descriptors avail failure = 36
    Jul 11 00:27:09 kernel em1: TX descriptors avail = 128
    Jul 11 00:27:09 kernel em1: Tx Queue Status = -2147483648
    Jul 11 00:27:09 kernel em1: hw tdh = 0, hw tdt = 896
    Jul 11 00:27:09 kernel em1: TX Queue 0 ------
    Jul 11 00:27:09 kernel Interface is RUNNING and ACTIVE
    Jul 11 00:27:09 kernel em1: Watchdog timeout Queue[0]– resetting

    Jul 10 23:39:44 check_reload_status Reloading filter
    Jul 10 23:39:44 php-fpm 72602 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Jul 10 23:39:31 check_reload_status Reloading filter
    Jul 10 23:39:31 check_reload_status updating dyndns lan
    Jul 10 23:39:28 check_reload_status Restarting ipsec tunnels
    Jul 10 23:39:28 php-fpm /rc.linkup: HOTPLUG: Configuring interface lan
    Jul 10 23:39:28 php-fpm /rc.linkup: DEVD Ethernet attached event for lan
    Jul 10 23:39:27 kernel em1: link state changed to UP
    Jul 10 23:39:27 check_reload_status Linkup starting em1
    Jul 10 23:39:24 check_reload_status Reloading filter
    Jul 10 23:39:24 php-fpm /rc.linkup: DEVD Ethernet detached event for lan
    Jul 10 23:39:23 kernel em1: link state changed to DOWN
    Jul 10 23:39:23 kernel em1: RX Next to Refresh = 0
    Jul 10 23:39:23 kernel em1: RX Next to Check = 0
    Jul 10 23:39:23 kernel em1: RX discarded packets = 0
    Jul 10 23:39:23 kernel em1: hw rdh = 0, hw rdt = 1023
    Jul 10 23:39:23 kernel em1: RX Queue 0 ------
    Jul 10 23:39:23 kernel em1: Tx Descriptors avail failure = 36
    Jul 10 23:39:23 kernel em1: TX descriptors avail = 128
    Jul 10 23:39:23 kernel em1: Tx Queue Status = -2147483648
    Jul 10 23:39:23 kernel em1: hw tdh = 0, hw tdt = 896
    Jul 10 23:39:23 kernel em1: TX Queue 0 ------
    Jul 10 23:39:23 kernel Interface is RUNNING and ACTIVE
    Jul 10 23:39:23 kernel em1: Watchdog timeout Queue[0]– resetting

    Jul 10 23:39:23 check_reload_status Linkup starting em1
    Jul 10 22:52:16 check_reload_status Reloading filter
    Jul 10 22:52:16 php-fpm 74711 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
    Jul 10 22:52:03 check_reload_status Reloading filter
    Jul 10 22:52:03 check_reload_status updating dyndns lan
    Jul 10 22:52:00 check_reload_status Restarting ipsec tunnels
    Jul 10 22:52:00 php-fpm /rc.linkup: HOTPLUG: Configuring interface lan
    Jul 10 22:52:00 php-fpm /rc.linkup: DEVD Ethernet attached event for lan
    Jul 10 22:51:59 kernel em1: link state changed to UP
    Jul 10 22:51:59 check_reload_status Linkup starting em1
    Jul 10 22:51:56 check_reload_status Reloading filter
    Jul 10 22:51:56 php-fpm /rc.linkup: DEVD Ethernet detached event for lan
    Jul 10 22:51:55 check_reload_status Linkup starting em1
    Jul 10 22:51:55 kernel em1: link state changed to DOWN
    Jul 10 22:51:55 kernel em1: RX Next to Refresh = 0
    Jul 10 22:51:55 kernel em1: RX Next to Check = 0
    Jul 10 22:51:55 kernel em1: RX discarded packets = 0
    Jul 10 22:51:55 kernel em1: hw rdh = 0, hw rdt = 1023
    Jul 10 22:51:55 kernel em1: RX Queue 0 ------
    Jul 10 22:51:55 kernel em1: Tx Descriptors avail failure = 36
    Jul 10 22:51:55 kernel em1: TX descriptors avail = 127
    Jul 10 22:51:55 kernel em1: Tx Queue Status = -2147483648
    Jul 10 22:51:55 kernel em1: hw tdh = 0, hw tdt = 897
    Jul 10 22:51:55 kernel em1: TX Queue 0 ------
    Jul 10 22:51:55 kernel Interface is RUNNING and ACTIVE
    Jul 10 22:51:55 kernel em1: Watchdog timeout Queue[0]– resetting

    This obviously resets all connections on that interface but more annoyingly, after this happens the firewall also "loses" all connected devices in its ARP table and only a reboot will put the firewall back in a functional state.

    Has anyone here had similar issues? I have already tried to tick "Disable hardware checksum offload" in System > Advanced > Networking as I came across this as a potential solution for a similar issue.

    Not sure how else to describe the problem, but I'm obviously more than happy to provide more information.

    Our version of pfSense is 2.3.4, network adapter on the device is: Intel(R) PRO/1000 Network Connection 7.6.1-k.

    Any help is very much appreciated and thanks in advance for any info.

    Thanks.

    Flo


Log in to reply