Netgate 6100 dropping LAN
-
I have a Netgate 6100, and it seems every few days to weeks, the LAN connection drops and resets itself a few seconds later.
This hasn’t been a huge issue so far as it’s always happened in the middle of the night (~3am), but it does cause the Avahi service to stop working properly until I restart it. This results in all HomeKit accessories (which are on a separate VLAN) from responding to devices on the main LAN.
Below are the log entries. It seems like when pfSense goes to reload the filter, everything blows up, but I’m not sure if that’s due to a hardware issue with the box or a software glitch within pfSense.
Nov 10 03:30:08 check_reload_status 392 Reloading filter Nov 10 03:30:08 check_reload_status 392 updating dyndns lan Nov 10 03:30:04 php-fpm 94529 /rc.newwanip: rc.newwanip: on (IP address: 10.5.3.1) (interface: OPT_GUEST[opt8]) (real interface: igc0.30). Nov 10 03:30:04 php-fpm 94529 /rc.newwanip: rc.newwanip: Info: starting on igc0.30. Nov 10 03:30:04 check_reload_status 392 Reloading filter Nov 10 03:30:04 php-fpm 57173 /rc.newwanip: rc.newwanip: on (IP address: 10.5.4.1) (interface: OPT_GAMING[opt12]) (real interface: igc0.40). Nov 10 03:30:04 php-fpm 16441 /rc.newwanip: rc.newwanip: on (IP address: 10.5.2.1) (interface: OPT_IOT[opt7]) (real interface: igc0.20). Nov 10 03:30:04 php-fpm 57173 /rc.newwanip: rc.newwanip: Info: starting on igc0.40. Nov 10 03:30:04 php-fpm 16441 /rc.newwanip: rc.newwanip: Info: starting on igc0.20. Nov 10 03:30:04 php-fpm 46810 /rc.newwanip: rc.newwanip: on (IP address: 10.5.10.1) (interface: OPT_PDF[opt10]) (real interface: igc0.100). Nov 10 03:30:04 php-fpm 94529 /rc.newwanip: rc.newwanip: on (IP address: 10.5.5.1) (interface: OPT_DMZ[opt9]) (real interface: igc0.50). Nov 10 03:30:04 php-fpm 46810 /rc.newwanip: rc.newwanip: Info: starting on igc0.100. Nov 10 03:30:04 php-fpm 94529 /rc.newwanip: rc.newwanip: Info: starting on igc0.50. Nov 10 03:30:03 check_reload_status 392 Restarting ipsec tunnels Nov 10 03:30:03 php-fpm 50468 /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. '' Nov 10 03:30:03 check_reload_status 392 rc.newwanip starting igc0.30 Nov 10 03:30:03 check_reload_status 392 rc.newwanip starting igc0.40 Nov 10 03:30:03 php-fpm 57173 /rc.linkup: Hotplug event detected for OPT_GUEST(opt8) static IP (10.5.3.1 ) Nov 10 03:30:03 check_reload_status 392 rc.newwanip starting igc0.20 Nov 10 03:30:03 check_reload_status 392 Reloading filter Nov 10 03:30:03 php-fpm 44389 /rc.linkup: Hotplug event detected for OPT_GAMING(opt12) static IP (10.5.4.1 ) Nov 10 03:30:03 check_reload_status 392 Reloading filter Nov 10 03:30:03 check_reload_status 392 rc.newwanip starting igc0.100 Nov 10 03:30:03 check_reload_status 392 rc.newwanip starting igc0.50 Nov 10 03:30:03 avahi-daemon 82979 New relevant interface igc0.IPv4 for mDNS. Nov 10 03:30:03 avahi-daemon 82979 Joining mDNS multicast group on interface igc0.IPv4 with address 10.5.0.1. Nov 10 03:30:03 php-fpm 10132 /rc.linkup: Hotplug event detected for OPT_IOT(opt7) static IP (10.5.2.1 ) Nov 10 03:30:03 php-fpm 50468 /rc.linkup: HOTPLUG: Configuring interface lan Nov 10 03:30:03 php-fpm 50468 /rc.linkup: DEVD Ethernet attached event for lan Nov 10 03:30:03 php-fpm 46810 /rc.linkup: Hotplug event detected for OPT_PDF(opt10) static IP (10.5.10.1 ) Nov 10 03:30:03 php-fpm 94529 /rc.linkup: Hotplug event detected for OPT_DMZ(opt9) static IP (10.5.5.1 ) Nov 10 03:30:02 check_reload_status 392 Linkup starting igc0.30 Nov 10 03:30:02 check_reload_status 392 Linkup starting igc0.40 Nov 10 03:30:02 check_reload_status 392 Linkup starting igc0.20 Nov 10 03:30:02 check_reload_status 392 Linkup starting igc0.100 Nov 10 03:30:02 check_reload_status 392 Linkup starting igc0.50 Nov 10 03:30:02 kernel igc0.30: link state changed to UP Nov 10 03:30:02 kernel igc0.40: link state changed to UP Nov 10 03:30:02 kernel igc0.20: link state changed to UP Nov 10 03:30:02 kernel igc0.100: link state changed to UP Nov 10 03:30:02 kernel igc0.50: link state changed to UP Nov 10 03:30:02 kernel igc0: link state changed to UP Nov 10 03:30:02 check_reload_status 392 Linkup starting igc0 Nov 10 03:30:00 check_reload_status 392 Reloading filter Nov 10 03:30:00 check_reload_status 392 Reloading filter Nov 10 03:30:00 php-fpm 10132 /rc.linkup: Hotplug event detected for OPT_GUEST(opt8) static IP (10.5.3.1 ) Nov 10 03:30:00 php-fpm 46810 /rc.linkup: Hotplug event detected for OPT_GAMING(opt12) static IP (10.5.4.1 ) Nov 10 03:30:00 avahi-daemon 82979 Interface igc0.IPv4 no longer relevant for mDNS. Nov 10 03:30:00 avahi-daemon 82979 Leaving mDNS multicast group on interface igc0.IPv4 with address 10.5.0.1. Nov 10 03:30:00 syslogd sendto: Network is down Nov 10 03:30:00 php-fpm 94529 /rc.linkup: Hotplug event detected for OPT_IOT(opt7) static IP (10.5.2.1 ) Nov 10 03:30:00 syslogd sendto: Network is down Nov 10 03:30:00 php-fpm 50468 /rc.linkup: Hotplug event detected for OPT_DMZ(opt9) static IP (10.5.5.1 ) Nov 10 03:30:00 syslogd sendto: Network is down Nov 10 03:30:00 php-fpm 57173 /rc.linkup: Hotplug event detected for OPT_PDF(opt10) static IP (10.5.10.1 ) Nov 10 03:30:00 syslogd sendto: Network is down Nov 10 03:30:00 php-fpm 16441 /rc.linkup: DEVD Ethernet detached event for lan Nov 10 03:30:00 syslogd sendto: Network is down Nov 10 03:30:00 syslogd sendto: Network is down Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 check_reload_status 392 Linkup starting igc0.30 Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 check_reload_status 392 Linkup starting igc0.40 Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 check_reload_status 392 Linkup starting igc0.20 Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 check_reload_status 392 Linkup starting igc0.100 Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 check_reload_status 392 Linkup starting igc0.50 Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 kernel igc0.30: link state changed to DOWN Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 kernel igc0.40: link state changed to DOWN Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 kernel igc0.20: link state changed to DOWN Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 kernel igc0.100: link state changed to DOWN Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 kernel igc0.50: link state changed to DOWN Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 kernel igc0: link state changed to DOWN Nov 10 03:29:59 syslogd sendto: Network is down Nov 10 03:29:59 check_reload_status 392 Linkup starting igc0 -
@rpungello said in Netgate 6100 dropping LAN:
link state changed to DOWN
Presumably a cable didn't get unplugged at 3:30 am, but that's what it looks like. Did you try a different cable? Different switch port?
When an interface is reconnected it triggers a handful of things, in case there is a new IP.
-
Is that the first part of that log sequence?:
Nov 10 03:29:59 check_reload_status 392 Linkup starting igc0What is igc connected to?
Do you have gateways defined on all those VLANs? I would not expect it to run rc.newwanip otherwise.
Steve
-
@steveits said in Netgate 6100 dropping LAN:
@rpungello said in Netgate 6100 dropping LAN:
link state changed to DOWN
Presumably a cable didn't get unplugged at 3:30 am, but that's what it looks like. Did you try a different cable? Different switch port?
When an interface is reconnected it triggers a handful of things, in case there is a new IP.
Correct, nothing was happening at 3:30am. Bad cable was my first though, but I replaced the cable after the first time it happened and it still happens every few days. I guess I’ll try a different port on the switch it’s connected to next.
-
@stephenw10 said in Netgate 6100 dropping LAN:
Is that the first part of that log sequence?:
Nov 10 03:29:59 check_reload_status 392 Linkup starting igc0What is igc connected to?
Do you have gateways defined on all those VLANs? I would not expect it to run rc.newwanip otherwise.
Steve
Yes, that’s the first log entry in over two hours.
igc0 is the LAN interface, which has a bunch of VLANs defined as well. There is an OpenVPN server running in pfSense as well, which I guess defines a gateway. Not sure if that counts as being on the LAN port (igc0) or WAN (ix3).
It’s physically connected to a Ubiquiti 10G Flex XG switch (so link running at 2.5G).
-
@rpungello Is there anything in the logs on the Ubiquiti around that time? To me these logs are saying the 6100 has detected link down/up events. That can happen from either end. If you have another switch, what about plugging it in between the 6100 and the Ubiquiti? That should give the 6100 a constant link and the Ubiquiti could flap all it wants.
-
@mer said in Netgate 6100 dropping LAN:
@rpungello Is there anything in the logs on the Ubiquiti around that time? To me these logs are saying the 6100 has detected link down/up events. That can happen from either end. If you have another switch, what about plugging it in between the 6100 and the Ubiquiti? That should give the 6100 a constant link and the Ubiquiti could flap all it wants.
No, radio silence from the Flex XG. Guess I’ll try plugging the Netgate box into the other (1G) switch, at least for now.
What’s interesting is the UniFi software doesn’t show a dropped connection to the Netgate. It thinks it’s been online for almost two months, but this has happened a few times in that timeframe.
-
Hmm, pretty much the only thing that can down/up an interface like that (other than it actually reconnecting) is if you gave Snort or Suricata running in in-line mode on the parent NIC. If the service is restarted it will appear like that.
It's still odd it shows rc.newwanip run for each vlan though without a gateway defined on them.Steve
-
@stephenw10 said in Netgate 6100 dropping LAN:
Hmm, pretty much the only thing that can down/up an interface like that (other than it actually reconnecting) is if you gave Snort or Suricata running in in-line mode on the parent NIC. If the service is restarted it will appear like that.
It's still odd it shows rc.newwanip run for each vlan though without a gateway defined on them.Steve
I'm pretty sure I don't have any extraneous gateways defined. Here's what shows up under Status > Gateways
No IDS/IPS running either.
-
Hmm, OK that looks fine.
Are you able to trigger that same action manually at all? From Status > Filter Reload for example?
-
On your dashboard are there any mentions of crashing / dumps or do you just see it drop and reconnect without any crash issue?
-
@nikolaosinlight said in Netgate 6100 dropping LAN:
On your dashboard are there any mentions of crashing / dumps or do you just see it drop and reconnect without any crash issue?
Nothing shows up on the pfSense dashboard, it just reconnects with almost no issues. The only issue is the Avahi service stops working when this happens until I restart it, which means HomeKit devices on the IoT VLAN stop responding. This is the only reason I even realized this is happening, as so far the disconnects have only been in the middle of the night, so I didn’t notice anything performance-wise.
-
Does the Avahi service actually stop? If so a possible workaround would be using the service watchdog package.
Steve
-
@stephenw10 said in Netgate 6100 dropping LAN:
Does the Avahi service actually stop? If so a possible workaround would be using the service watchdog package.
Steve
It doesn’t seem to stop, or if it does, it doesn’t seem to stay stopped as the icon in Status > Services shows a green check when I’ve gone in to restart it.
