Sporadic "Hotplug event detected" errors on different ports, only reboot fixes it

Steve 0

Dear all

I'm pulling my hair out here:
Running pfSense for a couple of years already and was very happy with it. But about two weeks ago I started to get the dreaded "Hotplug event detected" errors on my hardware (which hasn't changed at all). This means that the respective port goes down fully and doesn't come up again.

It happens sporadicly (between hours and days) and on both of my interfaces: sometimes the WAN port is affected, sometimes the LAN port.
What I find wierd is that the link comes up again (from logs: re1: link state changed to UP) but the respective services stay unreachable. Only a complete reboot fixes the issue.

I think this fact might be helpful in finding the solution and I would be very glad if I could sort this out. Again: I was using pfSense on this exact hardware without any issues and I did not change anything in the last month (running 2.5.2-RELEASE).

Thank you in advance for your help! Really appreciate it!

bmeeks

Your software has not changed either unless you specifically updated something. So that means the most logical cause of your issue is failing hardware. Could be the power supply, might even be one or more cables or connectors failing. Might be dirty contacts in an RJ45 port.

Cables can spontaneously go bad from something like slowly worsening corrosion, or perhaps an animal has chewed the insulation someplace, etc. Electrolytic capacitors can also begin to fail with age, and their failure typically shows up as random weirdness with the associated electronics.

Since you say more than one port is affected, my initial suspicion would be something either power supply or motherboard related in the firewall.

fireodo

@bmeeks said in Sporadic "Hotplug event detected" errors on different ports, only reboot fixes it:

Cables can spontaneously go bad

I had something similar these days - changing the cable resolved the issue ...

Steve 0

Thank you, you two for your help!

I really have to look at the cables then I believe: we moved and at the new house I used brand spanking new CAT 6a cables, fresh out of the packaging. I can't believe these are already bad but I will have a look.
At the same time I will clean the RJ45 ports with some compressed air.

Thanks again - I'll report back.

Steve 0

Sadly, all my troubleshooting did not help.

However, I have noticed that all the hotplug-events happen on round hours on the dot. Meaning f.e. 8:00am, 9:00am, 17:00am. This seams very strange. Is there a watchdog or a service runing every hour that might false-trigger these events?

Gertjan

@steve-0

Install the pfSense cron package, and look for yourself ;)
Or launch 'top' (command line) and look.

If all NICs trigger a disconnect connect event, I suspect a power issue.
Ethernet bus transceivers are analog devices, and quiet power hungry.
If the entire device got some jobs to do on every hour sharp, the power might get drained to much. Power bricks do die .... or become just 'just not enough any more' before dyeing.

Steve 0

A further google search has just revealed that pfSense seems to have exactly that problem with Realtek NICs, which seem to be installed in the box I'm using (re-flashed Sophos XG 85).

I will try to load the new driver this afternoon and hope for the best.

Gertjan

@steve-0 said in Sporadic "Hotplug event detected" errors on different ports, only reboot fixes it:

pfSense seems to have exactly that problem with Realtek NIC

Not pfSense. It's FreeBSD, the stock 're' drivers.
Normally, if you see "Realtek inside" you should just run. Or stick it in a Microsoft based device (I wouldn't even do that).

The driver proposed as a package seems to be better, that's true.

Steve 0

Didn't know, actually. Also I didn't really have a choice: I bought that stupid Sophos XG 85 for internal testing only to get my NFR license revoked after a year and finding out that a Sophos XG is basically worthless without a license. So I said f*ck it and installed pfSense on it (as it's basically a little UNIX machine) and was very happy that it actually worked.

Up until some weeks ago I was perfectly happy with that little capable machine - never really had a problem. Also the update from 2.4.5 to 2.5 worked flawlessly (except some unbound issues, that I was able to fix).

Looking forward to the new drivers. I hope that fixes the issue.

Steve 0

Note to everyone reading this: The problem was indeed caused by the crappy Realtek driver in FreeBSD. Loading in the realtek-re-kmod driver fixed the issue.

Sadly, once I upgraded my box to pfSense 2.6.0 the drivers where gone and I had to re-load these - this time directly from pkg.freebsd.org as instructed in this tutorial: https://www.reddit.com/r/PFSENSE/comments/t872mx/fix_issues_with_realtek_nic_on_pfsense_260