Electrical Transient / Spike Damaged EM0 Nic with Hardware Offloading Enabled Had Us Believing We Were Under DDOS Attack.
-
Hello.
It's been one hell of a week, and after making sure Switches and Antenna's are working as they should, we were left thinking, are we really in the middle of a DDOS attack, and leaving CAPSA running on the network, confirmed our suspicions. We were, except...we were not.
All those ARP, NETBIOS, HTTP excess packets from this address and that address, in the umteens per second, having us run from office to office, up stairs, down stairs, across town, back and forth like asses, and not believing the malware scans that said, "hey bud, you're clean!" turns out it was
a damaged network card with Hardware checksum offloading turned on.The last night, the network just fell flat on it's face, like that's it, I'm out! We couldn't get access to Pfsense, which is headless in the server room. No choice, we connected monitor and keyboard, checked pftop and it was just going CRAZY, like whoa! We turned off ISP GPON units, turned off all connected switches, waited 10 minutes and at 2200hrs fired up everything, including Pfsense, and checked pftop. It was none stop, constant scrolling of lo 127.0.0.1 interface talking to itself, and * * *'s at first sight. Eventually we were able to log into Pfsense and navigation was barely responsive. I ended up on Hardware Checksum page and just happened to remember an issue from long ago, and decided to turn all options off, reboot, and Murder she wrote... that was the end of that.
So now, I'm wondering if there are certain race conditions that indicate a hardware fault, related to hardware offloading issues, that would allow Pfsense to fire off an alert or an email?
For instance, I remember checking Sockets, Under Diagnostics earlier in the day and saw numerous Local 127.0.0.1 connections to Foreign 127.0.0.1, then further down just 127.0.0.1 connections with question marks littered across the display field.In any case, we are changing the network card EM0 type to Intel Pro Quad type in about a week, so if I can help by providing more information, creating the conditions again with the current faulty card still installed, and screen capture the indications, let me know.
That is all. Thanks, and regards....
-
Hmm, weird issue. Do you have any example logs or pcaps from when it was happening?
-
Good morning..
No pcaps, and logs are probably still on the system.
We've had to replace the system. We've changed network card after network card, and at best this has to be static electricity damage to the motherboard. On changing network cards, no matter how many times we applied the configuration, no dice. On reboot, it's back to square one. Put back in old network cards, no problem except now even though with hardware offloading off, by next day the network is crawling to a stop, so enough milling around, We're beating a dead horse at this point. Will be ordering Netgate replacement, but in meantime, we're running CE on VM. So, I'm not sure what kind of spike or transient got through, but now it appears two switches at other locations are affected as well.
