Network Stability Issues
We have a pfSense firewall in our office that we built using an Asrock Rack AD2550R/U3S3 (with an embedded Atom D2550), a 4GB stick of SODIMM, an Inwin 1U rackmount case, and an 8GB pen drive permanently inserted into a USB header on the motherboard. It's NOT an embedded setup, but rather pfSense was installed using a memstick, specifying the USB drive as the target during installation. It's run great for years, with not one single issue, ever. We have also built several identical ones and deployed them in customer offices, also with zero complaints.
The one in our office started randomly dropping packets recently. Pinging an external IP address (such as google DNS) from the console or the webconfigurator usually yields anywhere from a 20% to 80% packet loss. This makes the equipment in our office extremely fickle. It's always a hit or miss affair if a web site will load on the first try, with several seconds of praying while the browser struggles to resolve the page's address. Alexa will answer on one attempt, then complain about being offline 5 seconds later. Our Gb internet circuit seems to top out at 500Mb/s on its best day, wheras 800-900Mb/s was pretty standard before.
At first I blamed the ISP, had them run a new wire from the nest to our demarc, test signals, etc. Ultimately, we were able to bypass the pfSense and the rest of the network, connect a PC directly to the ISP-provided Hitron CDA3-35, and get perfect results 100% of the time. Times on the pings were also 1/3 to 1/4 as long as the successful ones performed by the pfSense. Something is up with it.
I hoped it would be something as simple as the patch cable between the pfSense and the modem, so we tried replacing that. Same result.
It seemed unlikely, but I thought maybe the USB2 interface, combined with NAND aging, was causing issues with the pen drive. We pulled it out, installed two 2.5" drives, and reinstalled pfSense using ZFS. No joy.
The CPU on the Asrock motherboard is not actively cooled, and the fans in the Inwin case are predictably LOUD (thus unplugged). The firewall was mounted in the rack directly below a managed switch. Thinking maybe heat was the issue, we plugged the noisy little fans into the motherboard and moved the firewall down a slot. There was an immediate improvement in temps, but the network issues remain.
Now I'm at a loss what to check next, and working is almost impossible. This HAS to be fixed very quickly. What should we try next?
chpalmer last edited by
While I can certainly appreciate a modem having a bad reputation, it is ISP provided and they aren't particularly friendly to BYOD customers (although they can be pushed).
At any rate, the modem has been eliminated as the source of the problem. Not only is it less than 3 months old, the problems completely disappear when a workstation is connected directly to the modem (as I said in the OP).
It isn't the spark plugs.
I assume you don't see packet loss internally?
Try re-assigning the NICs. Does the fault follow the physical port or remain on the WAN?
Yes, we tried swapping the interfaces with the same result.
This configuration is kind of a "test bed" for various pet projects before they are put into production at a customer site, so I started wondering if there was just a borked configuration buried somewhere. Instead of restoring the config from backup, we reset pfSense back to factory, reconfigured the interfaces, and away it went! All problems cleared up.
Now for the tedium of rewriting all of the NAT/Firewall rules, enabling/configuring services, etc. but at least we can get work done again!
chpalmer last edited by chpalmer
It isn't the spark plugs.
Sorry- I have allot of bad experience with the Hitron/Linksys models and pushy ISP's and it showed.. Thus we at our company here will walk off a job if the customer will not change them out first..
I would ask though if you have any UDP traffic that would show up on your firewall by plugging it in such as VPN's or a VOIP system of some sort. UDP traffic will make that modem fail this way if the firmware has not been updated which many have not.. Who is your ISP?
The case of your modem is also extremely flammable compared to others so no smoking around it.. :)
That Arris modem that the author of the badmodems site burned had to be "aided" with rubbing alcohol to get it to burn. My Linksys CM3024 lit up with a single match.
It's in a well-ventilated 42U rack with a dedicated 8k BTU portable unit, so I'm not worried about heat, but your comment about UDP has me curious.
We have an Obi single-line VoIP adapter for our fax, an OVPN server VM (but the clients only route DNS queries to it for ad and content filtering so it doesn't see much traffic), and a Lubuntu VM that uses an around-the-clock PIA connection. Maybe I should check on that firmware version.
The ISP is Suddenlink (now party of ucky-yucky Altice) and they are notorious for providing modems with antiquated firmware and stubbornly refusing to update them when asked. In my experience with the company and hundreds of their customers, my best hope is that the relatively new modem shipped from the factory with a current firmware. If it's old and causes problems, we're jacked without twisting Suddenlink's arm into allowing us to BYOD.
I will say that since 2009 when I was first introduced to PFsense, the few issues I've had have all been due to failing hardware... either a bad NIC or a failing MB in a PC. This may or may not be your case, but just sharing what I've experienced.
A factory reset obviously worked for you in this case, however, you're also running discontinued nettop/desktop hardware. You may want to consider moving to new hardware going forward.