Network Issues/Timeouts/Delays After Upgrading from 2.8.1 to 2.9.0
-
Hi network folks, it's been a while :)
I did an in-place upgrade from CE 2.8.1 to 2.9.0 last night and then I spent about 10 hours straight trying to figure out what went wrong. I’ve completely wracked my brain and ClaudeAI and Gemini haven’t been able to help me resolve, either. I’m hoping the real flesh-and-blood experts on here can put the robos to shame. I’ve tried various MTU and MSS values and everything else that I could think of, plus a bunch of useless and seemingly unrelated changes as per my AI-assistants. We are totally out of ideas.
First, the upgrade went perfectly smooth, no gui lockouts or BSOD or any of the more serious issues I’ve been seeing on here. My packages reinstalled themselves correctly, all settings seem to have been preserved and I don’t have any obvious issues like services being down. I’ve also rebooted all my network gear for good measure.
The hardware is a beelink eq12 mini PC with intel N100, dual 2.5gbe and 16gb ram. I have a few packages installed: pfblocker, Suricata (bound to LAN in LEGACY mode) and wireguard (more on this later). I’ve disabled pfblocker and Suricata temporarily to simplify this troubleshooting. As I write this I’m waiting for a BIOS update from beelink but their process is slow and inefficient – I need to submit a photo of the device that includes the serial # and then wait for someone to DM me the bios file. Totally jank, but I digress…
My setup isn’t too complicated, I have two gateways – one is the normal ISP-provided WAN PPPoE gateway and the other gateway is a wireguard tunnel that connects to NORD VPN. I use LAN firewall rules to explicitly define which internal devices use which gateway, with wireguard/nord being the default unless I manually make an exception. There’s another floating firewall rule that drops outbound-tagged packets if they somehow try to use the wrong gateway. Only 2-3 devices go out through the normal WAN and those devices are 100% solid and stable and I consistently get 2300 UL/DL and 0% packet loss. My issue is only on devices that use the nord/wireguard.
Immediately after the upgrade my devices using nord/wireguard gateway are getting timeouts when browsing popular websites, I’m getting errors like “Error Code: PR_END_OF_FILE_ERROR”. Some sites are trying to TLS handshake over and over forever. I don’t seem to have any packet loss and I experimented with setting very conservative MTU and MSS values for the wireguard gateway but it doesn’t have any effect. ClaudeAI asked for a pcap and the analysis was that there were many ~1 second delayed data bursts which cause my browser to reset the connection. I have not made any other recent changes other than updating pfsense to 2.9.0. This setup with this same exact hardware has been running flawlessly for about 16 months. I did create a pre-upgrade config backup when I was still on 2.8.1, so as a last resort I am prepared to reinstall the older pfsense and restore my config – but I do like to keep myself and my gear updated.
Unrelated to the more immediate issue at hand, but I’m wanting to move away from beelink entirely and buy a used Lenovo sff pc with an i5-8500. The pci-e slot would be nice and I can add a dual-port 10gb nic and get my actual 3gbs that I’m paying my ISP for. Any thoughts on that? Beelink’s process of getting firmware and drivers is a 101% turn-off for me. At least Lenovo publicly posts such files.
Anywhoo, I’m hoping someone here can help me after I’ve been going in circles all day and night with AI. I am more than happy and patient to try anything and provide any additional details.
Thanks in advance,
~TJ -
Tried playing around with System > Advanced > Networking > Checksum Offloading/LRO/TSO options? That EQ12 has Intel i225 NICs that should be pretty stable. But doesn't hurt to experiment with that.
Sorry I don't have much more to offer right now, but I'm going to follow this thread. Guessing PPPoE might have something to do with this. I know 2.9.0 had a number of PPP-related fixes, and I've seen some 2.9 posts mentioning weird behavior.
-
@luckman212
All good ideas but I already implemented these a while back when I first got into suricata; I did double-check that the update didn't enable hw acceleration. I wish I had another VPN provider that I could try - maybe the issue is with nord, but I got sucked into one of their 3-year promos and I'm right in the middle of that. -
If you think it's WireGuard specifically, Nord supports OpenVPN too, and with DCO it should be about as fast and pretty CPU-friendly. Could be an easier test than a full rollback to 2.8.1.
-
Yeah testing an OpenVPN connection to Nord would be a good test.
I'm not aware of any wireguard issues in 2.9.0 that present like that.
-
Chiming in just to say that I agree, the entire OP seems to distill down to a WireGuard configuration issue. (And the fact that traffic had been traversing the tunnel on 2.8.1 but apparently not on 2.9.0 is a red herring.)
-
@luckman212
good idea, I have a open_vpn already configured for nord, I still have the same issue and captured another pcap on that interface... below is what Gemini had to say about the new PCAP, I haven't taken any action yet:Because the exact same ~1-second delay and data bursts occur on: Native WAN (WireGuard)OpenVPN over NordMulti-device LAN traffic (Plex, Ring, Hue, CIRA DNS) The issue is entirely a software/kernel regression introduced in pfSense 2.9.0 / FreeBSD 14. Immediate Checklist to Eliminate the 1-Second Stalls -Disable PowerD & Deep C-States: Go to System > Advanced > Miscellaneous. Under Power Savings, set PowerD to Disabled. (In FreeBSD 14, deep CPU sleep modes prevent the NIC from interrupting the CPU until the 1Hz OS system timer fires). pfSense 2.9.0 contains a Dummynet timer quantum bug that defaults to 1Hz (1 second). -Disable Gateway Monitoring on VPN Interfaces: Go to System > Routing > Gateways. Edit your NordVPN OpenVPN/WireGuard gateway and check Disable Gateway Monitoring. -Disable Energy Efficient Ethernet (EEE) on Intel igc NICs: Go to System > Advanced > System Tunables. Add dev.igc.0.eee = 0 and dev.igc.1.eee = 0. Save, apply, and reboot pfSense. -
OK that is claiming you're seeing delays on native WAN as well as over VPN which is not what you said above. Are you actually seeing that or is it an AI hallucination?
If you really are seeing it on WAN too then, yes, checking any power saving settings. An i5 8500 would support speedshift and pfSense will use it by default.
-
No it's Gemini smoking crack, I didn't even give it any WAN details or pcaps because there was no reason to troubleshoot a working gateway. It totally made that part up. I think my issue has since resolved on its own; I gave up, took a day off and the magic elves must've taken over while I was asleep. Not the first I've chased software ghosts like this.
Totally unrelated to anything but I bought a barebones MINISFORUM MS-A2 with Ryzen 7 7745HX to replace the previous beelink eq12. It seems like a much better device overall and there are dual intel X710 ports onboard. I'm going to of course make a backup first but I plan to just reuse the ram and ssd from the beelink so I'm hoping it's as simple as one-time updating the interface assignments and then off to the races on the new box.
-
Yup you should just be able to reassign the NICs.
Privacy Policy · Cookie Policy