Poor WAN Performance Between Reboots
-
Hello,
I'm hoping someone can help me troubleshoot an issue with my WAN speeds. I'm using a beelink mini PC to run pfSense, it has an intel N100, dual 2.5 intel Ethernet, 16GB of RAM and a 512GB m.2 SSD. It is running the latest pfSense ver 2.7.2. This is my only router/firewall and I've been using it for about a year without any issues until this week. My network consists of a mix of 1/2.5/10G switches and devices. I have a fairly straight-forward setup without lots of vlans or multiple subnets.
I recently upgraded my internet package from 940mbs synchronous to 3gbs up and down. This upgrade came with some new hardware from my ISP but that has been placed into bridge mode which effectively makes my ISP's all-in-one device into a ONT. My IPv4 config type is PPPOE and those credentials are unchanged with this new internet package. I have IPv6 disabled. My WAN MTU is set at 1492 which I believe is correct for PPPOE.
Running speed test using speedtest-cli from within pfsense and also from my wired client devices give similar speeds of around 2200 up and 2200 down which is right around what I was expecting from a router with only 2.5GB NICS, but my problem is that I only get these speeds for about an hour or two. After my pfSense has been running for a time I am only able to achieve just under 1gbs (around 930bms up and down) regardless of which site or device I use to test. The speeds on my LAN are able to saturate my full 2.5gbs no matter the uptime of my pfSense and those LAN speeds stay fairly consistent even if I intentionally try to overload my network.
I'm wondering if my beelink mini PC is suffering from some kind of hardware issue such as a faulty/buggy NIC or maybe there is some pfSense config or tuning that I may have overlooked. This may have been an issue with my beelink mini PC all along and it's possible that the issue is only presenting itself now that my WAN speeds need to exceed 1gbs.
I am fairly technical but networking is not my specialty and it is one my least favourite things to deal with. I typically 'set and forget' when it comes to my network and I really haven't done a deep dive of pfSense in over a year. I'm hoping the community would have some suggestions or further troubleshooting steps for me to follow.
I appreciate any and all input,
TJ -
@tjs4ever if you can reach 2.2G up/down that would indicate that your pfSense box is capable enough. Are you connecting to the same Speedtest server every time (does your provider have one)?
Can you show a diagram of how your network looks. You write it's fairly simple but it does involve 1, 2.5 and 10G switches.
ISP modem(10G or 2.5G?) -> 2.5G(WAN):belink:(LAN)2.5G -> ?
930mbps is around what you can expect from a 1Gbit link, that is a bit strange.
-
@patient0 thanks for your reply :)
(some edits because I missed one of your questions)
I am letting speedtest.net auto-select my server each time when I test on my desktop PC. When I speedtest in pfsense one of the available hosts is VERY close to me and I left that server manually selected.
Under normal circumstances my traffic would exit my network through NordVPN which I have configured as a wireguard gateway in pfSense. Some of my devices are exempted from using Nord via firewall rules, I am only speed testing on devices that use my normal ISP gateway. When I'm getting full WAN speeds the devices that use Nord are only ~20% slower and I get aprox 1800 up and down. When I'm getting the slower WAN speeds Nord gives me 700-800 mbs.
I'm not sure if I'm clever enough to do a diagram but I can list out my network equipment and how they are all connected.
-
Bell Canada ISP 'gigahub' in bridge mode, it has multiple ports but only one of them is labeled as 10GB, I believe the others are all 1GB. The 10GB port is connected to the 2.5G WAN on my beelink. All of my cabling is either cat6 or SFP+. PfSense does not report the speed of the WAN interface, I think because it is PPPOE.
-
Beelink LAN is connected to a 2.5G switch.
-
There are 3x 2.5G client devices on that same 2.5G switch : 2x Win11 desktops that have intel NIC and 1x ubiquiti wireless AP that has a 2.5G port. I am using a POE injector to power the ubiquiti AP.
-
The 2.5G switch has a single 10GB SFP+ and that is connected to a HP Aruba managed switch. The HP Aruba has 1GB ethernet ports and 10GB SFP+
-
HP Aruba switch has two HP servers each connected via single 10GB SFP+, the other ports on this HP switch are 1GB, assorted slower client devices are all connected to the 1GB ports on the HP aruba (printer, TV tuner, audio receiver, smart TVs, etc)
-
The entire home is wired with cat6 and they all converge into a single patch panel and I patch either to the 1GB switch or the 2.5GB switch depending on the speed of the client device.
-
A small 1GB POE switch is patched from a 1GB port on the HP Aruba to another floor of the house. There is 1x IP camera and 1x ubiquiti wireless AP connected to that POE switch.
The only vlan that I am tagging is vlan 35 which was needed to get the PPPOE session with my ISP working. I do not have any LACP setup anywhere and I do not have any network loops.
-
-
@tjs4ever said in Poor WAN Performance Between Reboots:
I am letting speedtest.net auto-select my server each time when I test on my desktop PC. When I speedtest in pfsense one of the available hosts is VERY close to me and I left that server manually selected.
Do you see the same results if you test from one of the 10GBit connected servers?
I do use speedtest-go and always select the same server, otherwise you are introducing another variable (not sure you can select a server with the standard speedtest client). My ISP provides 2 and an ISP customer connected with 25Gbit provides one, which are excellent options for me. And I never test from pfSense since the test puts stress on pfSense which may lower the speedtest results.
You could:
- check with
top -HaSPhow hard your router is working while the speedtest is running. Any noticeable difference between a right-after-reboot and later speedtest? - can you check that the link speed between your modem and pfSense (
ifconfig ...from the console/SSH) is still at 2500? - is it possible that the ISP has not switched your profile and after reboot (with a new IP?) the ISP bandwidth limitation are not yet in place? But get applied after some traffic goes through?
- Have you done some test with the ISP all-in-one device in router mode?
- check with
-
@patient0 when I woke up this morning my WAN was completely down and would not come back online until I rebooted pfSense. The timing of this issue seems very coincidental as my new internet package was setup over the weekend. I've already spent many hours troubleshooting what I thought was only a speed issue with my ISP. The ISP only offers very basic support since I am using my own equipment - basically if they get the green light from their call centre to the modem that is the end of their troubleshooting. At this point I am at a total loss on what to do next.
-
@tjs4ever Have you checked the WAN interface for errors (Status->Interfaces)?
-
@WN1X no errors at this moment but my last reboot was around 40 minutes ago. I will check the status the next time it cuts out.
-
Check the actual link state of each NIC after some time in Status > Interfaces.
Check the CPU temperature and the current CPU clock speed. Those N100 platforms are known to have strangely behaving power management with anything but Windows. There are a few threads here details users efforts to make them behave rationally.
-
@stephenw10 I appreciate all of these helpful replies.
When I tax the beelink by doing a speedtest-cli within pfSense the CPU reports 2923mhz current and 806mhz max. CPU temp is currently 59C which is around 25 degrees above ambient. Memory usage is very low at 6%. The power mgmt in the beelink BIOS is set to whatever was decided at the factory, let me know if you think I should enable high performance mode or increase the TDP within the BIOS. I have reached out to beelink to ask about a BIOS update and am still waiting on a response.
I called my ISP again this morning, I suspected there was something 'off' about their modem's bridge mode. The tech unbridged the modem, factory reset it, he disabled the wifi, dhcp and all the other services. From his end he left the modem in an unactivated state : if you look at the modem now it is asking me to go online to activate but in this un-activated status my PPPOE in pfSense has been rock solid for the past 10 hours! The tech was calling this 'unofficial bridge mode'. I'm crossing my fingers that this was an issue with the modem all along but I'm going to wait a week or so before I consider this case-closed.
I don't believe I have any double-nat issues with the modem in this state but I honestly don't know how I can prove that.
As of this writing fast.com is saying that I am getting 2.7gbs down and 2.3gbs up, speedtest reports that I am getting 2316 and 2292 - I am perfectly happy with these speeds.
Thank you all for your quick and helpful replies, as one final question - would I be able to squeeze any additional performance out of this beelink by purchasing a pfplus license?
-
@tjs4ever said in Poor WAN Performance Between Reboots:
I don't believe I have any double-nat issues with the modem in this state but I honestly don't know how I can prove that.
Take a look at your assigned WAN address. Is it an RFC1918 address or is it public? RFC1918 would indicate you are double-nat'ed. A public address and you are good to go.
-
@WN1X The WAN interface IPv4 IP is the same IP address that I get when I visit https://whatismyipaddress.com/ using any of my (Nord exempt) client devices, so I take this to be correct then.
There's also the Gateway IPv4 address but it is a different value and I cannot ping it. I think this is yet another quirk with my ISP. A while back I had to change my gateway monitoring to use quad-8 for the monitoring IP - it just didn't ping one day.
-
@tjs4ever No double-nat there. You should be good now.
-
This post is deleted! -
You might be able to get more performance by tweaking the power/thermal management settings in the BIOS. Check the threads for N100 devices.
Since you're using PPPoE you should also get better performance by using the new if_pppoe driver in 2.8-beta.
-
So my first order of business this morning was to run some new speedtests and I am back to gigabit speeds on the WAN.
I checked ifconfig and the interface is connected as 1000baseT
It looks like I'm back to square 1. I'm going to swap out the cable that connects the modem to my pfSense to rule out a bad cable.
-
What NIC type are you using to connect the WAN?
You can probably set to negotiate the link at 2.5G only.
-
@stephenw10 the beelink is using dual Intel I225-V
Where can I hard-code the link speed for the WAN interface? I see that option under LAN but not under WAN settings.
-
@tjs4ever Interfaces->PORT1WAN Speed & Duplex.
-
@WN1X no such setting available under WAN, is it because it's PPPOE?
LAN has it.
The other interfaces : WAN and the two interfaces setup for my Nord have no such option.
-
Assign the parent NIC as a new interface and set it there. You can leave it as IP types none or set it to the modem/ONT subnet if you want to access that.
