Limited throughput - 500Mbps VS 1Gbps ISP
-
Hey team
I recently upgraded to 900Mbps downstream (100Mbps Up) from my ISP and thought I'd give the ISP router (UK Vodafone THX300/vox3.0) a go and use it for a bit.
So far I've seen dropouts on the WAN and WiFi so I'm switching back to PFSense and Unifi gear!
I have configured my PFSense as follows:
WAN - PPPOE - Connects fine:Status up PPPoE up Uptime 00:21:39 IPv4 Address 90.243.x.x (MASKED BY ME) Subnet mask IPv4 255.255.255.255 Gateway IPv4 84.6.x.x (MASKED BY ME) IPv6 Link Local fe80::a35:71ff:fefe:2f3e%igb2 DNS servers 90.255.255.90 90.255.255.255 MTU 1492 In/out packets 7781124/4072274 (10.35 GiB/1.68 GiB) In/out packets (pass) 7781124/4072274 (10.35 GiB/1.68 GiB) In/out packets (block) 720/2819 (37 KiB/640 KiB) In/out errors 0/0 Collisions 0System pfSense BIOS Vendor: American Megatrends Inc. Version: R1.03 Release Date: Wed May 18 2016 Version 2.6.0-RELEASE (amd64) built on Mon Jan 31 19:57:53 UTC 2022 FreeBSD 12.3-STABLE The system is on the latest version. Version information updated at Sun Sep 18 18:54:43 UTC 2022 CPU Type Intel(R) Atom(TM) CPU C2358 @ 1.74GHz 2 CPUs: 1 package(s) x 2 core(s) AES-NI CPU Crypto: Yes (inactive) QAT Crypto: Yes (inactive) Hardware crypto Kernel PTI Enabled MDS Mitigation Inactiveigb0@pci0:0:20:0: class=0x020000 card=0x000015bb chip=0x1f418086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection I354' class = network subclass = ethernet igb1@pci0:0:20:1: class=0x020000 card=0x000015bb chip=0x1f418086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection I354' class = network subclass = ethernet igb2@pci0:0:20:2: class=0x020000 card=0x000015bb chip=0x1f418086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection I354' class = network subclass = ethernet igb3@pci0:0:20:3: class=0x020000 card=0x000015bb chip=0x1f418086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection I354' class = network subclass = ethernetThe problem here is that I can only achieve around 500Mbps downstream in a speedtest, both from an Ethernet cable (1Gb) LAN side connected client and from the PFSense shell/CLI (actually shell is lower for some reasons but nevermind!)
I checked the CPU load monitor on the dashboard whilst performing the speedtests, and I see 69% highs so it does not look like a saturated CPU to me.
FYI, testing with the ISP router, I do get the advertised speeds, so I know the circuit can achieve what I have paid for here.
I am using the same cabling that was WAN and LAN side of the ISP router when it was used, tried alternate cables also.
Can anyone advise if this is a HW limitation issue or if somehow I can further troubleshoot this problem?
Ideally, I want full speeds of course and ideally, I want to continue using PFSense!
Thanks so much.
Dave
-
It's because PPPoE forces pfSense to use a single queue on the WAN NIC and hence can only use a single CPU core. There is some tuning you can set to mitigate that to some extent but you probably won't see 1G with a C2358.
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics
Steve
-
@stephenw10 Thanks a lot. This seems promising, at least for a potential performance gain. But what exactly do I need to "tune"?
The article mentioned:
Adding a System Tunable or loader.conf.local entry for net.isr.dispatch=deferred can lead to performance gains on affected hardware. Tuning the values of net.isr.maxthreads and net.isr.numthreads may yield additional performance gains.But not how to achieve this. I will start Googling now also :)
Thanks again!
-
Start by setting net.isr.dispatch to deferred. You can do that in the system tunables page:
https://docs.netgate.com/pfsense/en/latest/config/advanced-tunables.htmlThe others are probably set as 2 by default but you can check:
[22.05-RELEASE][admin@4100-2.stevew.lan]/root: sysctl net.isr.maxthreads net.isr.maxthreads: 2 [22.05-RELEASE][admin@4100-2.stevew.lan]/root: sysctl net.isr.numthreads net.isr.numthreads: 2I would test the result of doubling them initially. Since that CPU only has 2 cores anyway it may not help.
Steve
-
@stephenw10 said in Limited throughput - 500Mbps VS 1Gbps ISP:
et.isr.numthreads
Thank a lot!
I made those changes, as well as some others as below from Googling:
I see 100Mbps improvement, ~500 --to--> ~600, so definitely an improvement, but not quite there yet. Am I now at the limit of my HW and need to upgrade? Thx!!! -
@davecullen86 said in Limited throughput - 500Mbps VS 1Gbps ISP:
Am I now at the limit of my HW and need to upgrade?
Probably. Check the output of
top -HaSPat the CLI while you're testing. See it one CPU core is at 100%.
But, yes, I would not expect to pass much more than that using PPPoE and that CPU.Steve
-
@davecullen86 said in Limited throughput - 500Mbps VS 1Gbps ISP:
need to upgrade?
have a look at https://www.netgate.com/pfsense-plus-software/how-to-buy#appliances
-
@stephenw10
Here is a video, it does not look like 100% to me. What do you think?
https://file.io/LkYuqXCiaeHm
(File too big to attach here) -
@patch in an ideal world, totally I’d buy one of those appliances.
One main point to PFsense generally for me, is the ability to use my own hardware to make the cost a lot lower.That’s to my detriment here of course as I am hitting limitations.
My options I see here are:
-
Instead of PPPoE, use the ISP router in the WAN as a L3 hop. This means double NAT as bridge mode isn’t supported.
-
Buy new PFsense HW
-
Use some other 3rd party HW on the wan side to support what I need. Maybe a Draytek 166, anyone had experience with this?
Thank you guys, all responses are very much appreciated.
Dave
-
-
@davecullen86 Netgate have both specified and bench marked their hardware.
-
The best and safest solution is to buy a Netgate appliance.
-
If that is not possible you can compare hardware you have to a similar Netgate appliances to guess how it may perform. Note in practice that is not as easy as it first appears as Netgate have balanced the performance of each part of an appliance to achieve overall performance. As a result, equalling one aspect of the specification may not translate to overall performance.
-
-
Hmm, something odd about your ssh session there making it unclear. Looks like I missed a dash from my instructions! Should have read
top -HaSP. I expect the output to look like:last pid: 41269; load averages: 0.04, 0.08, 0.08 up 54+17:57:45 12:38:48 526 threads: 3 running, 510 sleeping, 13 waiting CPU 0: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle CPU 1: 0.0% user, 0.0% nice, 0.2% system, 0.0% interrupt, 99.8% idle Mem: 15M Active, 217M Inact, 422M Wired, 3143M Free ARC: 198M Total, 30M MFU, 162M MRU, 692K Anon, 912K Header, 4358K Other 100M Compressed, 202M Uncompressed, 2.03:1 Ratio Swap: 1024M Total, 1024M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0B 32K CPU1 1 1298.4 99.65% [idle{idle: cpu1}] 11 root 155 ki31 0B 32K RUN 0 1298.6 99.46% [idle{idle: cpu0}] 0 root -92 - 0B 544K - 1 187:07 0.27% [kernel{dummynet}] 0 root -76 - 0B 544K - 1 127:16 0.16% [kernel{if_config_tqg_0}] 41269 root 20 0 14M 4772K CPU0 0 0:00 0.15% top -HaSP 21459 root 20 0 17M 7460K select 1 35:11 0.06% /usr/local/sbin/openvpn --config /var/e 12 root -60 - 0B 208K WAIT 0 21:34 0.03% [intr{swi4: clock (0)}] 19 root -16 - 0B 16K pftm 1 28:20 0.02% [pf purge] 89724 dhcpd 20 0 22M 13M select 0 0:40 0.02% /usr/local/sbin/dhcpd -user dhcpd -grou 87627 root 20 0 20M 9524K select 1 0:00 0.02% sshd: admin@pts/0 (sshd) 0 root -76 - 0B 544K - 1 12:35 0.02% [kernel{if_io_tqg_1}] 20 root -16 - 0B 16K - 1 9:37 0.01% [rand_harvestq] 0 root -76 - 0B 544K - 0 3:28 0.01% [kernel{if_io_tqg_0}] 94357 root 20 0 11M 2788K nanslp 0 0:18 0.01% /usr/local/bin/dpinger -S -r 0 -i OPENV 94054 root 20 0 11M 2780K nanslp 1 0:15 0.01% /usr/local/bin/dpinger -S -r 0 -i 6100_Where you can see exactly what the loading is on each CPU core and what's causing it.
Steve
-
The problem here is that I can only achieve around
500Mbps downstream in a speedtest, both from an
Ethernet cable (1Gb) LAN side connected client andOk, but with a 2,0GHz CPU you might be able to archive
500 MBit/s according to the pfSense "Hardware recommendations" and you archive 500 - 600 MBit/s!So all should be fine for you!
I checked the CPU load monitor on the dashboard
whilst performing the speedtests, and I see 69% highs
so it does not look like a saturated CPU to me.This can be, but with PPPoE you will be "pinned" or "nailed" to only one CPU core! And 2,0GHz is for archiving
~500 MBit/s the minimum on an actual CPU and yours is from 2013 and we are in 2022, please don´t forget this.FYI, testing with the ISP router, I do get the advertised
speeds, so I know the circuit can achieve what I have
paid for here.They (ISPs) are often soldering special ASICs on this routers and with them you will be archive full 1 GBit/s
pfSense comes as a Software firewall to you, and no one
can imagine on what hardware it will be installed by you!I am using the same cabling that was WAN and LAN
side of the ISP router when it was used, tried alternate
cables also.Cables makes the CPU not faster and also are not speeding up like an ASICs.
Can anyone advise if this is a HW limitation issue or
if somehow I can further troubleshoot this problem?They are much tuning tips out, but this is not a single I
set it up and it works thing! You should be setting up or change much more points and then you combine them
this is often the "salt in the soup". I would considering to start at Friday over the weekend to get better in touch with it.Ideally, I want full speeds of course and ideally, I want
to continue using PFSense!We all want it, not only you!!
-
Hey guys, many thanks for your response. The more I look into this, the more I see so many others with the same issue. As you say the issue is implicit to the PPPoE single core factor and the clock speed of an individual core of my small appliance.
I have a solution! With another identical appliance, I have installed OpenWRT x86 and I am not getting close to 900Mbps throughput.
Now, THIS IS good enough for me :-). So I suggest is a good potential solution for others who are happy to offload the PPPoE function to another inline appliance.
Now I just need to work out if I can pass through the WAN IP somehow to my PFSense :-)
Thanks for your help again - I really appreciate the pointers that ultimately led me to get a working solution.
Cheers - Dave -
Let us know if you find a way to do that. I've looked into it before and came to the conclusion that it might be possible but it involves some unconventional setup!
-
Hey guys, many thanks for your response. The more
I look into this, the more I see so many others with the
same issue.I have some, a couple of PC ENgines APU boards, and
I run MikroTik RouterOS, OpenWRT, pfSense on them,
all Linux comes more to 1 GBit/s with lower powered
hardware, it is a little bit more near to the hardware
due to better driver support and here and there not
so "hardware hungry", but a router and a firewall
that can be turned into a real UTM device is als not
the same! As I see it personally, you could try out as @stephenw10 was suggesting to tune your pfSense
a little here and there. With DanOS you might be getting
nearly two streams with full GBit/s on the same hardware
(PC Engines APUx), owed to DPDK capable LAN ports such
Intel i210 / i211.As you say the issue is implicit to the PPPoE single core > factor and the clock speed of an individual core of my
small appliance.Like me, but I was high up the cpu frequency to another
level and play now around with some other tuneable`s,
to get here and there more out of my hardware pointed
to the entire throughput. But I also know that my appliance is better cooled then other and will never goes
higher then 65 C° - 70 C°!!!! The CPU is normally capable
of 1400 MHz and runs even only at 600 MHz - 1000 MHz
and now it is running from 1000 MHz till 1400 MHz, but
if something goes wrong, I don´t complain and be angry!I have a solution! With another identical appliance, I
have installed OpenWRT x86 and I am not getting close > to 900Mbps throughput.
And with DanOS you may be bidirectional getting fully
1 GBit/s out! But not a fully UTM in your Network!!!!!!!!- Firewall
- Captive Portal with voucher system (voucher over sms)
- FreeRadius with certificates and encryption
- Snort or Suricata for IDS/IPS
- pfBlocker-NG for less spam and other unwanted things
- Squid & SquidGiuard as a caching proxy in fron of LAN
- ClamAV scanning the entire network flow for viruses
- (perhaps at one day WiFi a/b/g/n/ax)
Now, THIS IS good enough for me :-). So I suggest is
a good potential solution for others who are happy to
offload the PPPoE function to another inline appliance.I run a AVM FB 7590ax in front of the pfSense and behind
I am running the pfSense firewall! No PPPoE anymore, but
double NAT situation! But all CPU cores in usage!- AVM is offering some interesting APPs (VPN, telephone,..)
- Really nice to connect from outside (internet) and being secure on the LAN side!
Now I just need to work out if I can pass through
the WAN IP somehow to my PFSense :-)- 1 LAN Port as "exposed host" to the WAN interface
of the pfSense firewall ("Experienced") - Double NAT Situation
Router:
network (net) 192.168.178.0/24 (255.255.255.0)
Router IP 192.168.178.1/24 (255.255.255.0)
Static IP Address to the pfSense a.e. 192.168.178.10/24
DHCP off: all IPs will be static given to the clientspfSense:
WAN IP 192.168.178.50/24 (255.255.255.0) static IP
LAN Net: 172.xx.xx.0/24 (255.255.255.0)
LAN IP 172.xx.xx.1/24 (255.255.255.0) static IP
DHCP: on/off (Like you need it and want it)Thanks for your help again - I really appreciate the
pointers that ultimately led me to get a working
solution.Not that problem, you are one from xyz sitting in the same
boat. I would also have a look on another appliance if I`ll
getting more then 50 MBit/s Internet speed!!!P.S.
Please don´t forget in the WAN setup to disable the
following point!
