Limited throughput - 500Mbps VS 1Gbps ISP

stephenw10

Start by setting net.isr.dispatch to deferred. You can do that in the system tunables page:
https://docs.netgate.com/pfsense/en/latest/config/advanced-tunables.html

The others are probably set as 2 by default but you can check:

[22.05-RELEASE][admin@4100-2.stevew.lan]/root: sysctl net.isr.maxthreads
net.isr.maxthreads: 2
[22.05-RELEASE][admin@4100-2.stevew.lan]/root: sysctl net.isr.numthreads
net.isr.numthreads: 2

I would test the result of doubling them initially. Since that CPU only has 2 cores anyway it may not help.

Steve

davecullen86

@stephenw10 said in Limited throughput - 500Mbps VS 1Gbps ISP:

et.isr.numthreads

Thank a lot!
I made those changes, as well as some others as below from Googling:

I see 100Mbps improvement, ~500 --to--> ~600, so definitely an improvement, but not quite there yet. Am I now at the limit of my HW and need to upgrade? Thx!!!

stephenw10

@davecullen86 said in Limited throughput - 500Mbps VS 1Gbps ISP:

Am I now at the limit of my HW and need to upgrade?

Probably. Check the output of top -HaSP at the CLI while you're testing. See it one CPU core is at 100%.
But, yes, I would not expect to pass much more than that using PPPoE and that CPU.

Steve

Patch

@davecullen86 said in Limited throughput - 500Mbps VS 1Gbps ISP:

need to upgrade?

have a look at https://www.netgate.com/pfsense-plus-software/how-to-buy#appliances

davecullen86

@stephenw10
Here is a video, it does not look like 100% to me. What do you think?
https://file.io/LkYuqXCiaeHm
(File too big to attach here)

davecullen86

@patch in an ideal world, totally I’d buy one of those appliances.
One main point to PFsense generally for me, is the ability to use my own hardware to make the cost a lot lower.

That’s to my detriment here of course as I am hitting limitations.

My options I see here are:

Instead of PPPoE, use the ISP router in the WAN as a L3 hop. This means double NAT as bridge mode isn’t supported.
Buy new PFsense HW
Use some other 3rd party HW on the wan side to support what I need. Maybe a Draytek 166, anyone had experience with this?

Thank you guys, all responses are very much appreciated.

Dave

Patch

@davecullen86 Netgate have both specified and bench marked their hardware.

The best and safest solution is to buy a Netgate appliance.
If that is not possible you can compare hardware you have to a similar Netgate appliances to guess how it may perform. Note in practice that is not as easy as it first appears as Netgate have balanced the performance of each part of an appliance to achieve overall performance. As a result, equalling one aspect of the specification may not translate to overall performance.

stephenw10

Hmm, something odd about your ssh session there making it unclear. Looks like I missed a dash from my instructions! Should have read top -HaSP. I expect the output to look like:

last pid: 41269;  load averages:  0.04,  0.08,  0.08                              up 54+17:57:45  12:38:48
526 threads:   3 running, 510 sleeping, 13 waiting
CPU 0:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle
CPU 1:  0.0% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.8% idle
Mem: 15M Active, 217M Inact, 422M Wired, 3143M Free
ARC: 198M Total, 30M MFU, 162M MRU, 692K Anon, 912K Header, 4358K Other
     100M Compressed, 202M Uncompressed, 2.03:1 Ratio
Swap: 1024M Total, 1024M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        155 ki31     0B    32K CPU1     1 1298.4  99.65% [idle{idle: cpu1}]
   11 root        155 ki31     0B    32K RUN      0 1298.6  99.46% [idle{idle: cpu0}]
    0 root        -92    -     0B   544K -        1 187:07   0.27% [kernel{dummynet}]
    0 root        -76    -     0B   544K -        1 127:16   0.16% [kernel{if_config_tqg_0}]
41269 root         20    0    14M  4772K CPU0     0   0:00   0.15% top -HaSP
21459 root         20    0    17M  7460K select   1  35:11   0.06% /usr/local/sbin/openvpn --config /var/e
   12 root        -60    -     0B   208K WAIT     0  21:34   0.03% [intr{swi4: clock (0)}]
   19 root        -16    -     0B    16K pftm     1  28:20   0.02% [pf purge]
89724 dhcpd        20    0    22M    13M select   0   0:40   0.02% /usr/local/sbin/dhcpd -user dhcpd -grou
87627 root         20    0    20M  9524K select   1   0:00   0.02% sshd: admin@pts/0 (sshd)
    0 root        -76    -     0B   544K -        1  12:35   0.02% [kernel{if_io_tqg_1}]
   20 root        -16    -     0B    16K -        1   9:37   0.01% [rand_harvestq]
    0 root        -76    -     0B   544K -        0   3:28   0.01% [kernel{if_io_tqg_0}]
94357 root         20    0    11M  2788K nanslp   0   0:18   0.01% /usr/local/bin/dpinger -S -r 0 -i OPENV
94054 root         20    0    11M  2780K nanslp   1   0:15   0.01% /usr/local/bin/dpinger -S -r 0 -i 6100_

Where you can see exactly what the loading is on each CPU core and what's causing it.

Steve

A Former User

The problem here is that I can only achieve around
500Mbps downstream in a speedtest, both from an
Ethernet cable (1Gb) LAN side connected client and

Ok, but with a 2,0GHz CPU you might be able to archive
500 MBit/s according to the pfSense "Hardware recommendations" and you archive 500 - 600 MBit/s!

So all should be fine for you!

I checked the CPU load monitor on the dashboard
whilst performing the speedtests, and I see 69% highs
so it does not look like a saturated CPU to me.

This can be, but with PPPoE you will be "pinned" or "nailed" to only one CPU core! And 2,0GHz is for archiving
~500 MBit/s the minimum on an actual CPU and yours is from 2013 and we are in 2022, please don´t forget this.

FYI, testing with the ISP router, I do get the advertised
speeds, so I know the circuit can achieve what I have
paid for here.

They (ISPs) are often soldering special ASICs on this routers and with them you will be archive full 1 GBit/s
pfSense comes as a Software firewall to you, and no one
can imagine on what hardware it will be installed by you!

I am using the same cabling that was WAN and LAN
side of the ISP router when it was used, tried alternate
cables also.

Cables makes the CPU not faster and also are not speeding up like an ASICs.

Can anyone advise if this is a HW limitation issue or
if somehow I can further troubleshoot this problem?

They are much tuning tips out, but this is not a single I
set it up and it works thing! You should be setting up or change much more points and then you combine them
this is often the "salt in the soup". I would considering to start at Friday over the weekend to get better in touch with it.

Ideally, I want full speeds of course and ideally, I want
to continue using PFSense!

We all want it, not only you!!

davecullen86

Hey guys, many thanks for your response. The more I look into this, the more I see so many others with the same issue. As you say the issue is implicit to the PPPoE single core factor and the clock speed of an individual core of my small appliance.

I have a solution! With another identical appliance, I have installed OpenWRT x86 and I am not getting close to 900Mbps throughput.

Now, THIS IS good enough for me :-). So I suggest is a good potential solution for others who are happy to offload the PPPoE function to another inline appliance.

Now I just need to work out if I can pass through the WAN IP somehow to my PFSense :-)
Thanks for your help again - I really appreciate the pointers that ultimately led me to get a working solution.
Cheers - Dave

stephenw10

Let us know if you find a way to do that. I've looked into it before and came to the conclusion that it might be possible but it involves some unconventional setup!

A Former User

@davecullen86

Hey guys, many thanks for your response. The more
I look into this, the more I see so many others with the
same issue.

I have some, a couple of PC ENgines APU boards, and
I run MikroTik RouterOS, OpenWRT, pfSense on them,
all Linux comes more to 1 GBit/s with lower powered
hardware, it is a little bit more near to the hardware
due to better driver support and here and there not
so "hardware hungry", but a router and a firewall
that can be turned into a real UTM device is als not
the same! As I see it personally, you could try out as @stephenw10 was suggesting to tune your pfSense
a little here and there. With DanOS you might be getting
nearly two streams with full GBit/s on the same hardware
(PC Engines APUx), owed to DPDK capable LAN ports such
Intel i210 / i211.

As you say the issue is implicit to the PPPoE single core > factor and the clock speed of an individual core of my
small appliance.

Like me, but I was high up the cpu frequency to another
level and play now around with some other tuneable`s,
to get here and there more out of my hardware pointed
to the entire throughput. But I also know that my appliance is better cooled then other and will never goes
higher then 65 C° - 70 C°!!!! The CPU is normally capable
of 1400 MHz and runs even only at 600 MHz - 1000 MHz
and now it is running from 1000 MHz till 1400 MHz, but
if something goes wrong, I don´t complain and be angry!

I have a solution! With another identical appliance, I
have installed OpenWRT x86 and I am not getting close > to 900Mbps throughput.
And with DanOS you may be bidirectional getting fully
1 GBit/s out! But not a fully UTM in your Network!!!!!!!!

Firewall
Captive Portal with voucher system (voucher over sms)
FreeRadius with certificates and encryption
Snort or Suricata for IDS/IPS
pfBlocker-NG for less spam and other unwanted things
Squid & SquidGiuard as a caching proxy in fron of LAN
ClamAV scanning the entire network flow for viruses
(perhaps at one day WiFi a/b/g/n/ax)

Now, THIS IS good enough for me :-). So I suggest is
a good potential solution for others who are happy to
offload the PPPoE function to another inline appliance.

I run a AVM FB 7590ax in front of the pfSense and behind
I am running the pfSense firewall! No PPPoE anymore, but
double NAT situation! But all CPU cores in usage!

AVM is offering some interesting APPs (VPN, telephone,..)
Really nice to connect from outside (internet) and being secure on the LAN side!

Now I just need to work out if I can pass through
the WAN IP somehow to my PFSense :-)

1 LAN Port as "exposed host" to the WAN interface
of the pfSense firewall ("Experienced")
Double NAT Situation

Router:
network (net) 192.168.178.0/24 (255.255.255.0)
Router IP 192.168.178.1/24 (255.255.255.0)
Static IP Address to the pfSense a.e. 192.168.178.10/24
DHCP off: all IPs will be static given to the clients

pfSense:
WAN IP 192.168.178.50/24 (255.255.255.0) static IP
LAN Net: 172.xx.xx.0/24 (255.255.255.0)
LAN IP 172.xx.xx.1/24 (255.255.255.0) static IP
DHCP: on/off (Like you need it and want it)

Thanks for your help again - I really appreciate the
pointers that ultimately led me to get a working
solution.

Not that problem, you are one from xyz sitting in the same
boat. I would also have a look on another appliance if I`ll
getting more then 50 MBit/s Internet speed!!!

P.S.
Please don´t forget in the WAN setup to disable the
following point!

WAN settings.jpg