2.4.1 - reduced performance

mloiterman

I’ve just updated from 2.3.4 to 2.4.1 and have noticed about a 10% drop in throuput as measured by speed tests from dslreports.com/speedtest and speedtest.net. Before 2.4.1 I was regularly getting downloads in the 930Mbs from dslreports. Now, I’m really only getting 830Mbs. Uploads remain unchanged.

My hardware is:

SUPERMICRO SYS-5015A-EHF-D525 1U Intel Atom D525

Intel(R) Atom(TM) CPU D525 @ 1.80GHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: No

My NIC is:
Intel E1G44HT Server Adapter I340-T4 10/100/1000Mbps PCI-Express 2.0 4 x RJ45

I have run the speed tests while running top and my CPU does appear to be straining at all. Is my hardware really just straining at running FreeBSD 11?

How can I go about beginning to determine a possible root cause?

kejianshi

What I get with speedtest.net changes from hour to hour and day to day.

mloiterman

Agreed, but dslreports is pretty consistent and I’m seeing a similar percentage drop on both.

I’m just wondering if I’m pushing the limits of my hardware or if there are new options in 2.4.1 that are tuned for better hardware?

I’ll likeky replace this hardware shortly, but in the meantime, I would like to squeeze out as much performance as I can.

JKnott

I noticed a significant speed drop, until I rebooted pfSense and cable modem. The nearest speedtest server is at my ISPs local office, about 8 Km away (driving distance). I have a 60/10 package, but generally get mid 70s down and 11 up.

I have noticed a delay in loading web pages etc. I have temporarily set this computers DNS to Google and things appear better, but I'll have to do more testing to be sure.

kejianshi

I'm not sure. I had to recently remove an atom powered machine and place it on a slower network where it could do its thing and not limit the ISP speed.
Its still going strong but the speeds you are getting are really very great for an atom like yours.

Guest

@mloiterman : Do you see an unusual change in the RTT and RTTsd values on your WAN Gateway? Are they in single digits?

mloiterman

No changes to Latency or packet loss. I have restarted pfsense machine as well as DNS server. It’s just the top end seems to level out at 830Mbs.

This is a fresh install with ZFS. I have 4 gigabytes of RAM and I’m only use about 15% of the memory.

Hardware seems to not even really be breaking a sweat.

I only have AutoBackup and OpenVPN client export packages installed.

left4pook

Hum… 10%

See what I posted here https://forum.pfsense.org/index.php?topic=139084.0

Add:

kern.random.harvest.mask: 351

to System / Advanced / System Tunables in the webGUI

Might get some of it back for you.

Guest

Intel(R) Atom(TM) CPU D525 @ 1.80GHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads

Perhaps this might be the entire available throughput without any tunings, that this hardware will be able to realize or serve.
ZFS may needs a bit more power but is never narrowing down the internet or WAN interface speed such like this.

I’ll likeky replace this hardware shortly, but in the meantime, I would like to squeeze out as much performance as I can.

This is what I also would to guess that a replacement will be mostly more helping out as all fid lings and tunings together.

It could be that the PowerD (high adaptive) is not working properly and the the CPU is now "nailed" at one MHz number
and is not really scaling up and down as needed.
The num.qeue size or amount is to high could also be used to get more throughput, narrow down to 1, 2 or 4 might be
a good chance too
The mbufsize itselfs will be mostly matching well too, to the num.queue size or amount, that is something you can "play"
around with but mostly with the best effect together. But with 4 GB of RAM please be carful that you are not ending up in
a so called booting loop!

If, and I really don´t know it really, but if the ZFS is working together with encryption and your CPU has now AES-NI
support, it can also be, that the CPU must do that now all by regular horse power and this is then a to small footprint
to get the normal WAN throughput back and perhaps also a nice hint, once more again if so, that we all need since the
version 2.5.x hardware that comes with AES-NI support. Perhaps not alone for that ZFS but also together with other
things that will be on a change or being new and on top of this.

mloiterman

I wonder if it has anything to do with zfs?

The strange thing is the CPU never seems to even approach 100%.

kejianshi

The whole cpu doesn't need to max out to make things slow. On my pfsense thru VPN for example, I can hit a wall by maxing out a single core out of 4. It may only say 30% load, but it will be effectively maxed out.