APU2 Slower than expected in iPerf3

Stewart

I'm trying to determine the max speed I can get out of my APU2 but I don't have a very good solution for a real world test at the moment. The good news is that with everything loaded (although it takes a bit to fully boot) it does everything well. Snort, Squid, clam, pfBlocker, etc all performs well with relatively low CPU overhead. The problem is that I can't get iPerf3 to show anything higher than 300-340Mb/s. With nothing running and power set to Maximum I get around 340Mb/s. With everything loaded and power set to Adaptive I only get around 300Mb/s.

Let's assume 2 PCs and the APU2. Here's some stats:
If I set the APU2 to be server and client I get around 1.6Gb/s
If I set the APU2 to be server and connect from either PC I get around 320Mb/s +- 20Mb/s.
If I set a PC to be server and connect from the other PC I get around 933Mb/s +- 10Mb/s.

If I use a crossover cable and connect a PC directly to the APU2 I get around 320Mb/s +- 20Mb/s.

The whole time during testing CPU doesn't go over 50%. How can I find what the bottleneck is and what's going on?

Thanks!

tirsojrp

Using the APU2 as server or client is useless, the only result you need is this:

If I set a PC to be server and connect from the other PC I get around 933Mb/s +- 10Mb/s.

Stewart

But that is PC to PC, without touching the router. I put that in to show that the PCs I were using weren't bottlenecking through the switch. How would going PC to PC show that everything is fine in the router?

tirsojrp

Ooops, sorry. I thought it was PC to PC through pfsense.

That's how it must be done.

Connect PC server on WAN and PC client on LAN.

stephenw10

^That.

Running the server or client and terminating the tcp connection on the box is not a valid test.

Set powerd to hiadaptive if you haven't already.

The actual throughput results will be very dependent on the settings used in iperf also.

Steve

Stewart

I'll try to get these tests run later today or tomorrow.

whosmatt

I can attest that iperf on any of my pfSense iterations is always slower than Linux or Windows on the same or similar hardware, just using default settings. Not sure this says anything about pfSense, maybe more about iperf. It's not a bad test, but your true throughput may or may not be indicated by an iperf test. IMO.

Stewart

I finally got those tests run. I ran iperf3 with the switch -t 60. I then took the utilization samples at 30-35 seconds and recorded the throughput shown at the end of the run.


Power	Snort	Squid	pfBlocker		CPU		%	Throughput Mb/s	
Off	On	On	On			Snort		99	Send	505
						Darkstat	7	Receive	505
						User		24		
						Nice		0		
						System		30	
						Interrupts	24	
						Idle		23		

Power	Snort	Squid	pfBlocker		CPU		%	Throughput Mb/s	
AdaptiveOn	On	On			Snort		100	Send	501
						Darkstat	7	Receive	501
						User		26		
						Nice		0		
						System		25	
						Interrupts	17	
						Idle		30		

Power	Snort	Squid	pfBlocker		CPU		%	Throughput Mb/s	
AdaptiveOff	On	On			Snort		0	Send	540
						Darkstat	7	Receive	540
						User		2		
						Nice		0		
						System		26	
						Interrupts	21	
						Idle		51

A couple of notes.
1. I never received more than 550Mb/s. While that is plenty for what we would use it for, I'm surprised it is so low.
2. I always had interrupts at 15-25%. I thought that would go away when we switched from Realtek to Intel NICs.
3. It looks like iperf3 only utilizes 2 cores. Is there a way to get it to use all available resources?

Edit: Clear up chart.

stephenw10

You can run multiple instances to use more cores.

It would be worth running```
top -aSH


Steve

Guest

In normal the APU2C4 is set up as the following;

install the mSATA and then pfSense on it
– perhaps activate the TRIM support for the storage
-- high up the mbuf size or numbers if that is really needed
-- activate Powerd (high adaptive)

PC1 (server) to PC2 (client) using two LAN Ports
PC1 (server) tp PC2 (client) using one LAN Port and the WAN Port (with a Switch in front of it)

This might be the best option to know what you might be expecting form that kind of hardware
but then please without any kind of installed packet such as Snort, Squid, SuiqdGuard, pfBlockerNG,
and so on and so on, so you see it would be the best to test it with a fresh installed pfSense and not
later after all gets installed, so you will get the raw numbers and the numbers scaled down based on
the installed packets later!!!