PC Engines apu2 experiences

Stewart

I'd like to add what I've found and compare the APU1D with the APU2C4. Each test was run 5 times and the average is shown:


_______________APU1D__________Without aes-ni Enabled in GUI_____AES-128
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
openssl speed -elapsed aes-128-cbc
aes-128-cbc      20,150.25	   21,593.45	 22,101.23	   55,892.72	 57,108.07
openssl speed -elapsed -evp aes-128-cbc
aes-128-cbc      20,879.17	   22,096.23	 22,604.51	   22,781.61	 22,756.18
openssl speed -elapsed aes-128-cbc -multi 2
aes-128-cbc      37,715.27	   42,234.96	 43,208.21	  108,581.00	108,638.48
openssl speed -elapsed -evp aes-128-cbc -multi 2
evp              41,202.15	   43,115.07	 43,609.43	   42,840.60	 44,048.14

_______________APU1D__________Without aes-ni Enabled in GUI_____AES-256
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
openssl speed -elapsed aes-256-cbc
aes-256 cbc      14,700.73	   15,444.83	 15,733.50	   41,247.34	 41,710.94
openssl speed -elapsed -evp aes-256-cbc
aes-256-cbc      15,243.62	   15,707.42	 15,961.70	   16,126.77	 15,934.53
openssl speed -elapsed aes-256-cbc -multi 2
aes-256 cbc      23,949.23	   26,988.24	 29,858.76	   65,845.54	 64,089.45
openssl speed -elapsed -evp aes-256-cbc -multi 2
evp              29,593.04	   26,244.35	 26,773.70	   28,397.03	 27,938.67

_______________APU1D__________With aes-ni Enabled in GUI_____AES-128
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
openssl speed -elapsed aes-128-cbc
aes-128 cbc      19755.576	  21431.89	   21989.752	55771.576	 55630.234
openssl speed -elapsed -evp aes-128-cbc
aes-128-cbc      20863.098	  22093.112	   22559.898	22602.114	 22531.338
openssl speed -elapsed aes-128-cbc -multi 2
aes-128 cbc      37336.336	  38520.556	   42471.264   105237.468	99426.206
openssl speed -elapsed -evp aes-128-cbc -multi 2
evp              36558.862	  40986.052	   42027.06	    40009.182	 41684.274

_______________APU1D__________With aes-ni Enabled in GUI_____AES-256
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
openssl speed -elapsed aes-256-cbc
aes-256 cbc      14591.166	  14837.534	   14614.882	39739.044	 40290.906
openssl speed -elapsed -evp aes-256-cbc
aes-256-cbc      14994.722	  15396.05	   16006.702	16093.2	     15921.974
openssl speed -elapsed aes-256-cbc -multi 2
aes-256 cbc      24330.116	  27610.256	   26142.88	    71589.386	 70645.116
openssl speed -elapsed -evp aes-256-cbc -multi 2
evp              25427.984	  27953.616	   26119.284	28292.242	 26312.212

_______________APU2C4__________Without aes-ni Enabled in GUI_____AES-128
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
openssl speed -elapsed aes-128-cbc
aes-128-cbc       14,602.14 	 15,604.71 	 16,020.81 	 41,673.96 	 42,613.15 
openssl speed -elapsed -evp aes-128-cbc
aes-128-cbc      116,857.16 	 167,172.30 	 205,183.44 	 216,286.74 	 219,179.69 
openssl speed -elapsed aes-128-cbc -multi 2
aes-128-cbc       52,436.02 	 58,305.43 	 58,527.76 	 154,819.86 	 162,012.23 
openssl speed -elapsed -evp aes-128-cbc -multi 2
evp               5,339.28 	 20,562.37 	 75,235.53 	 230,458.68 	 567,333.62 

_______________APU2C4__________Without aes-ni Enabled in GUI_____AES-256
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
openssl speed -elapsed aes-256-cbc
aes-256 cbc       10,657.51 	 11,205.91 	 11,310.90 	 30,765.00 	 31,377.54 
openssl speed -elapsed -evp aes-256-cbc
aes-256-cbc       96,810.10 	 129,034.06 	 150,190.10 	 156,638.07 	 158,143.28 
openssl speed -elapsed aes-256-cbc -multi 2
aes-256 cbc       39,620.04 	 40,461.33 	 40,217.14 	 120,696.35 	 117,217.43 
openssl speed -elapsed -evp aes-256-cbc -multi 2
evp               5,224.40 	 21,083.67 	 73,885.68 	 201,226.44 	 442,017.98 

_______________APU2C4__________With aes-ni Enabled in GUI_____AES-128
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
openssl speed -elapsed aes-128-cbc
aes-128 cbc       14,547.43 	 15,599.68 	 16,005.85 	 41,691.67 	 42,459.34 
openssl speed -elapsed -evp aes-128-cbc
aes-128-cbc       1,455.86 	 5,778.35 	 21,179.49 	 64,385.85 	 158,815.65 
openssl speed -elapsed aes-128-cbc -multi 2
aes-128 cbc       53,114.91 	 57,221.27 	 58,445.19 	 159,149.88 	 158,859.67 
openssl speed -elapsed -evp aes-128-cbc -multi 2
evp               5,355.99 	 21,216.93 	 75,614.86 	 228,806.89 	 572,782.12 

_______________APU2C4__________With aes-ni Enabled in GUI_____AES-256
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
openssl speed -elapsed aes-256-cbc
aes-256 cbc       10,657.26 	 11,111.40 	 11,175.44 	 30,771.72 	 31,289.62 
openssl speed -elapsed -evp aes-256-cbc
aes-256-cbc       1,404.00 	 5,528.13 	 19,735.86 	 55,687.85 	 119,758.85 
openssl speed -elapsed aes-256-cbc -multi 2
aes-256 cbc       39,908.48 	 39,509.88 	 41,580.65 	 117,316.88 	 117,157.87 
openssl speed -elapsed -evp aes-256-cbc -multi 2
evp              5,456.64 	 20,749.14 	 70,953.42 	 207,225.20 	 456,061.90

Some things stand out fairly obvious but I'll need some help determining what is going on.
First, it's obvious that enabling aes-ni in the Advanced-Misc section does something. It seems to have a modest affect on the APU1D and a fairly detrimental affect on the APU2C4.
Second, the APU2C4 only has about 73% of the performance of the APU1D using a single core and no hardware acceleration.
Third, the APU2D4 seems to excel in some circumstances and bomb in others. I expected to see a fairly consistent trend, but the charts seem to say otherwise.

Does anyone know which number is most representative of performance when running the openssl speed test? From what I'm seeing it appears that on the APU1D with aes-ni disabled we should be able to get 108MB/s on aes-128 with an 8k block size and 65MB/s on aes-256 with an 8k block size. It also appears that with the APU2D4 we should be able to get 567MB/s on aes-128 with an 8k block size and 442MB/s on aes-256 with an 8k block size. Those don't seem right to me. Can anyone care to elaborate?

Also, I've attached a screenshot of the spreadsheet I've put together with basic heatmaps. All values are the same and "are in 1000s of bytes per second processed". I just dropped the "k" so I could run calculations. Also, the commands that were run that include the multi switch were run as "multi 2" on the APU1D and as "multi 4" on the APU2C4.

Edit: I should note that these are all run with powerd on set to hiadpative. Also, added APU2C4 numbers that were forgotten.

APU1DvsAPU2C4.jpg_thumb

Stewart

The average I get running iperf across the LAN interfaces is:
APU1D -> APU2C4 = 235Mb/s
APU2C4 -> APU1D = 218Mb/s
APU1D -> Core2Box = 121Mb/s
APU2C4 -> Core2Box = 222Mb/s

Services enabled on the APU1D and APU2C4 in case they affect things are:
Squid
SquidGuard
AV Integrated in Squid
Snort
pfBlockerNG
darkstat
LightSquid

kevindd992002

Are you guys using a full install or a nanobsd install on an msata 16gb ssd on an apu2c4?

Stewart

I'm running a full install.

kevindd992002

Without any modifications like using ramdisks for /tmp and /var? Fulll install as is?

Stewart

Yes. Using an msata ssd.

mikeboss

@Stewart:

Second, the APU2C4 only has about 73% of the performance of the APU1D using a single core and no hardware acceleration.

I saw similar results, back when I got my first APPU2. it seemed to be odd but then I re-ran the benchmark a few times and I always got more or less the same score each time…

APU1 https://browser.primatelabs.com/geekbench3/4636493
APU2 https://browser.primatelabs.com/geekbench3/4635680

regards,
michael

kevindd992002

Are the APU2 developers already aware of this? This issue seems to be a major one that needs addressing, right?

Qinn

I read there is a module for using the 3 front LEDs on the apu2 boards (http://pcengines.ch/howto.htm#gpio), has anyone using pfSense experience with this?

Thanks in advance,

Cheers Qinn

maglub

Hi,

I just set up two APU2C4 boxes with pfSense 2.3.2. After a bit of fiddling with the USB stick for TinyCore, I managed to get it installed. On a 1Gbit connection, I had >800Mbit down and up through the firewall. No tuning what so ever.

I am using pfSense-CE-2.3.2-RELEASE-4g-amd64-nanobsd.img.gz

j4k3

Hey Guys, I wish I was getting maglub's performance out of the box.

I just received a APU2C4 in the mail and loaded pfSense-CE-memstick-serial-2.3.2-RELEASE-amd64 on it.

It manages to boot fine, and it's fully functional, the web interface moves much quicker and more fluid than the old ALIX2D3 it had replaced.

However when I run iperf on the LAN interface from another host, I'm only getting 300Mbps or so.

I found this thread about Intel I210AT nics and am attempting their suggestions:

https://redmine.pfsense.org/issues/1221

I see they're playing with the buffers and queues.

My question is this, if I edit loader.conf.local after looking at some of the options some options have "Quotes" around them, and others don't.

If I directly edit loader.conf.local when I specify variables in there, must they use " for the value? Or will just the value suffice?

hw.igb.num_queues="4" or hw.igb.num_queues=4 ?

Thanks

cwagz

I always use the quotes. I have not tried it without.

j4k3

Thanks for the insight, I managed to finally get close to 1Gbps on the lan interface.

I had to uncheck Disable hardware large receive offload, and Disable hardware TCP segmentation offload

Under System > Advanced > Networking

Based on what I've read so far I know this unit won't route more than 500 Mbps or so but I wanted to at least understand why, the nic was so hobbled right off the bat.

cwagz

I am looking for some opinions on downsizing my current pfSense system with an APU2C4.

Currently I have:
Supermicro A1SRI-2558
8GB Ram
120GB SSD
Akasa Fanless Enclosure

There are 6 people in my house and 30 or so devices. I am the only person that ever uses OpenVPN and it is usually from a mobile device on LTE so OpenVPN performance is probably not a huge deal. I run Squid and Squidguard to proxy the internet for my kids. Our internet connection is FiOS 150/150 Mbps.

It seems like I could build an apu2c4 and sell my current hardware. I would probably have money left over and a smaller, slightly cooler running device for pfSense.

Do you guys see any potential performance issues or reasons why this is a bad idea?

Stugots

Does anyone know of a way to enable TRIM support on the SSD without having to boot of a recovery device? Is there some sort of tunable where it can be enabled on the next reboot?

AndrewZ

@acascianelli:

Does anyone know of a way to enable TRIM support on the SSD without having to boot of a recovery device?

https://forum.pfsense.org/index.php?topic=66622.0
https://forum.pfsense.org/index.php?topic=113803.msg633795#msg633795

lachendichatz

I switched from A1SRI-2758/8GB to apu2c4 as I need the 2758 for another server.

Running iperf3 between two pcs connected by the apu (pfsense 2.3.2) I get
600Mbit/s in one direction
615Mbit/s in the other one.
CPU runs at 25% load, e.g. one core is maxed out.

I probably see different speeds because I already imported my old firewall rules and have three rules on one nic and around 20 on the other one.

I observed the same speed & load when I installed debian and configured a few iptable rules.

Adding more clients (iperf -P 8 ) gives me:
940Mbit/s direction a
690Mbit/s direction b
880MBit/s duplex

Enabling (disabling unchecked) segmentation offload gives me
940Mbit/s direction a
695Mbit/s direction b
940Mbit/s duplex
CPU runs at 85%

Speedtest (init7) shows me 930down/940up. Initially I got 720/920 & 670/920 but that was because of my slow laptop (sigh). Restarting firefox gave me consistent speeds around 940.

So I can route a single TCP connection at 600Mbit/s per core. One could probably achieve higher speeds by tuning ISR related configs but as I can saturate my gigabit line with multiple connections I won't change settings.

I set igb_numqueue to 4 and mbuf to 1mio. Unknown if it had an effect.
Somebody suggested to set a rx/tx level (or queue? dont remeber) to 8k. That did not have an effect.
powerD disabled/enabled (hiadaptive) did not make a difference

Stugots

@AndrewZ:

@acascianelli:

Does anyone know of a way to enable TRIM support on the SSD without having to boot of a recovery device?

https://forum.pfsense.org/index.php?topic=66622.0
https://forum.pfsense.org/index.php?topic=113803.msg633795#msg633795

Is there no way to set it so that it's enabled on the next reboot without going into single user mode?

Guest

Is there no way to set it so that it's enabled on the next reboot without going into single user mode?

Actually is there no way or workaround, as I am informed right.

kevindd992002

@j4k3:

Thanks for the insight, I managed to finally get close to 1Gbps on the lan interface.

I had to uncheck Disable hardware large receive offload, and Disable hardware TCP segmentation offload

Under System > Advanced > Networking

Based on what I've read so far I know this unit won't route more than 500 Mbps or so but I wanted to at least understand why, the nic was so hobbled right off the bat.

Does that mean these two should be unchecked to get the full potential of the NIC's of the APU2C4? Any disadvantages of keeping them unchecked (enabled)?