PC Engines apu2 experiences



  • Thanks for the insight, I managed to finally get close to 1Gbps on the lan interface.

    I had to uncheck Disable hardware large receive offload, and Disable hardware TCP segmentation offload

    Under System > Advanced > Networking

    Based on what I've read so far I know this unit won't route more than 500 Mbps or so but I wanted to at least understand why, the nic was so hobbled right off the bat.



  • I am looking for some opinions on downsizing my current pfSense system with an APU2C4.

    Currently I have:
    Supermicro A1SRI-2558
    8GB Ram
    120GB SSD
    Akasa Fanless Enclosure

    There are 6 people in my house and 30 or so devices.  I am the only person that ever uses OpenVPN and it is usually from a mobile device on LTE so OpenVPN performance is probably not a huge deal.  I run Squid and Squidguard to proxy the internet for my kids.  Our internet connection is FiOS 150/150 Mbps.

    It seems like I could build an apu2c4 and sell my current hardware.  I would probably have money left over and a smaller, slightly cooler running device for pfSense.

    Do you guys see any potential performance issues or reasons why this is a bad idea?



  • Does anyone know of a way to enable TRIM support on the SSD without having to boot of a recovery device?  Is there some sort of tunable where it can be enabled on the next reboot?



  • @acascianelli:

    Does anyone know of a way to enable TRIM support on the SSD without having to boot of a recovery device?

    https://forum.pfsense.org/index.php?topic=66622.0
    https://forum.pfsense.org/index.php?topic=113803.msg633795#msg633795



  • I switched from A1SRI-2758/8GB to apu2c4 as I need the 2758 for another server.

    Running iperf3 between two pcs connected by the apu (pfsense 2.3.2) I get
    600Mbit/s in one direction
    615Mbit/s in the other one.
    CPU runs at 25% load, e.g. one core is maxed out.

    I probably see different speeds because I already imported my old firewall rules and have three rules on one nic and around 20 on the other one.

    I observed the same speed & load when I installed debian and configured a few iptable rules.

    Adding more clients (iperf -P 8 ) gives me:
    940Mbit/s direction a
    690Mbit/s direction b
    880MBit/s duplex

    Enabling (disabling unchecked) segmentation offload gives me
    940Mbit/s direction a
    695Mbit/s direction b
    940Mbit/s duplex
    CPU runs at 85%

    Speedtest (init7) shows me 930down/940up. Initially I got 720/920 & 670/920 but that was because of my slow laptop (sigh). Restarting firefox gave me consistent speeds around 940.

    So I can route a single TCP connection at 600Mbit/s per core. One could probably achieve higher speeds by tuning ISR related configs but as I can saturate my gigabit line with multiple connections I won't change settings.

    I set igb_numqueue to 4 and mbuf to 1mio. Unknown if it had an effect.
    Somebody suggested to set a rx/tx level (or queue? dont remeber) to 8k. That did not have an effect.
    powerD disabled/enabled (hiadaptive) did not make a difference



  • @AndrewZ:

    @acascianelli:

    Does anyone know of a way to enable TRIM support on the SSD without having to boot of a recovery device?

    https://forum.pfsense.org/index.php?topic=66622.0
    https://forum.pfsense.org/index.php?topic=113803.msg633795#msg633795

    Is there no way to set it so that it's enabled on the next reboot without going into single user mode?



  • Is there no way to set it so that it's enabled on the next reboot without going into single user mode?

    Actually is there no way or workaround, as I am informed right.



  • @j4k3:

    Thanks for the insight, I managed to finally get close to 1Gbps on the lan interface.

    I had to uncheck Disable hardware large receive offload, and Disable hardware TCP segmentation offload

    Under System > Advanced > Networking

    Based on what I've read so far I know this unit won't route more than 500 Mbps or so but I wanted to at least understand why, the nic was so hobbled right off the bat.

    Does that mean these two should be unchecked to get the full potential of the NIC's of the APU2C4? Any disadvantages of keeping them unchecked (enabled)?



  • @kevindd992002:

    … Any disadvantages of keeping them unchecked (enabled)?

    Possibly, like no or a snappy WAN-PPPoE connection.



  • @acascianelli:

    …Is there no way to set it so that it's enabled on the next reboot without going into single user mode?

    https://forum.pfsense.org/index.php?topic=121515.msg673176#msg673176 / pfSense 2.4



  • @hda:

    @kevindd992002:

    … Any disadvantages of keeping them unchecked (enabled)?

    Possibly, like no or a snappy WAN-PPPoE connection.

    But why is the NIC performance hampered with these settings disabled anyway?

    @hda:

    @acascianelli:

    …Is there no way to set it so that it's enabled on the next reboot without going into single user mode?

    https://forum.pfsense.org/index.php?topic=121515.msg673176#msg673176 / pfSense 2.4

    So if I understand this correctly, a fresh install of 2.4 will already enabled TRIM automatically with no user intervention? And same goes with older versions of pfsense that upgrade 2.4, TRIM will be enabled?


  • Banned

    @kevindd992002:

    But why is the NIC performance hampered with these settings disabled anyway?

    You clearly are confused. When you check them, you DISable the HW offloading features.



  • I don't think I am. Clearly, unchecking the boxes = ENABLES these features. checking the boxes=DISABLES these features. It's very easy to distinguish between the two.

    j4k3 said in his post: "I had to uncheck Disable hardware large receive offload, and Disable hardware TCP segmentation offload". Which means that enabling (very different from "checking") them improves performance.

    So then I asked: "But why is the NIC performance hampered with these settings disabled anyway?". Or in other words: "why is the NIC performance hampered with the boxes CHECKED anyway?"

    Does that make sense? Again, disable=checked and enabled=unchecked. Please check the terminologies that I used in my posts.



  • Does that mean these two should be unchecked to get the full potential of the NIC's of the APU2C4?

    Here under this link you will be able to read what is really needed for getting 1 GBit/s at the
    WAN interface, there is told something likes, Server grade hardware and ~2,0GHz CPU speed.
    And as I see it right the APU1D4 and APU2C4 are only sorted with something around ~1,1GHz
    or 1,2GHz CPU power, that's it in short. Please read under under CPU selection

    Any disadvantages of keeping them unchecked (enabled)?

    Tunings and pimps can be done on each machine for sure to high up the
    throughput but in that case, you should be followed to that guidance
    from above at first.



  • @cwagz:

    I am looking for some opinions on downsizing my current pfSense system with an APU2C4.

    Currently I have:
    Supermicro A1SRI-2558
    8GB Ram
    120GB SSD
    Akasa Fanless Enclosure

    There are 6 people in my house and 30 or so devices.  I am the only person that ever uses OpenVPN and it is usually from a mobile device on LTE so OpenVPN performance is probably not a huge deal.  I run Squid and Squidguard to proxy the internet for my kids.  Our internet connection is FiOS 150/150 Mbps.

    It seems like I could build an apu2c4 and sell my current hardware.  I would probably have money left over and a smaller, slightly cooler running device for pfSense.

    Do you guys see any potential performance issues or reasons why this is a bad idea?

    I went ahead and built the apu2c4 and am very happy with the outcome.  The performance seems to be the same for our usage.  Also, the overall footprint and heat output into my small network cabinet is improved.



  • Hey,

    I recently took delivery of an APU2C4. It is certainly a decent performer for the size of it!

    I am wondering, has anyone got the AES-NI to work with the OpenVPN? The reason I ask is that I don't appear to see any acceleration happening with AES-128-CBC / AES-256-CBC. The rough maximum I have achieved is 30Mbps.

    I have tried enabling the AES-NI within Advanced Options, and then enabling the cryptodev within OpenVPN. As well as disabling AES-NI and leaving Cryptodev enabled vice-versa.

    However, I see no changes whatsoever.

    I am on the latest PFSense 2.3.x release

    Kindest Regards
    HC



  • I am wondering, has anyone got the AES-NI to work with the OpenVPN? The reason I ask is that I don't appear to see any acceleration happening with AES-128-CBC / AES-256-CBC. The rough maximum I have achieved is 30Mbps.

    From what total line speed you archived the 30Mbps? And how strong was the other VPN Peer end?

    I have tried enabling the AES-NI within Advanced Options, and then enabling the cryptodev within OpenVPN. As well as disabling AES-NI and leaving Cryptodev enabled vice-versa.

    At the moment only IPsec is really benefitting from the AES-NI, so you might be having
    perhaps more luck if the OpenVPN version 2.4 is out there.



  • From what total line speed you archived the 30Mbps? And how strong was the other VPN pear end?

    Connecting from a 317Mbps line, the other end is serviced by a 10Gbit (SFP) line @ Rackspace

    At the moment only IPsec is really benefitting from the AES-NI, so you might be having
    perhaps more luck if the OpenVPN version 2.4 is out there.

    I'll hold out, I'm not too fussed - I didn't expect a lot. But I expected a tad better as my old equipment was a dual core 800Mhz MiPS. I had tried the "fix" here:

    http://1101entrails.blogspot.co.uk/2016/05/getting-aes-ni-to-work-using-pfsense-on.html



  • @HackedComputer:

    At the moment only IPsec is really benefitting from the AES-NI, so you might be having
    perhaps more luck if the OpenVPN version 2.4 is out there.

    I'll hold out, I'm not too fussed - I didn't expect a lot. But I expected a tad better as my old equipment was a dual core 800Mhz MiPS. I had tried the "fix" here:

    http://1101entrails.blogspot.co.uk/2016/05/getting-aes-ni-to-work-using-pfsense-on.html

    That page is mostly correct–openvpn does use aes-ni, having pfsense try to load any cryptographic stuff will slow things down, and you should be getting significantly more than 30Mbps. Make sure you're connecting with aes on the client side and turn off all the hardware crypto settings in pfsense.



  • Just an update:

    So, changing the cryptographic options within pfSense didn't yield any differences. Perhaps, by 5Mbps.

    However, I looked more into the OpenVPN configuration and appended the following to the client configuration:

    sndbuf 393216;
    rcvbuf 393216

    and thus, this was achieved:



  • Here some "facts" from tests with APU2C4 and the latest pfSense (2.3.2_1)

    WAN Speed on a clean Gigabit Link: ~640 MBits/s
    OpenVPN Speed: ~50 MBits/s (AES-128-CBC with SHA256)

    By enabling "Hardware Checksum Offloading" (by unchecking the setting) and "Hardware TCP Segmentation Offloading" (by unchecking the setting) I was able to get a 20-30 MBits/s improvement. Better, but still way off real Gigabit Speed.

    Same for the OpenVPN throughput

    • Activating the AES-NI support doesn't currently do anything (should hopefully improve with OpenVPN Version 2.4+)

    • Activating BSD Crypto engine reduces (!) the throughput by ~5MBits/s

    • setting sndbuf 393216; rcvbuf 393216 in the client config as suggest above didn't do anything neither. But this might be due to the server side still using an older OpenVPN version (2.3.2) where there are low defaults. The OpenVPN log shows Socket Buffers: R=[42080->393216] S=[57344->393216]

    Looking forward to other reports and suggestions!



  • @FreeMinded:

    Activating the AES-NI support doesn't currently do anything (should hopefully improve with OpenVPN Version 2.4+)
    Activating BSD Crypto engine reduces (!) the throughput by ~5MBits/s

    AES-NI is always on in current openvpn. Activating cryptodev overrides the built-in AES-NI support and does generally make openvpn slower. This is not specific to the APU2. If AES-NI were actually off (this can be done by setting an environment variable to tell openssl to ignore AES-NI support when running openvpn from the command line) you'd get a substantial reduction in throughput (more like 30% than 5%; the actual difference in crypto rates is much greater but openvpn has other bottlenecks).



  • hi guys, has anyone managed to use the full capacity of your storage. I installed by this guide: http://pcengines.ch/howto.htm#OS_installation

    Problem is that it only uses a part of my storage capacity. Can anyone tell me how to install on all available storage capacity?


  • Banned

    @Af0x:

    hi guys, has anyone managed to use the full capacity of your storage.

    Stop using nanobsd.



  • @doktornotor:

    Stop using nanobsd.

    ok, thank you. I did that and want to know which kernel is suitable for the apu2? I think it is the embedded one, but am not sure.


  • Banned

    Cannot recall ever selecting something there.  Get the memstick-serial image and leave it at default. :)



  • Can anyone tell me how to install on all available storage capacity?

    Where you want to install pfSense? In a mSATA, USB drive or a real HDD/SSD?
    If you take an usb drive and put on it the MEMSTICK-Serial-AM64 image and the you install from there onto a…....
    That is a fresh and full install and might be the best bet for you.



  • hi, thanks you guys. I already downloaded and installed on my mSATA. During Installation you get the choice like on the picture:
    https://doc.pfsense.org/images/1/11/Installer_05_select_console.png

    I chose embedded but I read that it has reduced possibilities talking about packages.



  • @Af0x:

    …mSATA...

    APU2 & mSata ? >>> 64-bit Full Install.



  • apu1d4: throughput was arround 700mbit/s ~ 900mbit/s, when i first bought it pfsense did 500mbit/s later on that changed
    apu3a4: throughput is 250mbit/s ~ 300mbit/s

    What i do notice is that apu1d4 uses realtek(re) drivers and the apu3 uses intel(igb)
    for the igb driver: TSO LRO .. checksum offload turned on or off makes no difference

    strange thing is when I run iperf through vlan but not hosted on the router but on a server on a different subnet which is routed though the router i get

    –----------------------------------------------------------
    [  3] local 192.168.1.100 port 44774 connected with 172.18.1.11 port 5001
    [ ID] Interval      Transfer    Bandwidth
    [  3]  0.0-10.0 sec  715 MBytes  600 Mbits/sec

    when i turn on tso lro checksum offload, cpu rather then hiadapt to maximum

    ------------------------------------------------------------
    Client connecting to loki, TCP port 5001
    TCP window size: 85.0 KByte (default)
    ------------------------------------------------------------
    [  3] local 192.168.1.100 port 45468 connected with 172.18.1.11 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec   757 MBytes   635 Mbits/sec
    
    ------------------------------------------------------------
    Client connecting to loki, TCP port 5001
    TCP window size: 85.0 KByte (default)
    ------------------------------------------------------------
    [  3] local 192.168.1.100 port 45462 connected with 172.18.1.11 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec   742 MBytes   622 Mbits/sec
    


  • apu1d4: throughput was arround 700mbit/s ~ 900mbit/s, when i first bought it pfsense did 500mbit/s later on that changed

    Without PowerD (hi adaptive) it was more around ~450 - 550 MBit/s but with "turned on" PowerD it Comes nearly to that numbers
    like you are naming here, ~650 MBit/s - 750 MBit/s and yes for sure that can be different from OS to OS! From FreeBSD to Linux
    as the base system, such as IPFire or ZeroShell are based on due to the better driver support.

    apu3a4: throughput is 250mbit/s ~ 300mbit/s

    The APU1D4 and APU2C4 are regular boards that was produced for the whole public, but likes in the past too, there
    are options, that you might be calling for a customized board based on the number of boards you must then buy.
    Let us say 150 - 1000 as a minimum and in former times this boards where then only for you or that customer and
    not available for the rest of us or the whole public customers, this was changing now, and so rests of such a special
    version or customized board will now also available to the public by sale. But please accept that this APU3 boards are
    optimized for LTE usage, also the LAN ports as I am right informed and this might be then driving such a throughput
    test in a total other direction! If you are using 3G and/or LTE modems inside  this will be then perhaps once more
    again differing from what you see now or get out now!

    What i do notice is that apu1d4 uses realtek(re) drivers and the apu3 uses intel(igb)
    for the igb driver: TSO LRO .. checksum offload turned on or off makes no difference

    That might be shinning as the changing point for sure and first of all, but there are also two other things
    that will be interesting now. The PowerD is bursting the APU1 cpu (T40E dual core) much more then the
    APU2 cpu (GX-412TC quad core) that is taking not really a benefit well from activating this PowerD option
    and then the igb(4) driver is now also multi cpu threading but even not the PPPoE part!

    I don´t know if that "optimized option" will be able to change or something else, but perhaps this might be
    then pushing that number somehow in the high, if there will be an option to change them. Perhaps in the BIOS!?



  • Has anyone  updated to the latest BIOS?

    I noticed in pfSense that the system is now just seen as "pfSense" where as it was "PC Engines APU2"
    See screenshots.

    Not that the name really matters to me, was just wondering others had the same.






  • @Veldkornet:

    Has anyone  updated to the latest BIOS?

    I noticed in pfSense that the system is now just seen as "pfSense" where as it was "PC Engines APU2"
    See screenshots.

    Not that the name really matters to me, was just wondering others had the same.

    I didn't even know that there was an updated BIOS until I read your reply, lol. Is there a way to be notified with these updates from pcengines?

    I'd have to wait for a week to have access to a null modem, so I can't update my APU2 until then.



  • I just updated it via SSH.

    You just need to install the "flashrom" package.

    So basically:

    pkg install flashrom
    

    Download the firmware from here

    Unzip the download and copy the .rom file to /tmp. Then flash with the following:

    flashrom -w /tmp/apu2_v4.0.7.rom -p internal
    

    Once done, reboot.

    *Edit: Updated to use the flashrom package from pfsense



  • Oh ok, I didn't know you can do that. I was referring to following the BIOS update instructions in pcengine's website using tinycore.



  • Yeah, TinyCore in anyway uses flashrom.

    If you look here on the PC Engines GitHub site, they mention that you just need flashrom.


  • Banned

    @Veldkornet:

    I just updated it via SSH.

    You just need to install the "flashrom" package, which needs some dependencies of course. At least, this is what I've just done.

    No need to use FreeBSD repo for this or mess manually with dependencies. Flashrom is being used for Netgate_Coreboot_Upgrade package so it's already there in pfSense repo.

    
    # pkg search flashrom
    flashrom-0.9.9_1               Utility for reading, writing, verifying, and erasing flash ROM chips
    
    


  • @doktornotor:

    @Veldkornet:

    I just updated it via SSH.

    You just need to install the "flashrom" package, which needs some dependencies of course. At least, this is what I've just done.

    No need to use FreeBSD repo for this or mess manually with dependencies. Flashrom is being used for Netgate_Coreboot_Upgrade package so it's already there in pfSense repo.

    
    # pkg search flashrom
    flashrom-0.9.9_1               Utility for reading, writing, verifying, and erasing flash ROM chips
    
    

    Aha, nice, that's easier. I didn't even check that :) I updated my post to show the use of the pfsense version, to avoid any unnecessary confusion.



  • So i assume I just need to run the flashrom syntax under the pfsense shell and i'm all set?



  • @kevindd992002:

    So i assume I just need to run the flashrom syntax under the pfsense shell and i'm all set?

    Yup, pretty much. I've updated my initial post showing the use of the pfsense flashrom package instead. So it's a lot simpler now.