PC Engines apu2 experiences
-
@kinch
Try disabling ICMP Redirects to enable the faster tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0) -
@Qinn said in PC Engines apu2 experiences:
@sikita The whole filesystem zfs is running on 16GB mSATA SSD with ZFS only /var/run
4% of 3.4MiB - ufs in RAMOk, thanks for a hint
-
Hi,
First id like to say thanks for everyone advice.
Id like to ask for a little help from you.
Im having issues with slow LAN interface compared to the WAN.
I have a 100Mbit PPPoe HFC connection. When running a speed test from the apu4c4 or directly plugged in to the HFC box, i get 98Mbit but if i run it from my PC that's behind the apu4c4 its only at 71Mbit no matter what i do.
Ive tried:
- Swapping Ethernet cables
- Plugging directly in to LAN Port 2 and 3
- https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/
- disabling ICMP Redirects to enable the faster tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)
- Upgraded to 2.4.5-RELEASE-p1 (amd64) from 2.4.4
- Upgraded the BIOS to: Version: v4.11.0.4
- Factory reset with no packages enabled
All with no change to my LAN speed. I'm all out of ideas here, can anyone offer any more advice on what i can try next.
Thanks
-
How is your CPU utiluzation when you run your test?
Can you try and update the BIOS to 4.12.0.2 and see if this helps? -
@Stella_161 This is specific to your box, not a general issue with the devices themselves so you would get a much better response opening a general support thread for this. When you do, you'll need to give more detail, specifically what packages are installed and how you're performing the test. Try a mix of speedtest.net and fast.com and see what you get. Make sure to do them both with an ad-blocker running on your PC. I've seen many instances where advertisements can eat up a CPU on a computer and give artificially slow speed results.
I have many of these boxes that are doing up to 500Mbps so the hardware is more than capable. That being said, @sToRmInG is right in that the first place to look would be the CPU %. Something may be chewing up CPU cycles on either the APU2 or the laptop you are using. Obvious suspects could be Snort / Suricata, Squid / SquidGuard, and pfBlockerNG.
-
Thanks Guys, Ive opened a new thread for this.
Ive tried upgrading the BIOS with no change. After the BIOS upgrade i did a factory reset, no packages have only setup PPPoe and default LAN with ad blocker enabled on PC. Still only 71Mbit~.
CPU usage is minimal when running from PC: Speedtest.net and Fast.com with the same result
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 155 ki31 0K 64K RUN 3 42:59 100.00% [idle{idle: cpu3}]
11 root 155 ki31 0K 64K CPU1 1 42:57 100.00% [idle{idle: cpu1}]
11 root 155 ki31 0K 64K CPU2 2 42:56 98.49% [idle{idle: cpu2}]
11 root 155 ki31 0K 64K CPU0 0 42:42 88.96% [idle{idle: cpu0}]
12 root -92 - 0K 496K CPU0 0 0:31 14.70% [intr{irq256: igb0:que 0}]
12 root -92 - 0K 496K WAIT 2 0:08 5.66% [intr{irq259: igb1:que 0}]
12 root -92 - 0K 496K WAIT 3 0:07 1.66% [intr{irq260: igb1:que 1}]
CPU when running Speed Test CLI on pfsense:
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 155 ki31 0K 64K CPU1 1 47:13 86.77% [idle{idle: cpu1}]
11 root 155 ki31 0K 64K RUN 3 47:12 82.57% [idle{idle: cpu3}]
11 root 155 ki31 0K 64K CPU2 2 47:09 79.88% [idle{idle: cpu2}]
11 root 155 ki31 0K 64K RUN 0 46:46 71.29% [idle{idle: cpu0}]
12 root -92 - 0K 496K WAIT 0 0:50 33.25% [intr{irq256: igb0:que 0}]
52611 root 74 0 135M 71416K CPU3 3 0:02 7.96% /usr/local/bin/python3.7 /usr/local/bin/speedtest{python3.7}
Cheers
-
@Stella_161 Just to be sure, did you halt the system and then pulled the power plug, wait for a minute and then re-plug the power, this should be done after every bios update.
-
If I use a USB2.0 cable to the internal header of the APU2C4, how do I route it so that I can access it externally from the official black case provided by PCEngines?
-
@kevindd992002 said in PC Engines apu2 experiences:
If I use a USB2.0 cable to the internal header of the APU2C4, how do I route it so that I can access it externally from the official black case provided by PCEngines?
When I loaded from the USB header, I had the top of the case off. I think you'd have to modify the case if you wanted to put the top back on.
-
@dotdash said in PC Engines apu2 experiences:
@kevindd992002 said in PC Engines apu2 experiences:
If I use a USB2.0 cable to the internal header of the APU2C4, how do I route it so that I can access it externally from the official black case provided by PCEngines?
When I loaded from the USB header, I had the top of the case off. I think you'd have to modify the case if you wanted to put the top back on.
Yeah, drill through it or something. I would love to just use the USB2.0 header with external USB ports than wait for the devs to fix that long-standing USB3 issue.
-
@veldkornet said in PC Engines apu2 experiences:
@kinch said in PC Engines apu2 experiences:
I've examined it and found the following:
iperf3 LAB
No matter if APU2/3/4 I get a throughput with single connection of 750-850Mbit/s (LAN 2 OPT)With 2 or more connections I get about 945Mbit/s.
A Speedtest.net gives the value 934Mbit/s UP/DOWN
I have tested the following BIOS versions:
V4.9.0.2
V4.11.0.2
V4.12.0.1All the same.
But what I noticed, tested on APU2 and APU3, that pfsense version 2.4.5 does not reach the same throughput as 2.4.4-p3.
With Single Connection pfsense 2.4.5 reaches "only" about 400-500Mbit/s. With pfsense 2.4.5 I can only reach 940Mbps with 8 connections.I also noted something weird with 2.4.5.
My iPerf speeds were also much lower than normal. What I did to fix it was:
- disabling hardware offload (check the check boxes)
- save, no reboot (iPerf results the same)
- re-enabling hardware offload (clear the check boxes)
- save, no reboot (iPerf results back up to the expected levels)
The only thing I did between having good iPerf results and slow ones was reboot the device. It's as if after a reboot, it forgot that hardware offload was enabled, and I just needed to set it again.
Do you have any comments on this? As we discussed before, here are the tweaks I have:
/boot/loader.conf.local contains:
hw.igb.rx_process_limit=-1
Also, it looks like the guys here updated their article but they are still insisting to enable all three HW offloads as they see that it gives better speeds.
And what is the recommended setting for hardware checksum offload? Checked (disabled) or unchecked (enabled)?
I'm asking again because my ISP connection is at 400/400 now and I'm thinking of upgrading to 800/800 soon. I want to know if my APU2C4 can handle these speeds with the tweaks that we have.
Which of the HW offloads did you enable (unchecked) to get significantly better multi-connection speeds?
What is your comment on the HW offload settings? I see that you had a comment against the post of Veldkornet where he enabled those settings to get better speeds. If he saw better speeds with those enabled, is that an inaccurate test or something because it only helps the client and not end-to-end? Sorry if I misunderstood.
-
@kevindd992002 Why only the receive (rx) and not the transmit (tx) in the loader.conf.local file and why did you not uncheck?
Hardware TCP Segmentation Offloading
Hardware Large Receive Offloading -
@qinn said in PC Engines apu2 experiences:
@kevindd992002 Why only the receive (rx) and not the transmit (tx) in the loader.conf.local file and why did you not uncheck?
Hardware TCP Segmentation Offloading
Hardware Large Receive OffloadingAnswers to your questions:
https://forum.netgate.com/post/908003
-
@kevindd992002 You are refring to my own reply, but these settings are from a older date and referred to pre pfSense version 2.4.5-p1
You should now have these and then reboot
https://forum.netgate.com/topic/95148/pc-engines-apu2-experiences/384
-
@qinn said in PC Engines apu2 experiences:
@kevindd992002 You are refring to my own reply, but these settings are from a older date and referred to pre pfSense version 2.4.5-p1
You should now have these and then reboot
https://forum.netgate.com/topic/95148/pc-engines-apu2-experiences/384
I'm still confused. I wasn't referring to your reply. I was referring to @dugeem 's reply when we discussed this.
Again, read https://forum.netgate.com/post/908003
Let me quote some parts of his reply:
"Firstly TSO & LRO should always be disabled on routers. Netgate recommend this (hence pfSense defaults) as do others. BSDRP have even tested this and found routing performance drop negligible from enabling TSO & LRO (see link below)."
"hw.igb.tx_process_limit already defaults to -1 so no need to change this."
So those are the answers to your questions. Also, like I mentioned in my reply today:
"Also, it looks like the guys here updated their article but they are still insisting to enable all three HW offloads as they see that it gives better speeds."
So, to summarize, dugeem explained very well that TSO and LRO needs to be disabled (so checked) and that the teklager article is wrong. With their updated article, they are still insisting to keep those three offloads enabled.
Everything clear now?
-
@kevindd992002 said in PC Engines apu2 experiences:
Also, it looks like the guys here updated their article but they are still insisting to enable all three HW offloads as they see that it gives better speeds.
Hi,
Because of the behavior of the TSO and LRO, it is important ...
This is an endpoint question and not a routing and / or a router NIC setup ...this will definitely help you understand the behavior (FreeBSD):
https://calomel.org/freebsd_network_tuning.html
https://calomel.org/network_performance.html
https://blog.cloudflare.com/how-to-receive-a-million-packets/TSO and LRO are not good, if you are a router, but if you are an endpoint you can use them...
remember you're talking about an APU (pcEngines) stuff and an i211 thing, you'll do more if you could increase your PCie bus speed and / or RAM
With such a MOBO, the opportunity is not given...
(dmesg | grep -i msi)which can help a lot, disabling EEE and FC on igb0-3
Something like:
-
@daddygo said in PC Engines apu2 experiences:
@kevindd992002 said in PC Engines apu2 experiences:
Also, it looks like the guys here updated their article but they are still insisting to enable all three HW offloads as they see that it gives better speeds.
Hi,
Because of the behavior of the TSO and LRO, it is important ...
This is an endpoint question and not a routing and / or a router NIC setup ...this will definitely help you understand the behavior (FreeBSD):
https://calomel.org/freebsd_network_tuning.html
https://calomel.org/network_performance.html
https://blog.cloudflare.com/how-to-receive-a-million-packets/TSO and LRO are not good, if you are a router, but if you are an endpoint you can use them...
remember you're talking about an APU (pcEngines) stuff and an i211 thing, you'll do more if you could increase your PCie bus speed and / or RAM
With such a MOBO, the opportunity is not given...
(dmesg | grep -i msi)which can help a lot, disabling EEE and FC on igb0-3
Something like:
So are you recommending that TSO and LRO be enabled in this use case?
Also, I would really want to know what all those system tunables do before I implement them blindly to my system.
@dugeem any comment on this since you were the one who explained the routing concept for these parameters in the first place?
-
@kevindd992002 said in PC Engines apu2 experiences:
So are you recommending that TSO and LRO be enabled in this use case?
So we have been using APU4d4 stuff at the endpoints of our users and colleagues for years, so I can tell you first hand that their performance is finite....
The answer to your question is to never enable TSO and LRO on a router
+++edit:
this is an old thread, read my posts aboveBTW:
why do you think the TSO and LRO are disabled after a clean pfSense install (by Netgate) -
@daddygo said in PC Engines apu2 experiences:
@kevindd992002 said in PC Engines apu2 experiences:
So are you recommending that TSO and LRO be enabled in this use case?
So we have been using APU4d4 stuff at the endpoints of our users and colleagues for years, so I can tell you first hand that their performance is finite....
The answer to your question is to never enable TSO and LRO on a router
Yes, I'm aware that you are well-experienced with this boards. And I believe you. I'm just making sure that I'm running these APU2C4's to their max potential. And as you can see in my post yesterday, I have TSO and LRO disabled since early this year.
Also, I transitioned from using OpenVPN to IKEv2 IPsec and I can't believe my own eyes how fast IPsec is. I had to switch because I upgraded the Internet subscription of my remote site to 100/100. In any case, do you have any ideas on my question here?
-
@daddygo said in PC Engines apu2 experiences:
@kevindd992002 said in PC Engines apu2 experiences:
So are you recommending that TSO and LRO be enabled in this use case?
So we have been using APU4d4 stuff at the endpoints of our users and colleagues for years, so I can tell you first hand that their performance is finite....
The answer to your question is to never enable TSO and LRO on a router
+++edit:
this is an old thread, read my posts aboveBTW:
why do you think the TSO and LRO are disabled after a clean pfSense install (by Netgate)Where did I say that TSO and LRO are disabled after a clean install of pfSense? I never said that.
