Topton N100 Reporting 402 MHz
-
Hello everyone,
Unfortunately, I'm afraid I have very similar symptoms.
Initially I was happy to see this thread, but as time goes by and I start to doubt whether there is a solution.I have one of these Topton Intel N100 devices, 4 x 2.5 GBs Intel 226 NIC
My bios is BK-1264NP Ver: 1.5, 09/28/2023 17:23:35
Unfortunately, I don't see the "Performance temperature" settings in the BIOS :-(
However, I only noticed the possibility of changing "CPU Flex Ratio Settings"
The default value of "Disabled" can only be changed to "7". Then pfsense shows me the max speed is 691 Mhz
I have experimented with "7" and "8" (max speed is 800 Mhz ) but in all cases I still see the issue.
The issue is that during the idle mode, the pfsense is reporting quite high values of "Current CPU speed".
However, if I startiperf3test, than the speed decreases from 2 GHz to mentioned 402 MhzI can observe this also with iperf3 transfer speed which decreases from 2.5 Gbps to ~ 1 Gbps
This is weird to me. Summarizing the issue is about:
- when there is no traffic, the current CPU speed is 2GHz
- when I start iperf3 , which gives my pfsense lot of work to do :-), then CPU speed decreases to 402 MHz
-
@roxy Have you run the openssl benchmark referenced in this thread? That may help to determine whether the N100 is being throttled. Do you see CPU usage hitting 100% when your transfer speed drops?
I only got access to the power level settings in the BIOS that seemed to make the difference for me by loading a modded BIOS (also referenced in this thread). My whole LAN is only 1Gbps though so I can't run your same 2.5Gbps test as a point of comparison.
-
Yes, and the benchmark result depends how often I run it.
If the pfsense is in the idle mode, then I got very good results :-)You have chosen to measure elapsed time instead of user CPU time. Doing AES-256-CBC for 3s on 16 size blocks: 106710142 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 64 size blocks: 35718014 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 256 size blocks: 9240105 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 1024 size blocks: 2329749 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 8192 size blocks: 293649 AES-256-CBC's in 3.02s Doing AES-256-CBC for 3s on 16384 size blocks: 146186 AES-256-CBC's in 3.00s version: 3.0.12 built on: reproducible build, date unspecified options: bn(64,64) compiler: clang CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x98c007bc239ca7eb The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes AES-256-CBC 569120.76k 761984.30k 788488.96k 795220.99k 795641.59k 798370.47kBut, when I run the benchmakr several times, or if I start
iperf3, so the CPU is getting slower (~503 MHz this time) and slower, then I see the following resultsRESULTS when CPU speed is 503 MHz You have chosen to measure elapsed time instead of user CPU time. Doing AES-256-CBC for 3s on 16 size blocks: 20752586 AES-256-CBC's in 3.61s Doing AES-256-CBC for 3s on 64 size blocks: 11339018 AES-256-CBC's in 3.81s Doing AES-256-CBC for 3s on 256 size blocks: 2117270 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 1024 size blocks: 474071 AES-256-CBC's in 3.20s Doing AES-256-CBC for 3s on 8192 size blocks: 62937 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 16384 size blocks: 22461 AES-256-CBC's in 3.00s version: 3.0.12 built on: reproducible build, date unspecified options: bn(64,64) compiler: clang CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x98c007bc239ca7eb The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes AES-256-CBC 91994.15k 190346.79k 180673.71k 151925.27k 171859.97k 122667.01k -
That seems like you might be hitting thermal throttling. Check the per core temperatures.
-
I see the following output of
thermal sensorshw.acpi.thermal.tz0.temperature: 27.9C dev.cpu.3.temperature: 44.0C dev.cpu.2.temperature: 43.0C dev.cpu.1.temperature: 44.0C dev.cpu.0.temperature: 45.0CHowever, after few minutes of benchmark, the CPU speed increased to 600 - 800 MHz, and the transfer speed is about 1.9 Gbps
-
Hmm, seems fine if that's during the test.
-
S stephenw10 referenced this topic on
-
It's quite fine, but not bad ;-)
I really would like to have 2.5 GBps :-( all the time ,
I see 2.35 at the beginning and 1.0 after 15 seconds and 1.8 back after 30 secondsThis could be the result of bad cooling (but temperature is about 45 degrees).
I tried to simplify pfsense firewall rules (single rule with Pass all traffic) but didnt get 2.5I am wondering if NetGate hw really gets 2.5 GBps. I was considering to buy NetGate 4200 to check, but I habe no spare $500
Connecting to host 192.168.18.100, port 5201 [ 5] local 10.0.2.21 port 56216 connected to 192.168.18.100 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 283 MBytes 2.36 Gbits/sec [ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 2.00-3.01 sec 282 MBytes 2.35 Gbits/sec [ 5] 3.01-4.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 4.01-5.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 5.01-6.00 sec 275 MBytes 2.31 Gbits/sec [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 7.00-8.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 8.01-9.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 9.01-10.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 10.01-11.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 11.00-12.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 12.01-13.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 13.00-14.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 14.00-15.01 sec 282 MBytes 2.35 Gbits/sec [ 5] 15.01-16.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 16.00-17.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 17.00-18.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 18.01-19.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 19.01-20.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 20.01-21.01 sec 196 MBytes 1.65 Gbits/sec [ 5] 21.01-22.00 sec 226 MBytes 1.91 Gbits/sec [ 5] 22.00-23.01 sec 204 MBytes 1.71 Gbits/sec [ 5] 23.01-24.00 sec 190 MBytes 1.60 Gbits/sec [ 5] 24.00-25.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 25.00-26.00 sec 158 MBytes 1.32 Gbits/sec [ 5] 26.00-27.00 sec 158 MBytes 1.32 Gbits/sec [ 5] 27.00-28.01 sec 157 MBytes 1.32 Gbits/sec [ 5] 28.01-29.01 sec 155 MBytes 1.30 Gbits/sec [ 5] 29.01-30.01 sec 153 MBytes 1.28 Gbits/sec [ 5] 30.01-31.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 31.00-32.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 32.00-33.00 sec 158 MBytes 1.32 Gbits/sec [ 5] 33.00-34.00 sec 157 MBytes 1.32 Gbits/sec [ 5] 34.00-35.00 sec 113 MBytes 947 Mbits/sec [ 5] 35.00-36.01 sec 103 MBytes 860 Mbits/sec [ 5] 36.01-37.01 sec 187 MBytes 1.57 Gbits/sec [ 5] 37.01-38.01 sec 193 MBytes 1.62 Gbits/sec [ 5] 38.01-39.01 sec 213 MBytes 1.79 Gbits/sec [ 5] 39.01-40.01 sec 214 MBytes 1.80 Gbits/sec [ 5] 40.01-41.01 sec 214 MBytes 1.80 Gbits/sec [ 5] 41.01-42.01 sec 213 MBytes 1.79 Gbits/sec [ 5] 42.01-43.01 sec 212 MBytes 1.78 Gbits/sec [ 5] 43.01-44.01 sec 214 MBytes 1.80 Gbits/sec [ 5] 44.01-45.00 sec 213 MBytes 1.80 Gbits/sec [ 5] 45.00-46.00 sec 213 MBytes 1.79 Gbits/sec [ 5] 46.00-47.00 sec 214 MBytes 1.79 Gbits/sec [ 5] 47.00-48.01 sec 214 MBytes 1.79 Gbits/sec [ 5] 48.01-49.00 sec 212 MBytes 1.79 Gbits/sec [ 5] 49.00-50.01 sec 199 MBytes 1.66 Gbits/sec [ 5] 50.01-51.01 sec 215 MBytes 1.80 Gbits/sec [ 5] 51.01-52.01 sec 217 MBytes 1.82 Gbits/sec [ 5] 52.01-53.01 sec 213 MBytes 1.79 Gbits/sec [ 5] 53.01-54.00 sec 213 MBytes 1.79 Gbits/sec [ 5] 54.00-55.00 sec 214 MBytes 1.79 Gbits/sec [ 5] 55.00-56.00 sec 218 MBytes 1.83 Gbits/sec -
You see retransmissions at the other end when that's shown?
Do you see dropped packets in the interface stats?
-
I have iperf sunning in
SERVICESand the client inDMZ
I am not sure how to check number of dropped packages. I think the rules are "Pass" - so I should not see any dropped packets, right?
-
One end of the iperf test usually reports retransmission counts. In a nice result there should be no retransmissions.
Interface stats can be seen in Status > Interfaces or from
netstat -i.One other user seeing this recently had disabled flow control on the NIC and required it on that particular connection. Some devices just don't play nicely with/without it so it's worth testing.
-
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll igc2 1500 <Link#3> 00:d0:b4:02:1c:b8 645357575 0 0 21639068 0 0 igc2 - fe80::%igc2/64 fe80::2d0:b4ff:fe02:1cb8%igc2 0 - - 1 - - igc2 - 10.0.2.0/24 10.0.2.1 24 - - 28181 - - igc3 1500 <Link#4> 00:d0:b4:02:1c:b9 21751498 0 0 646024233 0 0 igc3 - fe80::%igc3/64 fe80::2d0:b4ff:fe02:1cb9%igc3 0 - - 0 - - igc3 - 192.168.18.0/24 192.168.18.1 3130 - - 3115 - -
-
After 10 minutes the speed test converges to 2 Gbps ;-)
and the power consumption increased by ~ 5W
-
Hmm, well that seems suspicious!
No retries/retransmissions shown at either end? You might have to run it verbose perhaps. I expect to see something like:
steve@steve-NUC9i9QNX:~$ iperf3 -c 4200.stevew.lan Connecting to host 4200.stevew.lan, port 5201 [ 5] local 172.21.16.8 port 43826 connected to 172.21.16.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 114 MBytes 958 Mbits/sec 0 397 KBytes [ 5] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0 417 KBytes [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 417 KBytes [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 417 KBytes [ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec 0 417 KBytes [ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 417 KBytes [ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 0 417 KBytes [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 0 417 KBytes [ 5] 8.00-9.00 sec 113 MBytes 945 Mbits/sec 0 417 KBytes [ 5] 9.00-10.00 sec 112 MBytes 936 Mbits/sec 0 417 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver iperf Done.Check for ASPM in the BIOS if you haven't already. That can cause a lot of problems.
-
I will chack the ASPM tomorrow, but I am afraid I have no option in my version.
Here is the result of
iperfrunning for 10 minutes❯ iperf3 -c 192.168.18.200 -t 600 ─╯ Connecting to host 192.168.18.200, port 5201 [ 5] local 10.0.2.21 port 56698 connected to 192.168.18.200 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 284 MBytes 2.37 Gbits/sec [ 5] 1.01-2.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 2.01-3.00 sec 279 MBytes 2.35 Gbits/sec [ 5] 3.00-4.01 sec 282 MBytes 2.35 Gbits/sec [ 5] 4.01-5.01 sec 279 MBytes 2.34 Gbits/sec [ 5] 5.01-6.00 sec 279 MBytes 2.35 Gbits/sec . . . [ 5] 594.00-595.01 sec 182 MBytes 1.52 Gbits/sec [ 5] 595.01-596.01 sec 178 MBytes 1.50 Gbits/sec [ 5] 596.01-597.00 sec 174 MBytes 1.47 Gbits/sec [ 5] 597.00-598.00 sec 179 MBytes 1.50 Gbits/sec [ 5] 598.00-599.01 sec 170 MBytes 1.42 Gbits/sec [ 5] 599.01-600.00 sec 168 MBytes 1.42 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-600.00 sec 108 GBytes 1.54 Gbits/sec sender [ 5] 0.00-600.01 sec 108 GBytes 1.54 Gbits/sec receiver -
Try running with
-Vfor more verbose output. It would be interesting to see if it's reducing he window size because or retries. That is common. -
@stephenw10
iperf -V#server side - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] (sender statistics not available) [ 5] 0.00-60.00 sec 12.9 GBytes 1.84 Gbits/sec receiver rcv_tcp_congestion cubic iperf 3.9 Linux iperf 6.5.13-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-3 (2024-03-20T10:45Z) x86_64 # client side [ 5] 57.00-58.00 sec 222 MBytes 1.86 Gbits/sec [ 5] 58.00-59.01 sec 237 MBytes 1.98 Gbits/sec [ 5] 59.01-60.00 sec 223 MBytes 1.88 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] 0.00-60.00 sec 12.9 GBytes 1.84 Gbits/sec sender [ 5] 0.00-60.00 sec 12.9 GBytes 1.84 Gbits/sec receiver CPU Utilization: local/sender 8.1% (0.1%u/8.0%s), remote/receiver 7.6% (0.3%u/7.4%s) rcv_tcp_congestion cubic -
Odd. Your output seems to have a space where retr and cwnd would be shown, but isn't. Same iperf version.
-
@roxy Apologies if I've somewhat lost the thread here but do you have a control, so to speak? That is to say, have you run iperf to the pfSense box from more than one host on your LAN and seen the same results? And, ideally, have you also run iperf tests directly between two hosts on your LAN and seen sustained 2.5Gbps throughput? Just trying to isolate variables. I wish I could help because it sounds like we have nearly (maybe exactly) identical pfSense hardware, except my whole LAN is still 1Gbps. I noticed from BIOS screen shots I took that I actually have ASPM enabled; haven't noticed any ill effects, but may disable it when I get a chance.
Also have you tried a UDP iperf test? I think
iperf3 -c <pfsense_ip> -u -b 2.5Gshould work. I'm not sure what we'll necessarily learn if that goes worse/same/better but it's another data point to gather. -
@roxy I think we have the same hardware and BIOS. I am able to sustain about ~2.3 Gbits/sec constantly using iperf now that I have new 2.5Gbe switches. Based on your openssl benchmarks also getting slower over time while you say temperatures are fine I'd guess you are hitting power limits just like @TheNarc seemed to be doing. I'm not sure why my box never experienced these issues with the same BIOS version but I guess the default settings which are normally hidden might be different. You might have to flash the same custom BIOS @TheNarc did and play with the PL1 settings it can unlock.
An explanation could be that the manufacturer of these boxes have realized they are not really built to passively cool a sustained heatload from these CPUs and have therefore added low sustained power limits (PL1) to remove crashes (which is an issue I experienced instead). So even though my box was performing much better out-of-the-box it would instead crash from sustained high temperatures and the only solution I found was to add an external fan blowing on the case. The CPU is fine with much higher temperatures but the other components in the very small and very badly ventilated box will cook the RAM and NVMe drives inside making them unstable because of the high ambient temperatures the CPU causes.
-
Of course, I am open to talk.
Thets right, we may have both similar hardware
First, let me paste the output on iperf3 (client, MacOS) with your options
-u -b 2.5G
I see stable 2.4 Gbps
for 60 secs.❯ iperf3 -c 192.168.18.200 -t 60 -u -b 2.5G -V ─╯ iperf 3.16 Darwin roxys-mbp.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64 Control connection MSS 1448 Setting UDP block size to 1448 Time: Wed, 03 Apr 2024 19:32:03 UTC Connecting to host 192.168.18.200, port 5201 Cookie: q4zcsvcxyn4vkw6ppthdaoxjmthkmldp3eai Target Bitrate: 2500000000 [ 5] local 10.0.2.21 port 61476 connected to 192.168.18.200 port 5201 Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 287 MBytes 2.40 Gbits/sec 207546 [ 5] 1.00-2.00 sec 285 MBytes 2.39 Gbits/sec 206445 [ 5] 2.00-3.00 sec 285 MBytes 2.39 Gbits/sec 206381 [ 5] 3.00-4.00 sec 285 MBytes 2.39 Gbits/sec 206378 [ 5] 4.00-5.00 sec 285 MBytes 2.39 Gbits/sec 206381 [ 5] 5.00-6.00 sec 285 MBytes 2.39 Gbits/sec 206333 [ 5] 6.00-7.00 sec 285 MBytes 2.39 Gbits/sec 206466 [ 5] 7.00-8.00 sec 285 MBytes 2.39 Gbits/sec 206470 [ 5] 8.00-9.00 sec 285 MBytes 2.39 Gbits/sec 206268 [ 5] 9.00-10.00 sec 285 MBytes 2.39 Gbits/sec 206338 [ 5] 10.00-11.00 sec 285 MBytes 2.39 Gbits/sec 206531 [ 5] 11.00-12.00 sec 285 MBytes 2.39 Gbits/sec 206339 [ 5] 12.00-13.00 sec 285 MBytes 2.39 Gbits/sec 206485 [ 5] 13.00-14.00 sec 285 MBytes 2.39 Gbits/sec 206331 [ 5] 14.00-15.00 sec 285 MBytes 2.39 Gbits/sec 206399 [ 5] 15.00-16.00 sec 285 MBytes 2.39 Gbits/sec 206370 [ 5] 16.00-17.00 sec 285 MBytes 2.39 Gbits/sec 206357 [ 5] 17.00-18.00 sec 285 MBytes 2.39 Gbits/sec 206353 [ 5] 18.00-19.00 sec 285 MBytes 2.39 Gbits/sec 206394 [ 5] 19.00-20.00 sec 285 MBytes 2.39 Gbits/sec 206312 [ 5] 20.00-21.00 sec 285 MBytes 2.39 Gbits/sec 206508 [ 5] 21.00-22.00 sec 285 MBytes 2.39 Gbits/sec 206282 [ 5] 22.00-23.00 sec 285 MBytes 2.39 Gbits/sec 206509 [ 5] 23.00-24.00 sec 285 MBytes 2.39 Gbits/sec 206332 [ 5] 24.00-25.00 sec 285 MBytes 2.39 Gbits/sec 206459 [ 5] 25.00-26.00 sec 285 MBytes 2.39 Gbits/sec 206396 [ 5] 26.00-27.00 sec 285 MBytes 2.39 Gbits/sec 206338 [ 5] 27.00-28.00 sec 285 MBytes 2.39 Gbits/sec 206402 [ 5] 28.00-29.00 sec 285 MBytes 2.39 Gbits/sec 206351 [ 5] 29.00-30.00 sec 285 MBytes 2.39 Gbits/sec 206362 [ 5] 30.00-31.00 sec 285 MBytes 2.39 Gbits/sec 206462 [ 5] 31.00-32.00 sec 285 MBytes 2.39 Gbits/sec 206355 [ 5] 32.00-33.00 sec 285 MBytes 2.39 Gbits/sec 206347 [ 5] 33.00-34.00 sec 285 MBytes 2.39 Gbits/sec 206477 [ 5] 34.00-35.00 sec 285 MBytes 2.39 Gbits/sec 206256 [ 5] 35.00-36.00 sec 285 MBytes 2.39 Gbits/sec 206372 [ 5] 36.00-37.00 sec 285 MBytes 2.39 Gbits/sec 206484 [ 5] 37.00-38.00 sec 285 MBytes 2.39 Gbits/sec 206253 [ 5] 38.00-39.00 sec 285 MBytes 2.39 Gbits/sec 206534 [ 5] 39.00-40.00 sec 285 MBytes 2.39 Gbits/sec 206246 [ 5] 40.00-41.00 sec 285 MBytes 2.39 Gbits/sec 206405 [ 5] 41.00-42.00 sec 285 MBytes 2.39 Gbits/sec 206485 [ 5] 42.00-43.00 sec 285 MBytes 2.39 Gbits/sec 206312 [ 5] 43.00-44.00 sec 285 MBytes 2.39 Gbits/sec 206413 [ 5] 44.00-45.00 sec 285 MBytes 2.39 Gbits/sec 206487 [ 5] 45.00-46.00 sec 285 MBytes 2.39 Gbits/sec 206258 [ 5] 46.00-47.00 sec 285 MBytes 2.39 Gbits/sec 206448 [ 5] 47.00-48.00 sec 285 MBytes 2.39 Gbits/sec 206659 [ 5] 48.00-49.00 sec 285 MBytes 2.39 Gbits/sec 206452 [ 5] 49.00-50.00 sec 285 MBytes 2.39 Gbits/sec 206271 [ 5] 50.00-51.00 sec 285 MBytes 2.39 Gbits/sec 206343 [ 5] 51.00-52.00 sec 285 MBytes 2.39 Gbits/sec 206516 [ 5] 52.00-53.00 sec 285 MBytes 2.39 Gbits/sec 206392 [ 5] 53.00-54.00 sec 285 MBytes 2.39 Gbits/sec 206273 [ 5] 54.00-55.00 sec 285 MBytes 2.39 Gbits/sec 206523 [ 5] 55.00-56.00 sec 285 MBytes 2.39 Gbits/sec 206201 [ 5] 56.00-57.00 sec 285 MBytes 2.39 Gbits/sec 206410 [ 5] 57.00-58.00 sec 285 MBytes 2.39 Gbits/sec 206433 [ 5] 58.00-59.00 sec 285 MBytes 2.39 Gbits/sec 206317 [ 5] 59.00-60.00 sec 285 MBytes 2.39 Gbits/sec 206401 - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-60.00 sec 16.7 GBytes 2.39 Gbits/sec 0.000 ms 0/12384601 (0%) sender [ 5] 0.00-60.01 sec 10.0 GBytes 1.44 Gbits/sec 0.014 ms 0/12379713 (0%) receiver CPU Utilization: local/sender 90.9% (18.3%u/72.5%s), remote/receiver 26.0% (2.8%u/23.1%s) iperf Done.The option
-uis using UDP rather than TCP.
When I runiperf3without-u, i.e. with TCP, I still the speed decreased to 1.2 Gbps (but if if the test takes longer, the speed converges to 1.9 GBps)❯ iperf3 -c 192.168.18.200 -t 60 -V ─╯ iperf 3.16 Darwin roxys-mbp.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64 Control connection MSS 1448 Time: Wed, 03 Apr 2024 19:33:50 UTC Connecting to host 192.168.18.200, port 5201 Cookie: oybbl36xu7jrh7vmufngybr4arqe6fdymo7m TCP MSS: 1448 (default) [ 5] local 10.0.2.21 port 57879 connected to 192.168.18.200 port 5201 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 283 MBytes 2.37 Gbits/sec [ 5] 1.00-2.00 sec 282 MBytes 2.35 Gbits/sec [ 5] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 4.00-5.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 5.00-6.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 10.00-11.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 11.00-12.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 12.00-13.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 13.00-14.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 14.00-15.00 sec 279 MBytes 2.34 Gbits/sec [ 5] 15.00-16.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 16.00-17.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 17.00-18.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 18.00-19.00 sec 183 MBytes 1.53 Gbits/sec [ 5] 19.00-20.00 sec 246 MBytes 2.07 Gbits/sec [ 5] 20.00-21.00 sec 176 MBytes 1.48 Gbits/sec [ 5] 21.00-22.00 sec 154 MBytes 1.29 Gbits/sec [ 5] 22.00-23.00 sec 134 MBytes 1.12 Gbits/sec [ 5] 23.00-24.00 sec 138 MBytes 1.16 Gbits/sec [ 5] 24.00-25.00 sec 146 MBytes 1.22 Gbits/sec [ 5] 25.00-26.00 sec 147 MBytes 1.24 Gbits/sec [ 5] 26.00-27.00 sec 150 MBytes 1.26 Gbits/sec [ 5] 27.00-28.00 sec 150 MBytes 1.26 Gbits/sec [ 5] 28.00-29.00 sec 141 MBytes 1.19 Gbits/sec [ 5] 29.00-30.00 sec 155 MBytes 1.30 Gbits/sec [ 5] 30.00-31.00 sec 134 MBytes 1.12 Gbits/sec [ 5] 31.00-32.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 32.00-33.00 sec 154 MBytes 1.29 Gbits/sec [ 5] 33.00-34.01 sec 154 MBytes 1.29 Gbits/sec [ 5] 34.01-35.00 sec 154 MBytes 1.30 Gbits/sec [ 5] 35.00-36.01 sec 153 MBytes 1.28 Gbits/sec [ 5] 36.01-37.01 sec 155 MBytes 1.30 Gbits/sec [ 5] 37.01-38.00 sec 149 MBytes 1.26 Gbits/sec [ 5] 38.00-39.00 sec 148 MBytes 1.23 Gbits/sec [ 5] 39.00-40.00 sec 155 MBytes 1.30 Gbits/sec [ 5] 40.00-41.00 sec 155 MBytes 1.30 Gbits/sec [ 5] 41.00-42.00 sec 154 MBytes 1.29 Gbits/sec [ 5] 42.00-43.00 sec 152 MBytes 1.27 Gbits/sec [ 5] 43.00-44.00 sec 155 MBytes 1.29 Gbits/sec [ 5] 44.00-45.00 sec 178 MBytes 1.50 Gbits/sec [ 5] 45.00-46.00 sec 178 MBytes 1.50 Gbits/sec [ 5] 46.00-47.00 sec 192 MBytes 1.61 Gbits/sec [ 5] 47.00-48.01 sec 214 MBytes 1.79 Gbits/sec [ 5] 48.01-49.00 sec 200 MBytes 1.69 Gbits/sec [ 5] 49.00-50.01 sec 221 MBytes 1.85 Gbits/sec [ 5] 50.01-51.00 sec 209 MBytes 1.76 Gbits/sec [ 5] 51.00-52.01 sec 190 MBytes 1.59 Gbits/sec [ 5] 52.01-53.01 sec 188 MBytes 1.58 Gbits/sec [ 5] 53.01-54.00 sec 205 MBytes 1.73 Gbits/sec [ 5] 54.00-55.01 sec 206 MBytes 1.72 Gbits/sec [ 5] 55.01-56.00 sec 208 MBytes 1.75 Gbits/sec [ 5] 56.00-57.01 sec 211 MBytes 1.76 Gbits/sec [ 5] 57.01-58.00 sec 189 MBytes 1.58 Gbits/sec [ 5] 58.00-59.00 sec 199 MBytes 1.67 Gbits/sec [ 5] 59.00-60.00 sec 208 MBytes 1.74 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] 0.00-60.00 sec 12.0 GBytes 1.72 Gbits/sec sender [ 5] 0.00-60.01 sec 12.0 GBytes 1.72 Gbits/sec receiver CPU Utilization: local/sender 7.9% (0.1%u/7.8%s), remote/receiver 4.5% (0.2%u/4.3%s) rcv_tcp_congestion cubic iperf Done.Let me describe my setup.
I have two Intel N100 :-) machines. Both have 4 x 2.5 Gbps Intel NIC i226V . Both from TopTon ;-)
One is quite fresh, and I see diferent BIOS-es on both of them. First machine (older one) does have Performance in bios Advanced tab. Second machine (newer on) does not :-(I have installed Proxmox on first (older) machine, and some time ago I experimented with pfsense running in VM.
But now, I have installed pfsense on bare metal second machine (newer one).The pfsense NICs are physical ports and I named them as follows: (WAN) ISP, LAN, DMZ and SERVICES.
The proxmox ports are as follows- vmbr1 (SERVICES) : enp4s0
- vmbr2 (LAN) : enp2s0
- vmbr3 (DMZ): enp1s0, enp3s0 - yeas, I have bridged two NICs
enp1s0 is connected to pfsense's DMZ
enp3s0 is connected to my MacBook's eth card (usb 2.5 Gbps)I have started two Linux Containerx in Proxmox (ubuntu 22.04 + iperf3 running):
- Container1: connected with vmbr3 (DMZ)
- Container2: connected with vmbr1 (SERVICES)
When I run
iperf3 -c <ip>from my Macboox I am in the DMZ network, so the traffic goes to Proxmox's enp3s0 and directly to vmb3 and to theiperf3server running in VM (Container1).
Then I see no traffic in pfsense. This is ok to me. I would expect such behaviour, since this looks like connecting two devices in the same network (connected with vmbr - switch).
But when I askiperf3 -c <ip>withipfromSERVICESnetwork, then I see ..... what I presented above.
The traffic goes to proxmox first, but there is no bridge, and the traffic has to go from Proxmox machine to my pfsense through
Macbookx -> proxmox enp3s0 : vmbr3 : enp1s0 -> pfsense DMZ --- firewall rules --- pfsense SERVICES -> proxmox enp4s0 : vmbr1 --> Container 2I think this really tests if my pfsense is able to handle 2.5 Gbps
I tried to simply my pfsense rules onDMZinterface, but I didn't see any difference if there was only one rulePass Any Traffic