Topton N100 Reporting 402 MHz
-
@roxy Apologies if I've somewhat lost the thread here but do you have a control, so to speak? That is to say, have you run iperf to the pfSense box from more than one host on your LAN and seen the same results? And, ideally, have you also run iperf tests directly between two hosts on your LAN and seen sustained 2.5Gbps throughput? Just trying to isolate variables. I wish I could help because it sounds like we have nearly (maybe exactly) identical pfSense hardware, except my whole LAN is still 1Gbps. I noticed from BIOS screen shots I took that I actually have ASPM enabled; haven't noticed any ill effects, but may disable it when I get a chance.
Also have you tried a UDP iperf test? I think
iperf3 -c <pfsense_ip> -u -b 2.5Gshould work. I'm not sure what we'll necessarily learn if that goes worse/same/better but it's another data point to gather. -
@roxy I think we have the same hardware and BIOS. I am able to sustain about ~2.3 Gbits/sec constantly using iperf now that I have new 2.5Gbe switches. Based on your openssl benchmarks also getting slower over time while you say temperatures are fine I'd guess you are hitting power limits just like @TheNarc seemed to be doing. I'm not sure why my box never experienced these issues with the same BIOS version but I guess the default settings which are normally hidden might be different. You might have to flash the same custom BIOS @TheNarc did and play with the PL1 settings it can unlock.
An explanation could be that the manufacturer of these boxes have realized they are not really built to passively cool a sustained heatload from these CPUs and have therefore added low sustained power limits (PL1) to remove crashes (which is an issue I experienced instead). So even though my box was performing much better out-of-the-box it would instead crash from sustained high temperatures and the only solution I found was to add an external fan blowing on the case. The CPU is fine with much higher temperatures but the other components in the very small and very badly ventilated box will cook the RAM and NVMe drives inside making them unstable because of the high ambient temperatures the CPU causes.
-
Of course, I am open to talk.
Thets right, we may have both similar hardware
First, let me paste the output on iperf3 (client, MacOS) with your options
-u -b 2.5G
I see stable 2.4 Gbps
for 60 secs.❯ iperf3 -c 192.168.18.200 -t 60 -u -b 2.5G -V ─╯ iperf 3.16 Darwin roxys-mbp.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64 Control connection MSS 1448 Setting UDP block size to 1448 Time: Wed, 03 Apr 2024 19:32:03 UTC Connecting to host 192.168.18.200, port 5201 Cookie: q4zcsvcxyn4vkw6ppthdaoxjmthkmldp3eai Target Bitrate: 2500000000 [ 5] local 10.0.2.21 port 61476 connected to 192.168.18.200 port 5201 Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 287 MBytes 2.40 Gbits/sec 207546 [ 5] 1.00-2.00 sec 285 MBytes 2.39 Gbits/sec 206445 [ 5] 2.00-3.00 sec 285 MBytes 2.39 Gbits/sec 206381 [ 5] 3.00-4.00 sec 285 MBytes 2.39 Gbits/sec 206378 [ 5] 4.00-5.00 sec 285 MBytes 2.39 Gbits/sec 206381 [ 5] 5.00-6.00 sec 285 MBytes 2.39 Gbits/sec 206333 [ 5] 6.00-7.00 sec 285 MBytes 2.39 Gbits/sec 206466 [ 5] 7.00-8.00 sec 285 MBytes 2.39 Gbits/sec 206470 [ 5] 8.00-9.00 sec 285 MBytes 2.39 Gbits/sec 206268 [ 5] 9.00-10.00 sec 285 MBytes 2.39 Gbits/sec 206338 [ 5] 10.00-11.00 sec 285 MBytes 2.39 Gbits/sec 206531 [ 5] 11.00-12.00 sec 285 MBytes 2.39 Gbits/sec 206339 [ 5] 12.00-13.00 sec 285 MBytes 2.39 Gbits/sec 206485 [ 5] 13.00-14.00 sec 285 MBytes 2.39 Gbits/sec 206331 [ 5] 14.00-15.00 sec 285 MBytes 2.39 Gbits/sec 206399 [ 5] 15.00-16.00 sec 285 MBytes 2.39 Gbits/sec 206370 [ 5] 16.00-17.00 sec 285 MBytes 2.39 Gbits/sec 206357 [ 5] 17.00-18.00 sec 285 MBytes 2.39 Gbits/sec 206353 [ 5] 18.00-19.00 sec 285 MBytes 2.39 Gbits/sec 206394 [ 5] 19.00-20.00 sec 285 MBytes 2.39 Gbits/sec 206312 [ 5] 20.00-21.00 sec 285 MBytes 2.39 Gbits/sec 206508 [ 5] 21.00-22.00 sec 285 MBytes 2.39 Gbits/sec 206282 [ 5] 22.00-23.00 sec 285 MBytes 2.39 Gbits/sec 206509 [ 5] 23.00-24.00 sec 285 MBytes 2.39 Gbits/sec 206332 [ 5] 24.00-25.00 sec 285 MBytes 2.39 Gbits/sec 206459 [ 5] 25.00-26.00 sec 285 MBytes 2.39 Gbits/sec 206396 [ 5] 26.00-27.00 sec 285 MBytes 2.39 Gbits/sec 206338 [ 5] 27.00-28.00 sec 285 MBytes 2.39 Gbits/sec 206402 [ 5] 28.00-29.00 sec 285 MBytes 2.39 Gbits/sec 206351 [ 5] 29.00-30.00 sec 285 MBytes 2.39 Gbits/sec 206362 [ 5] 30.00-31.00 sec 285 MBytes 2.39 Gbits/sec 206462 [ 5] 31.00-32.00 sec 285 MBytes 2.39 Gbits/sec 206355 [ 5] 32.00-33.00 sec 285 MBytes 2.39 Gbits/sec 206347 [ 5] 33.00-34.00 sec 285 MBytes 2.39 Gbits/sec 206477 [ 5] 34.00-35.00 sec 285 MBytes 2.39 Gbits/sec 206256 [ 5] 35.00-36.00 sec 285 MBytes 2.39 Gbits/sec 206372 [ 5] 36.00-37.00 sec 285 MBytes 2.39 Gbits/sec 206484 [ 5] 37.00-38.00 sec 285 MBytes 2.39 Gbits/sec 206253 [ 5] 38.00-39.00 sec 285 MBytes 2.39 Gbits/sec 206534 [ 5] 39.00-40.00 sec 285 MBytes 2.39 Gbits/sec 206246 [ 5] 40.00-41.00 sec 285 MBytes 2.39 Gbits/sec 206405 [ 5] 41.00-42.00 sec 285 MBytes 2.39 Gbits/sec 206485 [ 5] 42.00-43.00 sec 285 MBytes 2.39 Gbits/sec 206312 [ 5] 43.00-44.00 sec 285 MBytes 2.39 Gbits/sec 206413 [ 5] 44.00-45.00 sec 285 MBytes 2.39 Gbits/sec 206487 [ 5] 45.00-46.00 sec 285 MBytes 2.39 Gbits/sec 206258 [ 5] 46.00-47.00 sec 285 MBytes 2.39 Gbits/sec 206448 [ 5] 47.00-48.00 sec 285 MBytes 2.39 Gbits/sec 206659 [ 5] 48.00-49.00 sec 285 MBytes 2.39 Gbits/sec 206452 [ 5] 49.00-50.00 sec 285 MBytes 2.39 Gbits/sec 206271 [ 5] 50.00-51.00 sec 285 MBytes 2.39 Gbits/sec 206343 [ 5] 51.00-52.00 sec 285 MBytes 2.39 Gbits/sec 206516 [ 5] 52.00-53.00 sec 285 MBytes 2.39 Gbits/sec 206392 [ 5] 53.00-54.00 sec 285 MBytes 2.39 Gbits/sec 206273 [ 5] 54.00-55.00 sec 285 MBytes 2.39 Gbits/sec 206523 [ 5] 55.00-56.00 sec 285 MBytes 2.39 Gbits/sec 206201 [ 5] 56.00-57.00 sec 285 MBytes 2.39 Gbits/sec 206410 [ 5] 57.00-58.00 sec 285 MBytes 2.39 Gbits/sec 206433 [ 5] 58.00-59.00 sec 285 MBytes 2.39 Gbits/sec 206317 [ 5] 59.00-60.00 sec 285 MBytes 2.39 Gbits/sec 206401 - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-60.00 sec 16.7 GBytes 2.39 Gbits/sec 0.000 ms 0/12384601 (0%) sender [ 5] 0.00-60.01 sec 10.0 GBytes 1.44 Gbits/sec 0.014 ms 0/12379713 (0%) receiver CPU Utilization: local/sender 90.9% (18.3%u/72.5%s), remote/receiver 26.0% (2.8%u/23.1%s) iperf Done.The option
-uis using UDP rather than TCP.
When I runiperf3without-u, i.e. with TCP, I still the speed decreased to 1.2 Gbps (but if if the test takes longer, the speed converges to 1.9 GBps)❯ iperf3 -c 192.168.18.200 -t 60 -V ─╯ iperf 3.16 Darwin roxys-mbp.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64 Control connection MSS 1448 Time: Wed, 03 Apr 2024 19:33:50 UTC Connecting to host 192.168.18.200, port 5201 Cookie: oybbl36xu7jrh7vmufngybr4arqe6fdymo7m TCP MSS: 1448 (default) [ 5] local 10.0.2.21 port 57879 connected to 192.168.18.200 port 5201 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 283 MBytes 2.37 Gbits/sec [ 5] 1.00-2.00 sec 282 MBytes 2.35 Gbits/sec [ 5] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 4.00-5.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 5.00-6.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 10.00-11.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 11.00-12.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 12.00-13.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 13.00-14.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 14.00-15.00 sec 279 MBytes 2.34 Gbits/sec [ 5] 15.00-16.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 16.00-17.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 17.00-18.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 18.00-19.00 sec 183 MBytes 1.53 Gbits/sec [ 5] 19.00-20.00 sec 246 MBytes 2.07 Gbits/sec [ 5] 20.00-21.00 sec 176 MBytes 1.48 Gbits/sec [ 5] 21.00-22.00 sec 154 MBytes 1.29 Gbits/sec [ 5] 22.00-23.00 sec 134 MBytes 1.12 Gbits/sec [ 5] 23.00-24.00 sec 138 MBytes 1.16 Gbits/sec [ 5] 24.00-25.00 sec 146 MBytes 1.22 Gbits/sec [ 5] 25.00-26.00 sec 147 MBytes 1.24 Gbits/sec [ 5] 26.00-27.00 sec 150 MBytes 1.26 Gbits/sec [ 5] 27.00-28.00 sec 150 MBytes 1.26 Gbits/sec [ 5] 28.00-29.00 sec 141 MBytes 1.19 Gbits/sec [ 5] 29.00-30.00 sec 155 MBytes 1.30 Gbits/sec [ 5] 30.00-31.00 sec 134 MBytes 1.12 Gbits/sec [ 5] 31.00-32.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 32.00-33.00 sec 154 MBytes 1.29 Gbits/sec [ 5] 33.00-34.01 sec 154 MBytes 1.29 Gbits/sec [ 5] 34.01-35.00 sec 154 MBytes 1.30 Gbits/sec [ 5] 35.00-36.01 sec 153 MBytes 1.28 Gbits/sec [ 5] 36.01-37.01 sec 155 MBytes 1.30 Gbits/sec [ 5] 37.01-38.00 sec 149 MBytes 1.26 Gbits/sec [ 5] 38.00-39.00 sec 148 MBytes 1.23 Gbits/sec [ 5] 39.00-40.00 sec 155 MBytes 1.30 Gbits/sec [ 5] 40.00-41.00 sec 155 MBytes 1.30 Gbits/sec [ 5] 41.00-42.00 sec 154 MBytes 1.29 Gbits/sec [ 5] 42.00-43.00 sec 152 MBytes 1.27 Gbits/sec [ 5] 43.00-44.00 sec 155 MBytes 1.29 Gbits/sec [ 5] 44.00-45.00 sec 178 MBytes 1.50 Gbits/sec [ 5] 45.00-46.00 sec 178 MBytes 1.50 Gbits/sec [ 5] 46.00-47.00 sec 192 MBytes 1.61 Gbits/sec [ 5] 47.00-48.01 sec 214 MBytes 1.79 Gbits/sec [ 5] 48.01-49.00 sec 200 MBytes 1.69 Gbits/sec [ 5] 49.00-50.01 sec 221 MBytes 1.85 Gbits/sec [ 5] 50.01-51.00 sec 209 MBytes 1.76 Gbits/sec [ 5] 51.00-52.01 sec 190 MBytes 1.59 Gbits/sec [ 5] 52.01-53.01 sec 188 MBytes 1.58 Gbits/sec [ 5] 53.01-54.00 sec 205 MBytes 1.73 Gbits/sec [ 5] 54.00-55.01 sec 206 MBytes 1.72 Gbits/sec [ 5] 55.01-56.00 sec 208 MBytes 1.75 Gbits/sec [ 5] 56.00-57.01 sec 211 MBytes 1.76 Gbits/sec [ 5] 57.01-58.00 sec 189 MBytes 1.58 Gbits/sec [ 5] 58.00-59.00 sec 199 MBytes 1.67 Gbits/sec [ 5] 59.00-60.00 sec 208 MBytes 1.74 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] 0.00-60.00 sec 12.0 GBytes 1.72 Gbits/sec sender [ 5] 0.00-60.01 sec 12.0 GBytes 1.72 Gbits/sec receiver CPU Utilization: local/sender 7.9% (0.1%u/7.8%s), remote/receiver 4.5% (0.2%u/4.3%s) rcv_tcp_congestion cubic iperf Done.Let me describe my setup.
I have two Intel N100 :-) machines. Both have 4 x 2.5 Gbps Intel NIC i226V . Both from TopTon ;-)
One is quite fresh, and I see diferent BIOS-es on both of them. First machine (older one) does have Performance in bios Advanced tab. Second machine (newer on) does not :-(I have installed Proxmox on first (older) machine, and some time ago I experimented with pfsense running in VM.
But now, I have installed pfsense on bare metal second machine (newer one).The pfsense NICs are physical ports and I named them as follows: (WAN) ISP, LAN, DMZ and SERVICES.
The proxmox ports are as follows- vmbr1 (SERVICES) : enp4s0
- vmbr2 (LAN) : enp2s0
- vmbr3 (DMZ): enp1s0, enp3s0 - yeas, I have bridged two NICs
enp1s0 is connected to pfsense's DMZ
enp3s0 is connected to my MacBook's eth card (usb 2.5 Gbps)I have started two Linux Containerx in Proxmox (ubuntu 22.04 + iperf3 running):
- Container1: connected with vmbr3 (DMZ)
- Container2: connected with vmbr1 (SERVICES)
When I run
iperf3 -c <ip>from my Macboox I am in the DMZ network, so the traffic goes to Proxmox's enp3s0 and directly to vmb3 and to theiperf3server running in VM (Container1).
Then I see no traffic in pfsense. This is ok to me. I would expect such behaviour, since this looks like connecting two devices in the same network (connected with vmbr - switch).
But when I askiperf3 -c <ip>withipfromSERVICESnetwork, then I see ..... what I presented above.
The traffic goes to proxmox first, but there is no bridge, and the traffic has to go from Proxmox machine to my pfsense through
Macbookx -> proxmox enp3s0 : vmbr3 : enp1s0 -> pfsense DMZ --- firewall rules --- pfsense SERVICES -> proxmox enp4s0 : vmbr1 --> Container 2I think this really tests if my pfsense is able to handle 2.5 Gbps
I tried to simply my pfsense rules onDMZinterface, but I didn't see any difference if there was only one rulePass Any Traffic -
@roxy said in Topton N100 Reporting 402 MHz:
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-60.00 sec 16.7 GBytes 2.39 Gbits/sec 0.000 ms 0/12384601 (0%) sender
[ 5] 0.00-60.01 sec 10.0 GBytes 1.44 Gbits/sec 0.014 ms 0/12379713 (0%) receiver
CPU Utilization: local/sender 90.9% (18.3%u/72.5%s), remote/receiver 26.0% (2.8%u/23.1%s)That's the important part there. When testing with UDP there is no reply traffic so the sender just sends at whatever speed you tell it to. BUT the receiver only actually saw 1.4Gbps. Unclear why it doesn't show a large loss % there unless it didn't in fact actually send.
What does the server end show for that test? -
I would be very thankful, if you could share the link to bios and the ISO file with instruction how to upgrade the bios.
My machine with pfsense has no option about Power Performance in the BIOS.
-
Here is the output of
iperfon server sideServer listening on 5201 ----------------------------------------------------------- Time: Wed, 03 Apr 2024 20:07:28 GMT Accepted connection from 10.0.2.21, port 58347 Cookie: kmci352mh4io6chfwtgjqwhuehic7m62q6hd Target Bitrate: 2500000000 [ 5] local 192.168.18.200 port 5201 connected to 10.0.2.21 port 63601 Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 279 MBytes 2.34 Gbits/sec 0.008 ms 2966/205296 (1.4%) [ 5] 1.00-2.00 sec 281 MBytes 2.36 Gbits/sec 0.004 ms 2978/206332 (1.4%) [ 5] 2.00-3.00 sec 267 MBytes 2.23 Gbits/sec 0.141 ms 12941/206131 (6.3%) [ 5] 3.00-4.00 sec 280 MBytes 2.36 Gbits/sec 0.002 ms 3544/206604 (1.7%) [ 5] 4.00-5.00 sec 282 MBytes 2.37 Gbits/sec 0.002 ms 1698/206253 (0.82%) [ 5] 5.00-6.00 sec 282 MBytes 2.37 Gbits/sec 0.003 ms 2240/206403 (1.1%) [ 5] 6.00-7.00 sec 283 MBytes 2.37 Gbits/sec 0.003 ms 1406/206361 (0.68%) [ 5] 7.00-8.00 sec 269 MBytes 2.26 Gbits/sec 0.003 ms 11370/206372 (5.5%) [ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec 0.002 ms 3777/206369 (1.8%) [ 5] 9.00-10.00 sec 283 MBytes 2.37 Gbits/sec 0.005 ms 1385/206401 (0.67%) [ 5] 10.00-11.00 sec 281 MBytes 2.36 Gbits/sec 0.002 ms 3063/206405 (1.5%) [ 5] 11.00-12.00 sec 283 MBytes 2.37 Gbits/sec 0.008 ms 1682/206392 (0.81%) [ 5] 12.00-13.00 sec 274 MBytes 2.30 Gbits/sec 0.011 ms 7850/206174 (3.8%) [ 5] 13.00-14.00 sec 279 MBytes 2.34 Gbits/sec 0.002 ms 3002/205331 (1.5%) [ 5] 14.00-15.00 sec 220 MBytes 1.85 Gbits/sec 0.004 ms 46443/205734 (23%) [ 5] 15.00-16.00 sec 176 MBytes 1.48 Gbits/sec 0.004 ms 78575/206230 (38%) [ 5] 16.00-17.00 sec 174 MBytes 1.46 Gbits/sec 0.013 ms 80466/206449 (39%) [ 5] 17.00-18.00 sec 173 MBytes 1.46 Gbits/sec 0.017 ms 80174/205795 (39%) [ 5] 18.00-19.00 sec 170 MBytes 1.43 Gbits/sec 0.014 ms 83626/206832 (40%) [ 5] 19.00-20.00 sec 174 MBytes 1.46 Gbits/sec 0.010 ms 80084/206016 (39%) [ 5] 20.00-21.00 sec 168 MBytes 1.41 Gbits/sec 0.011 ms 84685/206555 (41%) [ 5] 21.00-22.00 sec 175 MBytes 1.46 Gbits/sec 0.012 ms 80001/206417 (39%) [ 5] 22.00-23.00 sec 170 MBytes 1.43 Gbits/sec 0.004 ms 83025/206048 (40%) [ 5] 23.00-24.00 sec 174 MBytes 1.46 Gbits/sec 0.004 ms 80145/206175 (39%) [ 5] 24.00-25.00 sec 174 MBytes 1.46 Gbits/sec 0.012 ms 80689/206907 (39%) [ 5] 25.00-26.00 sec 176 MBytes 1.47 Gbits/sec 0.004 ms 78746/205936 (38%) [ 5] 26.00-27.00 sec 175 MBytes 1.47 Gbits/sec 0.004 ms 79499/206541 (38%) [ 5] 27.00-28.00 sec 174 MBytes 1.46 Gbits/sec 0.008 ms 80868/206754 (39%) [ 5] 28.00-29.00 sec 175 MBytes 1.47 Gbits/sec 0.004 ms 79228/206089 (38%) [ 5] 29.00-30.00 sec 174 MBytes 1.46 Gbits/sec 0.013 ms 80242/206141 (39%) [ 5] 30.00-31.00 sec 174 MBytes 1.46 Gbits/sec 0.013 ms 80401/206639 (39%) [ 5] 31.00-32.00 sec 171 MBytes 1.44 Gbits/sec 0.004 ms 82607/206593 (40%) [ 5] 32.00-33.00 sec 163 MBytes 1.37 Gbits/sec 0.004 ms 87459/205775 (43%) [ 5] 33.00-34.00 sec 163 MBytes 1.37 Gbits/sec 0.004 ms 88392/206567 (43%) [ 5] 34.00-35.00 sec 164 MBytes 1.38 Gbits/sec 0.004 ms 87857/206668 (43%) [ 5] 35.00-36.00 sec 165 MBytes 1.38 Gbits/sec 0.014 ms 87006/206455 (42%) [ 5] 36.00-37.00 sec 165 MBytes 1.38 Gbits/sec 0.007 ms 86617/205829 (42%) [ 5] 37.00-38.00 sec 161 MBytes 1.35 Gbits/sec 0.005 ms 89475/206368 (43%) [ 5] 38.00-39.00 sec 163 MBytes 1.37 Gbits/sec 0.012 ms 88176/206412 (43%) [ 5] 39.00-40.00 sec 165 MBytes 1.38 Gbits/sec 0.012 ms 87100/206534 (42%) [ 5] 40.00-41.00 sec 163 MBytes 1.37 Gbits/sec 0.013 ms 88299/206425 (43%) [ 5] 41.00-42.00 sec 164 MBytes 1.38 Gbits/sec 0.004 ms 87527/206649 (42%) [ 5] 42.00-43.00 sec 160 MBytes 1.34 Gbits/sec 0.004 ms 90439/206193 (44%) [ 5] 43.00-44.00 sec 164 MBytes 1.37 Gbits/sec 0.009 ms 87612/206096 (43%) [ 5] 44.00-45.00 sec 159 MBytes 1.33 Gbits/sec 0.004 ms 91141/206329 (44%) [ 5] 45.00-46.00 sec 165 MBytes 1.38 Gbits/sec 0.007 ms 87291/206462 (42%) [ 5] 46.00-47.00 sec 163 MBytes 1.37 Gbits/sec 0.013 ms 88283/206609 (43%) [ 5] 47.00-48.00 sec 164 MBytes 1.37 Gbits/sec 0.014 ms 87942/206502 (43%) [ 5] 48.00-49.00 sec 164 MBytes 1.38 Gbits/sec 0.008 ms 87637/206516 (42%) [ 5] 49.00-50.00 sec 165 MBytes 1.38 Gbits/sec 0.006 ms 87192/206386 (42%) [ 5] 50.00-51.00 sec 163 MBytes 1.37 Gbits/sec 0.011 ms 88272/206318 (43%) [ 5] 51.00-52.00 sec 164 MBytes 1.38 Gbits/sec 0.013 ms 86997/205996 (42%) [ 5] 52.00-53.00 sec 162 MBytes 1.36 Gbits/sec 0.004 ms 89590/206568 (43%) [ 5] 53.00-54.00 sec 163 MBytes 1.36 Gbits/sec 0.006 ms 88809/206489 (43%) [ 5] 54.00-55.00 sec 164 MBytes 1.37 Gbits/sec 0.009 ms 88019/206431 (43%) [ 5] 55.00-56.00 sec 165 MBytes 1.38 Gbits/sec 0.004 ms 87189/206516 (42%) [ 5] 56.00-57.00 sec 165 MBytes 1.38 Gbits/sec 0.004 ms 86963/206396 (42%) [ 5] 57.00-58.00 sec 164 MBytes 1.38 Gbits/sec 0.014 ms 87700/206487 (42%) [ 5] 58.00-59.00 sec 163 MBytes 1.36 Gbits/sec 0.004 ms 87945/205709 (43%) [ 5] 59.00-60.00 sec 165 MBytes 1.38 Gbits/sec 0.005 ms 87183/206435 (42%) [ 5] 60.00-60.00 sec 672 KBytes 1.36 Gbits/sec 0.007 ms 716/1191 (60%) - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] (sender statistics not available) [SUM] 0.0-60.0 sec 18746 datagrams received out-of-order [ 5] 0.00-60.00 sec 11.4 GBytes 1.63 Gbits/sec 0.007 ms 3944234/12379986 (32%) receiver iperf 3.9 Linux iperf 6.5.13-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-3 (2024-03-20T10:45Z) x86_64 ----------------------------------------------------------- Server listening on 5201 -----------------------------------------------------------and here is the output on client side (MacOS)
❯ iperf3 -c 192.168.18.200 -t 60 -u -b 2.5G -V ─╯ iperf 3.16 Darwin roxys-mbp.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64 Control connection MSS 1448 Setting UDP block size to 1448 Time: Wed, 03 Apr 2024 20:07:28 UTC Connecting to host 192.168.18.200, port 5201 Cookie: kmci352mh4io6chfwtgjqwhuehic7m62q6hd Target Bitrate: 2500000000 [ 5] local 10.0.2.21 port 63601 connected to 192.168.18.200 port 5201 Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 288 MBytes 2.41 Gbits/sec 208278 [ 5] 1.00-2.00 sec 285 MBytes 2.39 Gbits/sec 206327 [ 5] 2.00-3.00 sec 285 MBytes 2.39 Gbits/sec 206373 [ 5] 3.00-4.00 sec 285 MBytes 2.39 Gbits/sec 206285 [ 5] 4.00-5.00 sec 285 MBytes 2.39 Gbits/sec 206448 [ 5] 5.00-6.00 sec 285 MBytes 2.39 Gbits/sec 206320 [ 5] 6.00-7.00 sec 285 MBytes 2.39 Gbits/sec 206484 [ 5] 7.00-8.00 sec 285 MBytes 2.39 Gbits/sec 206329 [ 5] 8.00-9.00 sec 285 MBytes 2.39 Gbits/sec 206472 [ 5] 9.00-10.00 sec 285 MBytes 2.39 Gbits/sec 206409 [ 5] 10.00-11.00 sec 285 MBytes 2.39 Gbits/sec 206371 [ 5] 11.00-12.00 sec 285 MBytes 2.39 Gbits/sec 206434 [ 5] 12.00-13.00 sec 285 MBytes 2.39 Gbits/sec 206296 [ 5] 13.00-14.00 sec 285 MBytes 2.39 Gbits/sec 206422 [ 5] 14.00-15.00 sec 285 MBytes 2.39 Gbits/sec 206395 [ 5] 15.00-16.00 sec 285 MBytes 2.39 Gbits/sec 206292 [ 5] 16.00-17.00 sec 285 MBytes 2.39 Gbits/sec 206444 [ 5] 17.00-18.00 sec 285 MBytes 2.39 Gbits/sec 206422 [ 5] 18.00-19.00 sec 285 MBytes 2.39 Gbits/sec 206325 [ 5] 19.00-20.00 sec 285 MBytes 2.39 Gbits/sec 206399 [ 5] 20.00-21.00 sec 285 MBytes 2.39 Gbits/sec 206316 [ 5] 21.00-22.00 sec 285 MBytes 2.39 Gbits/sec 206491 [ 5] 22.00-23.00 sec 285 MBytes 2.39 Gbits/sec 206291 [ 5] 23.00-24.00 sec 285 MBytes 2.39 Gbits/sec 206453 [ 5] 24.00-25.00 sec 285 MBytes 2.39 Gbits/sec 206358 [ 5] 25.00-26.00 sec 285 MBytes 2.39 Gbits/sec 206349 [ 5] 26.00-27.00 sec 285 MBytes 2.39 Gbits/sec 206477 [ 5] 27.00-28.00 sec 285 MBytes 2.39 Gbits/sec 206393 [ 5] 28.00-29.00 sec 285 MBytes 2.39 Gbits/sec 206454 [ 5] 29.00-30.00 sec 285 MBytes 2.39 Gbits/sec 206262 [ 5] 30.00-31.00 sec 285 MBytes 2.39 Gbits/sec 206515 [ 5] 31.00-32.00 sec 285 MBytes 2.39 Gbits/sec 206328 [ 5] 32.00-33.00 sec 285 MBytes 2.39 Gbits/sec 206272 [ 5] 33.00-34.00 sec 285 MBytes 2.39 Gbits/sec 206456 [ 5] 34.00-35.00 sec 285 MBytes 2.39 Gbits/sec 206449 [ 5] 35.00-36.00 sec 285 MBytes 2.39 Gbits/sec 206259 [ 5] 36.00-37.00 sec 285 MBytes 2.39 Gbits/sec 206438 [ 5] 37.00-38.00 sec 285 MBytes 2.39 Gbits/sec 206453 [ 5] 38.00-39.00 sec 285 MBytes 2.39 Gbits/sec 206382 [ 5] 39.00-40.00 sec 285 MBytes 2.39 Gbits/sec 206296 [ 5] 40.00-41.00 sec 285 MBytes 2.39 Gbits/sec 206490 [ 5] 41.00-42.00 sec 285 MBytes 2.39 Gbits/sec 206334 [ 5] 42.00-43.00 sec 285 MBytes 2.39 Gbits/sec 206429 [ 5] 43.00-44.00 sec 285 MBytes 2.39 Gbits/sec 206423 [ 5] 44.00-45.00 sec 285 MBytes 2.39 Gbits/sec 206342 [ 5] 45.00-46.00 sec 285 MBytes 2.39 Gbits/sec 206319 [ 5] 46.00-47.00 sec 285 MBytes 2.39 Gbits/sec 206422 [ 5] 47.00-48.00 sec 285 MBytes 2.39 Gbits/sec 206468 [ 5] 48.00-49.00 sec 285 MBytes 2.39 Gbits/sec 206198 [ 5] 49.00-50.00 sec 285 MBytes 2.39 Gbits/sec 206460 [ 5] 50.00-51.00 sec 285 MBytes 2.39 Gbits/sec 206363 [ 5] 51.00-52.00 sec 285 MBytes 2.39 Gbits/sec 206525 [ 5] 52.00-53.00 sec 285 MBytes 2.39 Gbits/sec 206222 [ 5] 53.00-54.00 sec 285 MBytes 2.39 Gbits/sec 206451 [ 5] 54.00-55.00 sec 285 MBytes 2.39 Gbits/sec 206297 [ 5] 55.00-56.00 sec 285 MBytes 2.39 Gbits/sec 206428 [ 5] 56.00-57.00 sec 285 MBytes 2.39 Gbits/sec 206497 [ 5] 57.00-58.00 sec 285 MBytes 2.39 Gbits/sec 206403 [ 5] 58.00-59.00 sec 285 MBytes 2.39 Gbits/sec 206335 [ 5] 59.00-60.00 sec 285 MBytes 2.39 Gbits/sec 206447 - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-60.00 sec 16.7 GBytes 2.39 Gbits/sec 0.000 ms 0/12385070 (0%) sender [ 5] 0.00-60.00 sec 11.4 GBytes 1.63 Gbits/sec 0.007 ms 0/12379986 (0%) receiver CPU Utilization: local/sender 91.6% (18.2%u/73.3%s), remote/receiver 24.8% (2.8%u/21.9%s) iperf Done.I have also started
netstat -iwhen server was saying about1.4 Gbpsigc2 1500 <Link#3> 00:d0:b4:02:1c:b8 889310806 0 0 28809147 0 0 igc2 - fe80::%igc2/64 fe80::2d0:b4ff:fe02:1cb8%igc2 0 - - 1 - - igc2 - 10.0.2.0/24 10.0.2.1 27 - - 45202 - - igc3 1500 <Link#4> 00:d0:b4:02:1c:b9 28898629 0 0 877373098 0 0 igc3 - fe80::%igc3/64 fe80::2d0:b4ff:fe02:1cb9%igc3 0 - - 0 - - igc3 - 192.168.18.0/24 192.168.18.1 5279 - - 5249 - - -
@roxy said in Topton N100 Reporting 402 MHz:
[ 5] 0.00-60.00 sec 11.4 GBytes 1.63 Gbits/sec 0.007 ms 3944234/12379986 (32%) receiver
Yup so 36% packet loss. I guess that's a bug in the earlier iperf version at the sender end that it doesn't show the loss value correctly.
Try testing in reverse (
-R) to make sure it behaves the same both ways.Try sending to/from pfSense directly to see if he throttling is obvious in one direction or on one NIC there.
-
Note it's actually pretty much the same as TCP so that rules out TCP windowing as an issue.
-
I tried to run
-Rin both directions:iperf -Rstarted on Macboox
❯ iperf3 -c 192.168.18.200 -t 60 -u -b 2.5G -V -R ─╯ iperf 3.16 Darwin roxys-mbp.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64 Control connection MSS 1448 Setting UDP block size to 1448 Time: Wed, 03 Apr 2024 20:17:03 UTC Connecting to host 192.168.18.200, port 5201 Reverse mode, remote host 192.168.18.200 is sending Cookie: pnrcf6hfoswg5hh3kkkmawoajjqzhmxamjxr Target Bitrate: 2500000000 [ 5] local 10.0.2.21 port 58920 connected to 192.168.18.200 port 5201 Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-1.01 sec 284 MBytes 2.37 Gbits/sec 0.006 ms 4390/210049 (2.1%) [ 5] 1.01-2.01 sec 281 MBytes 2.36 Gbits/sec 0.006 ms 0/203620 (0%) [ 5] 2.01-3.01 sec 281 MBytes 2.36 Gbits/sec 0.007 ms 0/203441 (0%) [ 5] 3.01-4.00 sec 280 MBytes 2.35 Gbits/sec 0.006 ms 0/203058 (0%) [ 5] 4.00-5.01 sec 283 MBytes 2.37 Gbits/sec 0.005 ms 0/204846 (0%) [ 5] 5.01-6.00 sec 282 MBytes 2.37 Gbits/sec 0.008 ms 0/204482 (0%) [ 5] 6.00-7.00 sec 284 MBytes 2.38 Gbits/sec 0.007 ms 2/205329 (0.00097%) [ 5] 7.00-8.01 sec 278 MBytes 2.33 Gbits/sec 0.006 ms 0/201284 (0%) [ 5] 8.01-9.00 sec 280 MBytes 2.35 Gbits/sec 0.007 ms 0/202408 (0%) [ 5] 9.00-10.01 sec 283 MBytes 2.36 Gbits/sec 0.006 ms 0/204621 (0%) [ 5] 10.01-11.01 sec 281 MBytes 2.36 Gbits/sec 0.022 ms 1/203717 (0.00049%) [ 5] 11.01-12.01 sec 283 MBytes 2.38 Gbits/sec 0.006 ms 0/205052 (0%) [ 5] 12.01-13.01 sec 281 MBytes 2.35 Gbits/sec 0.004 ms 0/203161 (0%) [ 5] 13.01-14.01 sec 282 MBytes 2.37 Gbits/sec 0.006 ms 0/204246 (0%) [ 5] 14.01-15.00 sec 281 MBytes 2.37 Gbits/sec 0.006 ms 0/203680 (0%) [ 5] 15.00-16.01 sec 284 MBytes 2.37 Gbits/sec 0.006 ms 0/205526 (0%) [ 5] 16.01-17.01 sec 281 MBytes 2.36 Gbits/sec 0.005 ms 0/203429 (0%) [ 5] 17.01-18.01 sec 280 MBytes 2.35 Gbits/sec 0.006 ms 0/202459 (0%) [ 5] 18.01-19.00 sec 282 MBytes 2.37 Gbits/sec 0.005 ms 0/204534 (0%) [ 5] 19.00-20.01 sec 283 MBytes 2.38 Gbits/sec 0.006 ms 0/205153 (0%) [ 5] 20.01-21.01 sec 283 MBytes 2.37 Gbits/sec 0.005 ms 0/204738 (0%) [ 5] 21.01-22.00 sec 282 MBytes 2.37 Gbits/sec 0.007 ms 0/203882 (0%) [ 5] 22.00-23.01 sec 276 MBytes 2.30 Gbits/sec 0.008 ms 3353/203151 (1.7%) [ 5] 23.01-24.01 sec 253 MBytes 2.13 Gbits/sec 0.009 ms 19458/202949 (9.6%) [ 5] 24.01-25.01 sec 214 MBytes 1.79 Gbits/sec 0.007 ms 49268/204091 (24%) [ 5] 25.01-26.01 sec 177 MBytes 1.49 Gbits/sec 0.011 ms 72849/201227 (36%) [ 5] 26.01-27.01 sec 175 MBytes 1.47 Gbits/sec 0.008 ms 72156/199099 (36%) [ 5] 27.01-28.01 sec 177 MBytes 1.48 Gbits/sec 0.008 ms 74643/202743 (37%) [ 5] 28.01-29.01 sec 177 MBytes 1.48 Gbits/sec 0.009 ms 76163/204323 (37%) [ 5] 29.01-30.01 sec 177 MBytes 1.48 Gbits/sec 0.006 ms 76239/204163 (37%) [ 5] 30.01-31.01 sec 177 MBytes 1.48 Gbits/sec 0.008 ms 77237/205285 (38%) [ 5] 31.01-32.00 sec 176 MBytes 1.48 Gbits/sec 0.009 ms 75752/203409 (37%) [ 5] 32.00-33.01 sec 173 MBytes 1.45 Gbits/sec 0.008 ms 77610/203051 (38%) [ 5] 33.01-34.01 sec 177 MBytes 1.48 Gbits/sec 0.007 ms 75793/203859 (37%) [ 5] 34.01-35.01 sec 176 MBytes 1.48 Gbits/sec 0.009 ms 76527/204223 (37%) [ 5] 35.01-36.01 sec 177 MBytes 1.48 Gbits/sec 0.007 ms 76114/204129 (37%) [ 5] 36.01-37.01 sec 176 MBytes 1.48 Gbits/sec 0.010 ms 75646/203456 (37%) [ 5] 37.01-38.01 sec 177 MBytes 1.48 Gbits/sec 0.007 ms 73664/201515 (37%) [ 5] 38.01-39.01 sec 177 MBytes 1.48 Gbits/sec 0.006 ms 76484/204554 (37%) [ 5] 39.01-40.00 sec 177 MBytes 1.49 Gbits/sec 0.006 ms 75950/204111 (37%) [ 5] 40.00-41.01 sec 177 MBytes 1.48 Gbits/sec 0.007 ms 74295/202266 (37%) [ 5] 41.01-42.01 sec 176 MBytes 1.48 Gbits/sec 0.009 ms 75209/202965 (37%) [ 5] 42.01-43.01 sec 177 MBytes 1.48 Gbits/sec 0.012 ms 75755/203587 (37%) [ 5] 43.01-44.01 sec 177 MBytes 1.48 Gbits/sec 0.010 ms 77618/205504 (38%) [ 5] 44.01-45.01 sec 177 MBytes 1.48 Gbits/sec 0.011 ms 76334/204282 (37%) [ 5] 45.01-46.01 sec 177 MBytes 1.48 Gbits/sec 0.006 ms 75007/203013 (37%) [ 5] 46.01-47.01 sec 169 MBytes 1.42 Gbits/sec 0.011 ms 79994/202506 (40%) [ 5] 47.01-48.01 sec 166 MBytes 1.39 Gbits/sec 0.012 ms 82280/202449 (41%) [ 5] 48.01-49.01 sec 166 MBytes 1.39 Gbits/sec 0.010 ms 84088/204417 (41%) [ 5] 49.01-50.01 sec 166 MBytes 1.39 Gbits/sec 0.006 ms 84663/204999 (41%) [ 5] 50.01-51.01 sec 166 MBytes 1.39 Gbits/sec 0.007 ms 83931/204321 (41%) [ 5] 51.01-52.01 sec 166 MBytes 1.39 Gbits/sec 0.011 ms 84151/204381 (41%) [ 5] 52.01-53.01 sec 166 MBytes 1.39 Gbits/sec 0.011 ms 73779/194017 (38%) [ 5] 53.01-54.01 sec 166 MBytes 1.39 Gbits/sec 0.008 ms 64941/185143 (35%) [ 5] 54.01-55.01 sec 166 MBytes 1.39 Gbits/sec 0.010 ms 82550/202878 (41%) [ 5] 55.01-56.01 sec 166 MBytes 1.39 Gbits/sec 0.009 ms 79876/200038 (40%) [ 5] 56.01-57.01 sec 138 MBytes 1.16 Gbits/sec 0.010 ms 33284/133357 (25%) [ 5] 57.01-58.00 sec 156 MBytes 1.31 Gbits/sec 0.009 ms 56809/169839 (33%) [ 5] 58.00-59.01 sec 163 MBytes 1.37 Gbits/sec 0.009 ms 87254/205243 (43%) [ 5] 59.01-60.01 sec 165 MBytes 1.38 Gbits/sec 0.007 ms 84383/203939 (41%) - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-60.04 sec 16.3 GBytes 2.33 Gbits/sec 0.000 ms 0/0 (0%) sender [SUM] 0.0-60.0 sec 830 datagrams received out-of-order [ 5] 0.00-60.01 sec 12.6 GBytes 1.81 Gbits/sec 0.007 ms 2725500/12091197 (23%) receiver iperf Done.iperfstarted on proxmox Contrainer
root@iperf:~# iperf3 -c 10.0.2.21 -t 60 -u -b 2.5G -V -R iperf 3.9 Linux iperf 6.5.13-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-3 (2024-03-20T10:45Z) x86_64 Control connection MSS 1448 Setting UDP block size to 1448 Time: Wed, 03 Apr 2024 20:21:39 GMT Connecting to host 10.0.2.21, port 5201 Reverse mode, remote host 10.0.2.21 is sending Cookie: ezs42tpqznw6q2vpixnajg2qdxrgg3vmd433 Target Bitrate: 2500000000 [ 5] local 192.168.18.200 port 39124 connected to 10.0.2.21 port 5201 Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 279 MBytes 2.34 Gbits/sec 0.010 ms 1305/203601 (0.64%) [ 5] 1.00-2.00 sec 276 MBytes 2.32 Gbits/sec 0.009 ms 6152/206187 (3%) [ 5] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec 0.003 ms 3145/206375 (1.5%) [ 5] 3.00-4.00 sec 278 MBytes 2.33 Gbits/sec 0.011 ms 5006/206302 (2.4%) [ 5] 4.00-5.00 sec 279 MBytes 2.34 Gbits/sec 0.009 ms 3981/206297 (1.9%) [ 5] 5.00-6.00 sec 282 MBytes 2.37 Gbits/sec 0.002 ms 1988/206290 (0.96%) [ 5] 6.00-7.00 sec 279 MBytes 2.34 Gbits/sec 0.003 ms 4495/206340 (2.2%) [ 5] 7.00-8.00 sec 281 MBytes 2.36 Gbits/sec 0.008 ms 2863/206301 (1.4%) [ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec 0.026 ms 3270/206375 (1.6%) [ 5] 9.00-10.00 sec 278 MBytes 2.33 Gbits/sec 0.002 ms 4800/206264 (2.3%) [ 5] 10.00-11.00 sec 280 MBytes 2.35 Gbits/sec 0.008 ms 3205/206314 (1.6%) [ 5] 11.00-12.00 sec 263 MBytes 2.20 Gbits/sec 0.005 ms 16108/206319 (7.8%) [ 5] 12.00-13.00 sec 280 MBytes 2.35 Gbits/sec 0.010 ms 3777/206400 (1.8%) [ 5] 13.00-14.00 sec 283 MBytes 2.37 Gbits/sec 0.004 ms 1523/206418 (0.74%) [ 5] 14.00-15.00 sec 283 MBytes 2.38 Gbits/sec 0.008 ms 1272/206398 (0.62%) [ 5] 15.00-16.00 sec 282 MBytes 2.37 Gbits/sec 0.010 ms 2087/206365 (1%) [ 5] 16.00-17.00 sec 282 MBytes 2.36 Gbits/sec 0.007 ms 2276/206342 (1.1%) [ 5] 17.00-18.00 sec 248 MBytes 2.08 Gbits/sec 0.020 ms 25072/204614 (12%) [ 5] 18.00-19.00 sec 172 MBytes 1.44 Gbits/sec 0.005 ms 81499/206211 (40%) [ 5] 19.00-20.00 sec 161 MBytes 1.35 Gbits/sec 0.012 ms 89286/206132 (43%) [ 5] 20.00-21.00 sec 160 MBytes 1.35 Gbits/sec 0.004 ms 90586/206723 (44%) [ 5] 21.00-22.00 sec 159 MBytes 1.33 Gbits/sec 0.005 ms 90978/205917 (44%) [ 5] 22.00-23.00 sec 159 MBytes 1.34 Gbits/sec 0.005 ms 91444/206691 (44%) [ 5] 23.00-24.00 sec 163 MBytes 1.37 Gbits/sec 0.006 ms 88122/206110 (43%) [ 5] 24.00-25.00 sec 158 MBytes 1.33 Gbits/sec 0.007 ms 92125/206642 (45%) [ 5] 25.00-26.00 sec 163 MBytes 1.37 Gbits/sec 0.010 ms 88414/206610 (43%) [ 5] 26.00-27.00 sec 161 MBytes 1.35 Gbits/sec 0.004 ms 89341/205915 (43%) [ 5] 27.00-28.00 sec 162 MBytes 1.36 Gbits/sec 0.004 ms 89756/206770 (43%) [ 5] 28.00-29.00 sec 163 MBytes 1.36 Gbits/sec 0.020 ms 88767/206481 (43%) [ 5] 29.00-30.00 sec 161 MBytes 1.35 Gbits/sec 0.012 ms 89527/206107 (43%) [ 5] 30.00-31.00 sec 164 MBytes 1.37 Gbits/sec 0.004 ms 87933/206530 (43%) [ 5] 31.00-32.00 sec 164 MBytes 1.38 Gbits/sec 0.035 ms 87351/206370 (42%) [ 5] 32.00-33.00 sec 164 MBytes 1.38 Gbits/sec 0.012 ms 87615/206338 (42%) [ 5] 33.00-34.00 sec 159 MBytes 1.34 Gbits/sec 0.005 ms 90831/206324 (44%) [ 5] 34.00-35.00 sec 166 MBytes 1.39 Gbits/sec 0.011 ms 86341/206488 (42%) [ 5] 35.00-36.00 sec 165 MBytes 1.39 Gbits/sec 0.013 ms 86779/206444 (42%) [ 5] 36.00-37.00 sec 166 MBytes 1.39 Gbits/sec 0.004 ms 86181/206453 (42%) [ 5] 37.00-38.00 sec 164 MBytes 1.37 Gbits/sec 0.017 ms 87728/206418 (43%) [ 5] 38.00-39.00 sec 154 MBytes 1.30 Gbits/sec 0.005 ms 94067/205943 (46%) [ 5] 39.00-40.00 sec 151 MBytes 1.27 Gbits/sec 0.038 ms 97251/206697 (47%) [ 5] 40.00-41.00 sec 152 MBytes 1.28 Gbits/sec 0.014 ms 95802/206210 (46%) [ 5] 41.00-42.00 sec 153 MBytes 1.28 Gbits/sec 0.014 ms 95903/206515 (46%) [ 5] 42.00-43.00 sec 146 MBytes 1.22 Gbits/sec 0.018 ms 100668/206266 (49%) [ 5] 43.00-44.00 sec 152 MBytes 1.28 Gbits/sec 0.013 ms 96132/206551 (47%) [ 5] 44.00-45.00 sec 154 MBytes 1.29 Gbits/sec 0.009 ms 94836/206011 (46%) [ 5] 45.00-46.00 sec 154 MBytes 1.29 Gbits/sec 0.005 ms 95242/206582 (46%) [ 5] 46.00-47.00 sec 152 MBytes 1.28 Gbits/sec 0.012 ms 96113/206386 (47%) [ 5] 47.00-48.00 sec 150 MBytes 1.26 Gbits/sec 0.072 ms 97513/206004 (47%) [ 5] 48.00-49.00 sec 154 MBytes 1.29 Gbits/sec 0.015 ms 95460/206722 (46%) [ 5] 49.00-50.00 sec 152 MBytes 1.27 Gbits/sec 0.013 ms 96495/206513 (47%) [ 5] 50.00-51.00 sec 151 MBytes 1.27 Gbits/sec 0.005 ms 96439/205973 (47%) [ 5] 51.00-52.00 sec 149 MBytes 1.25 Gbits/sec 0.046 ms 98260/206434 (48%) [ 5] 52.00-53.00 sec 151 MBytes 1.26 Gbits/sec 0.014 ms 97434/206634 (47%) [ 5] 53.00-54.00 sec 154 MBytes 1.29 Gbits/sec 0.014 ms 94585/206313 (46%) [ 5] 54.00-55.00 sec 153 MBytes 1.28 Gbits/sec 0.005 ms 95334/206006 (46%) [ 5] 55.00-56.00 sec 153 MBytes 1.29 Gbits/sec 0.022 ms 95523/206559 (46%) [ 5] 56.00-57.00 sec 154 MBytes 1.29 Gbits/sec 0.048 ms 95663/206890 (46%) [ 5] 57.00-58.00 sec 152 MBytes 1.27 Gbits/sec 0.004 ms 95607/205594 (47%) [ 5] 58.00-59.00 sec 154 MBytes 1.29 Gbits/sec 0.026 ms 95114/206488 (46%) [ 5] 59.00-60.00 sec 154 MBytes 1.29 Gbits/sec 0.011 ms 95238/206729 (46%) - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-60.00 sec 16.7 GBytes 2.39 Gbits/sec 0.000 ms 0/12385241 (0%) sender [SUM] 0.0-60.0 sec 19367 datagrams received out-of-order [ 5] 0.00-60.00 sec 11.3 GBytes 1.62 Gbits/sec 0.011 ms 3977608/12377196 (32%) receiver iperf Done. -
-
You only need to use
-Rat the client end. It's useful because you can test both ways from the client only. Old iperf couldn't do that.
Anyway it's the same both ways so it doesn't matter which NICs are sending or receiving in pfSense. That's good they should behave identically.
So testing to/from pfSense as the server directly would show if the problem is specific to sending or receiving traffic. -
Thanks, I was able to find your post before, so I downloaded
bios4.rom, but unfortunately I am not able to open the link to ISO
To many Chineese characters in the iso link{"code":500,"message":"failed link: failed to get file: failed get parent list: failed get dir: failed get parent list: failed get dir: object not found","data":null} -
-
Thanks. But I am afraid with flashing the bios now. I will wait till I have more free time to fix bricked devices ;-)
@TheNarc , does you hardware look exactly like mine ???
4 NICs ???@roxy said in Topton N100 Reporting 402 MHz:
I would be very thankful, if you could share the link to bios and the ISO file with instruction how to upgrade the bios.
My machine with pfsense has no option about Power Performance in the BIOS.
-
@roxy Yeah I understand. For what it's worth, when it worked for me it "just worked". The difficulty I had was with the flash command, because the environment that the ISO boots into is a little odd. I think there was a "mount" command I had to use even though it seemed more like a DOS environment (uses drive letters). Anyway, if you decide to try it and have trouble I can see if my memory serves me well enough to help.
-
Good to know.
BTW, I have found , probably noewer ISO file
CW-AL-4L-V2.0(先锋四网N100-N200-I3-N305-V2支持5个M.2版本)2023.11.08.iso
;-) but I am even more afraid to use it ;-)
see
https://pan.x86pi.cn/d/BIOS%E6%9B%B4%E6%96%B0/1.Intel%E8%BF%B7%E4%BD%A0%E4%B8%BB%E6%9C%BA%E7%B3%BB%E5%88%97BIOS/1.%E7%AC%AC12%E4%BB%A3AlderLake-U-P-N%E5%85%A8%E7%B3%BB%E5%88%97/1.%E7%AC%AC12%E4%BB%A3AlderLake-N%E5%85%88%E9%94%8B%E5%9B%9B%E7%BD%91N95-N100-N200-N305%E7%B3%BB%E5%88%97-V1-V2/2.%E7%AC%AC12%E4%BB%A3AlderLake-N%E5%85%88%E9%94%8B%E5%9B%9B%E7%BD%91%E7%B3%BB%E5%88%97-V2/AlderLake-N%E5%85%88%E9%94%8B%E5%9B%9B%E7%BD%91N100-N200-I3-N305-V2_2023-11-08%E6%9B%B4%E6%96%B0/CW-AL-4L-V2.0(%E5%85%88%E9%94%8B%E5%9B%9B%E7%BD%91N100-N200-I3-N305-V2%E6%94%AF%E6%8C%815%E4%B8%AAM.2%E7%89%88%E6%9C%AC)2023.11.08.iso?sign=io2D4-Uq4Iu8MnJFDXPoCslAE1W3XC2bL0XReRoMGF4=:0 -
@roxy Yes as far as I can tell we have the exact same hardware. BIOS reports the same too (info doesn't change in the modded version):
-
Okay,
before I start flashing the bios in my Intel N100 device, I would like to summarize all the information and consider the pros and cons.
I have the device: AlderLake ULX Intel(R) N100 device with the bios
BK-1264NP Ver: 1.5(build date09/28/2023 17:23:35).
I have pfsense2.7.2-RELEASE (amb64)built onWed Dec 6 21:10:00 CET 2023installed on the device ontoFreeBSD 14.0-CURRENT.I didn't change the thermal paste yet; I left the original one. The device is running 24/7, and the temperature is around 40-45 degrees Celsius.
I have a 128GB NVME installed, and I have a 8GB RAM installed.In my BIOS settings, I can't see any options to change Power Limit 1 and Power Limit 2.
I don't have any issues with the thermal throttling, and the device is running smoothly.
The only issue (right now) is the speed of transfering files between two NICs.
The device is supposed to route the traffic between two NICs, and the speed should be around 2.5Gbps.I have observed that running
iperf3between two NICs gives 2.35Gbps at the beginning, but after 20-30 seconds, the speed drops to 1.2Gbps and increases to 1.8Gbps and stays around 1.8Gbps.Now, lets rethink the pros and cons of flashing the bios:
Pros:- I will be able to change the Power Limit 1 and Power Limit 2; I can set them to 15W and 25W.
- I will be able to set boot perfromance to
Turbomode. - I will be able to enable
C-States.
Cons:
- I will spend time on flashing the bios.
- I will to change the thermal paste and probably add new thermal pads and USB fan.
- I can brick the device and I will have to buy a new one or move to another device and install pfSense on it, and configure it again.
- My wife will be angry because I will spend time on the device instead of spending time with her.
- My kids will be angry because I will spend time on the device instead of playing with them.
Assuming the optimistic scenario, I will flash the bios, change the thermal paste, add new thermal pads, and add a USB fan and this could give me 2.35Gb/s speed between two NICs instead of current 1.8 Gbps, which is around 30% increase in speed, which I usually don't need.
My current speed usage is around 54-100Mbps (because of WIFI speed). The only case I need 2.5Gbps is when I am transferring files between two NICs, which is not a common case. I need to connect with a cable and transfer the files. I really barely do that. I can use a USB stick to transfer the files.
Moreover, I can always change thevmbr1setting in my proxmox in order to avoid traffic between pfsense interfaces :-) -> this gives me 2.5Gbps speed of transferring files between.@TheNarc , are there any more advantages?
Conclusion:
I asked co-pilot what to do and I got:
co-pilot suggesiton:I will not flash the bios, I will not change the thermal paste, I will not add new thermal pads, and I will not add a USB fan. I will spend time with my wife and kids and I will play with them. I will use a USB stick to transfer the files between two NICs. I will change thevmbr1setting in my proxmox in order to avoid traffic between pfsense interfaces. I will enjoy my life.but, I am reaaly torn between the other option, which is
dark side suggestion:I will flash the bios, I will change the thermal paste, I will add new thermal pads, and I will add a USB fan. I will spend time on the device instead of spending time with my wife and kids. I will transfer the files between two NICs with 2.3Gbps speed. I will enjoy my life.In both cases, I will enjoy my life.
Don't know which one to choose.
I need to think about it. -
You are not forced to re-paste or add a fan by flashing the BIOS. You can just leave the settings the same but it gives you the option. You would be able to set the power levels higher and see if that impacts the transfer speeds. And if it does you've probably found the problem and can stop looking.
Alternatively you could do nothing at all. It's still passing traffic at 1.5Gbps even after slowing down.
-
@roxy Haha, I think you pretty much nailed it on the pros and cons. But as @stephenw10 said you are not forced to do all the changes after flashing the BIOS. Assuming the flashing of the BIOS goes well and your device still works everything should keep working the same until you start further tinkering with it.
IF you do end up increasing the PL1 limits though it is probably a good idea to monitor the temperatures and test stability.
In my case, it seems like my box is able to run normally for about 1-2 weeks before crashing with general protection fault or page faults if the temperature is in the 55-60C range (this is where my box seems to end up with only passive cooling and light pfSense usage), but if it creeps up much higher during heavy benchmarking it can fail in as little as ~1-2 hours (with only passive cooling my box would reach 70C+ during memtest86+ and the whole box would become too hot to touch).Whether your box can survive at higher temperatures or not is probably very different from device to device. It will depend on brand and silicon quality of the RAM and NVMe drive. (In my case it is the RAM that ends up failing at high temperatures).
Atttached below is a picture of how my final solution looks like (I did not have to change any thermal paste in the box, just attach this fan on top). With that USB fan on the "medium" speed setting and the lid to the whole media wall box closed the temperature ends up in the 38-45 C degree range with very short spikes usually no higher than ~55C.
20240323_150654_small.jpg
