Topton N100 Reporting 402 MHz
-
Hmm, seems fine if that's during the test.
-
S stephenw10 referenced this topic on
-
It's quite fine, but not bad ;-)
I really would like to have 2.5 GBps :-( all the time ,
I see 2.35 at the beginning and 1.0 after 15 seconds and 1.8 back after 30 secondsThis could be the result of bad cooling (but temperature is about 45 degrees).
I tried to simplify pfsense firewall rules (single rule with Pass all traffic) but didnt get 2.5I am wondering if NetGate hw really gets 2.5 GBps. I was considering to buy NetGate 4200 to check, but I habe no spare $500
Connecting to host 192.168.18.100, port 5201 [ 5] local 10.0.2.21 port 56216 connected to 192.168.18.100 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 283 MBytes 2.36 Gbits/sec [ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 2.00-3.01 sec 282 MBytes 2.35 Gbits/sec [ 5] 3.01-4.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 4.01-5.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 5.01-6.00 sec 275 MBytes 2.31 Gbits/sec [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 7.00-8.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 8.01-9.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 9.01-10.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 10.01-11.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 11.00-12.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 12.01-13.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 13.00-14.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 14.00-15.01 sec 282 MBytes 2.35 Gbits/sec [ 5] 15.01-16.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 16.00-17.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 17.00-18.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 18.01-19.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 19.01-20.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 20.01-21.01 sec 196 MBytes 1.65 Gbits/sec [ 5] 21.01-22.00 sec 226 MBytes 1.91 Gbits/sec [ 5] 22.00-23.01 sec 204 MBytes 1.71 Gbits/sec [ 5] 23.01-24.00 sec 190 MBytes 1.60 Gbits/sec [ 5] 24.00-25.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 25.00-26.00 sec 158 MBytes 1.32 Gbits/sec [ 5] 26.00-27.00 sec 158 MBytes 1.32 Gbits/sec [ 5] 27.00-28.01 sec 157 MBytes 1.32 Gbits/sec [ 5] 28.01-29.01 sec 155 MBytes 1.30 Gbits/sec [ 5] 29.01-30.01 sec 153 MBytes 1.28 Gbits/sec [ 5] 30.01-31.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 31.00-32.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 32.00-33.00 sec 158 MBytes 1.32 Gbits/sec [ 5] 33.00-34.00 sec 157 MBytes 1.32 Gbits/sec [ 5] 34.00-35.00 sec 113 MBytes 947 Mbits/sec [ 5] 35.00-36.01 sec 103 MBytes 860 Mbits/sec [ 5] 36.01-37.01 sec 187 MBytes 1.57 Gbits/sec [ 5] 37.01-38.01 sec 193 MBytes 1.62 Gbits/sec [ 5] 38.01-39.01 sec 213 MBytes 1.79 Gbits/sec [ 5] 39.01-40.01 sec 214 MBytes 1.80 Gbits/sec [ 5] 40.01-41.01 sec 214 MBytes 1.80 Gbits/sec [ 5] 41.01-42.01 sec 213 MBytes 1.79 Gbits/sec [ 5] 42.01-43.01 sec 212 MBytes 1.78 Gbits/sec [ 5] 43.01-44.01 sec 214 MBytes 1.80 Gbits/sec [ 5] 44.01-45.00 sec 213 MBytes 1.80 Gbits/sec [ 5] 45.00-46.00 sec 213 MBytes 1.79 Gbits/sec [ 5] 46.00-47.00 sec 214 MBytes 1.79 Gbits/sec [ 5] 47.00-48.01 sec 214 MBytes 1.79 Gbits/sec [ 5] 48.01-49.00 sec 212 MBytes 1.79 Gbits/sec [ 5] 49.00-50.01 sec 199 MBytes 1.66 Gbits/sec [ 5] 50.01-51.01 sec 215 MBytes 1.80 Gbits/sec [ 5] 51.01-52.01 sec 217 MBytes 1.82 Gbits/sec [ 5] 52.01-53.01 sec 213 MBytes 1.79 Gbits/sec [ 5] 53.01-54.00 sec 213 MBytes 1.79 Gbits/sec [ 5] 54.00-55.00 sec 214 MBytes 1.79 Gbits/sec [ 5] 55.00-56.00 sec 218 MBytes 1.83 Gbits/sec -
You see retransmissions at the other end when that's shown?
Do you see dropped packets in the interface stats?
-
I have iperf sunning in
SERVICESand the client inDMZ
I am not sure how to check number of dropped packages. I think the rules are "Pass" - so I should not see any dropped packets, right?
-
One end of the iperf test usually reports retransmission counts. In a nice result there should be no retransmissions.
Interface stats can be seen in Status > Interfaces or from
netstat -i.One other user seeing this recently had disabled flow control on the NIC and required it on that particular connection. Some devices just don't play nicely with/without it so it's worth testing.
-
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll igc2 1500 <Link#3> 00:d0:b4:02:1c:b8 645357575 0 0 21639068 0 0 igc2 - fe80::%igc2/64 fe80::2d0:b4ff:fe02:1cb8%igc2 0 - - 1 - - igc2 - 10.0.2.0/24 10.0.2.1 24 - - 28181 - - igc3 1500 <Link#4> 00:d0:b4:02:1c:b9 21751498 0 0 646024233 0 0 igc3 - fe80::%igc3/64 fe80::2d0:b4ff:fe02:1cb9%igc3 0 - - 0 - - igc3 - 192.168.18.0/24 192.168.18.1 3130 - - 3115 - -
-
After 10 minutes the speed test converges to 2 Gbps ;-)
and the power consumption increased by ~ 5W
-
Hmm, well that seems suspicious!
No retries/retransmissions shown at either end? You might have to run it verbose perhaps. I expect to see something like:
steve@steve-NUC9i9QNX:~$ iperf3 -c 4200.stevew.lan Connecting to host 4200.stevew.lan, port 5201 [ 5] local 172.21.16.8 port 43826 connected to 172.21.16.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 114 MBytes 958 Mbits/sec 0 397 KBytes [ 5] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0 417 KBytes [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 417 KBytes [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 417 KBytes [ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec 0 417 KBytes [ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 417 KBytes [ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 0 417 KBytes [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 0 417 KBytes [ 5] 8.00-9.00 sec 113 MBytes 945 Mbits/sec 0 417 KBytes [ 5] 9.00-10.00 sec 112 MBytes 936 Mbits/sec 0 417 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver iperf Done.Check for ASPM in the BIOS if you haven't already. That can cause a lot of problems.
-
I will chack the ASPM tomorrow, but I am afraid I have no option in my version.
Here is the result of
iperfrunning for 10 minutes❯ iperf3 -c 192.168.18.200 -t 600 ─╯ Connecting to host 192.168.18.200, port 5201 [ 5] local 10.0.2.21 port 56698 connected to 192.168.18.200 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 284 MBytes 2.37 Gbits/sec [ 5] 1.01-2.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 2.01-3.00 sec 279 MBytes 2.35 Gbits/sec [ 5] 3.00-4.01 sec 282 MBytes 2.35 Gbits/sec [ 5] 4.01-5.01 sec 279 MBytes 2.34 Gbits/sec [ 5] 5.01-6.00 sec 279 MBytes 2.35 Gbits/sec . . . [ 5] 594.00-595.01 sec 182 MBytes 1.52 Gbits/sec [ 5] 595.01-596.01 sec 178 MBytes 1.50 Gbits/sec [ 5] 596.01-597.00 sec 174 MBytes 1.47 Gbits/sec [ 5] 597.00-598.00 sec 179 MBytes 1.50 Gbits/sec [ 5] 598.00-599.01 sec 170 MBytes 1.42 Gbits/sec [ 5] 599.01-600.00 sec 168 MBytes 1.42 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-600.00 sec 108 GBytes 1.54 Gbits/sec sender [ 5] 0.00-600.01 sec 108 GBytes 1.54 Gbits/sec receiver -
Try running with
-Vfor more verbose output. It would be interesting to see if it's reducing he window size because or retries. That is common. -
@stephenw10
iperf -V#server side - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] (sender statistics not available) [ 5] 0.00-60.00 sec 12.9 GBytes 1.84 Gbits/sec receiver rcv_tcp_congestion cubic iperf 3.9 Linux iperf 6.5.13-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-3 (2024-03-20T10:45Z) x86_64 # client side [ 5] 57.00-58.00 sec 222 MBytes 1.86 Gbits/sec [ 5] 58.00-59.01 sec 237 MBytes 1.98 Gbits/sec [ 5] 59.01-60.00 sec 223 MBytes 1.88 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] 0.00-60.00 sec 12.9 GBytes 1.84 Gbits/sec sender [ 5] 0.00-60.00 sec 12.9 GBytes 1.84 Gbits/sec receiver CPU Utilization: local/sender 8.1% (0.1%u/8.0%s), remote/receiver 7.6% (0.3%u/7.4%s) rcv_tcp_congestion cubic -
Odd. Your output seems to have a space where retr and cwnd would be shown, but isn't. Same iperf version.
-
@roxy Apologies if I've somewhat lost the thread here but do you have a control, so to speak? That is to say, have you run iperf to the pfSense box from more than one host on your LAN and seen the same results? And, ideally, have you also run iperf tests directly between two hosts on your LAN and seen sustained 2.5Gbps throughput? Just trying to isolate variables. I wish I could help because it sounds like we have nearly (maybe exactly) identical pfSense hardware, except my whole LAN is still 1Gbps. I noticed from BIOS screen shots I took that I actually have ASPM enabled; haven't noticed any ill effects, but may disable it when I get a chance.
Also have you tried a UDP iperf test? I think
iperf3 -c <pfsense_ip> -u -b 2.5Gshould work. I'm not sure what we'll necessarily learn if that goes worse/same/better but it's another data point to gather. -
@roxy I think we have the same hardware and BIOS. I am able to sustain about ~2.3 Gbits/sec constantly using iperf now that I have new 2.5Gbe switches. Based on your openssl benchmarks also getting slower over time while you say temperatures are fine I'd guess you are hitting power limits just like @TheNarc seemed to be doing. I'm not sure why my box never experienced these issues with the same BIOS version but I guess the default settings which are normally hidden might be different. You might have to flash the same custom BIOS @TheNarc did and play with the PL1 settings it can unlock.
An explanation could be that the manufacturer of these boxes have realized they are not really built to passively cool a sustained heatload from these CPUs and have therefore added low sustained power limits (PL1) to remove crashes (which is an issue I experienced instead). So even though my box was performing much better out-of-the-box it would instead crash from sustained high temperatures and the only solution I found was to add an external fan blowing on the case. The CPU is fine with much higher temperatures but the other components in the very small and very badly ventilated box will cook the RAM and NVMe drives inside making them unstable because of the high ambient temperatures the CPU causes.
-
Of course, I am open to talk.
Thets right, we may have both similar hardware
First, let me paste the output on iperf3 (client, MacOS) with your options
-u -b 2.5G
I see stable 2.4 Gbps
for 60 secs.❯ iperf3 -c 192.168.18.200 -t 60 -u -b 2.5G -V ─╯ iperf 3.16 Darwin roxys-mbp.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64 Control connection MSS 1448 Setting UDP block size to 1448 Time: Wed, 03 Apr 2024 19:32:03 UTC Connecting to host 192.168.18.200, port 5201 Cookie: q4zcsvcxyn4vkw6ppthdaoxjmthkmldp3eai Target Bitrate: 2500000000 [ 5] local 10.0.2.21 port 61476 connected to 192.168.18.200 port 5201 Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 287 MBytes 2.40 Gbits/sec 207546 [ 5] 1.00-2.00 sec 285 MBytes 2.39 Gbits/sec 206445 [ 5] 2.00-3.00 sec 285 MBytes 2.39 Gbits/sec 206381 [ 5] 3.00-4.00 sec 285 MBytes 2.39 Gbits/sec 206378 [ 5] 4.00-5.00 sec 285 MBytes 2.39 Gbits/sec 206381 [ 5] 5.00-6.00 sec 285 MBytes 2.39 Gbits/sec 206333 [ 5] 6.00-7.00 sec 285 MBytes 2.39 Gbits/sec 206466 [ 5] 7.00-8.00 sec 285 MBytes 2.39 Gbits/sec 206470 [ 5] 8.00-9.00 sec 285 MBytes 2.39 Gbits/sec 206268 [ 5] 9.00-10.00 sec 285 MBytes 2.39 Gbits/sec 206338 [ 5] 10.00-11.00 sec 285 MBytes 2.39 Gbits/sec 206531 [ 5] 11.00-12.00 sec 285 MBytes 2.39 Gbits/sec 206339 [ 5] 12.00-13.00 sec 285 MBytes 2.39 Gbits/sec 206485 [ 5] 13.00-14.00 sec 285 MBytes 2.39 Gbits/sec 206331 [ 5] 14.00-15.00 sec 285 MBytes 2.39 Gbits/sec 206399 [ 5] 15.00-16.00 sec 285 MBytes 2.39 Gbits/sec 206370 [ 5] 16.00-17.00 sec 285 MBytes 2.39 Gbits/sec 206357 [ 5] 17.00-18.00 sec 285 MBytes 2.39 Gbits/sec 206353 [ 5] 18.00-19.00 sec 285 MBytes 2.39 Gbits/sec 206394 [ 5] 19.00-20.00 sec 285 MBytes 2.39 Gbits/sec 206312 [ 5] 20.00-21.00 sec 285 MBytes 2.39 Gbits/sec 206508 [ 5] 21.00-22.00 sec 285 MBytes 2.39 Gbits/sec 206282 [ 5] 22.00-23.00 sec 285 MBytes 2.39 Gbits/sec 206509 [ 5] 23.00-24.00 sec 285 MBytes 2.39 Gbits/sec 206332 [ 5] 24.00-25.00 sec 285 MBytes 2.39 Gbits/sec 206459 [ 5] 25.00-26.00 sec 285 MBytes 2.39 Gbits/sec 206396 [ 5] 26.00-27.00 sec 285 MBytes 2.39 Gbits/sec 206338 [ 5] 27.00-28.00 sec 285 MBytes 2.39 Gbits/sec 206402 [ 5] 28.00-29.00 sec 285 MBytes 2.39 Gbits/sec 206351 [ 5] 29.00-30.00 sec 285 MBytes 2.39 Gbits/sec 206362 [ 5] 30.00-31.00 sec 285 MBytes 2.39 Gbits/sec 206462 [ 5] 31.00-32.00 sec 285 MBytes 2.39 Gbits/sec 206355 [ 5] 32.00-33.00 sec 285 MBytes 2.39 Gbits/sec 206347 [ 5] 33.00-34.00 sec 285 MBytes 2.39 Gbits/sec 206477 [ 5] 34.00-35.00 sec 285 MBytes 2.39 Gbits/sec 206256 [ 5] 35.00-36.00 sec 285 MBytes 2.39 Gbits/sec 206372 [ 5] 36.00-37.00 sec 285 MBytes 2.39 Gbits/sec 206484 [ 5] 37.00-38.00 sec 285 MBytes 2.39 Gbits/sec 206253 [ 5] 38.00-39.00 sec 285 MBytes 2.39 Gbits/sec 206534 [ 5] 39.00-40.00 sec 285 MBytes 2.39 Gbits/sec 206246 [ 5] 40.00-41.00 sec 285 MBytes 2.39 Gbits/sec 206405 [ 5] 41.00-42.00 sec 285 MBytes 2.39 Gbits/sec 206485 [ 5] 42.00-43.00 sec 285 MBytes 2.39 Gbits/sec 206312 [ 5] 43.00-44.00 sec 285 MBytes 2.39 Gbits/sec 206413 [ 5] 44.00-45.00 sec 285 MBytes 2.39 Gbits/sec 206487 [ 5] 45.00-46.00 sec 285 MBytes 2.39 Gbits/sec 206258 [ 5] 46.00-47.00 sec 285 MBytes 2.39 Gbits/sec 206448 [ 5] 47.00-48.00 sec 285 MBytes 2.39 Gbits/sec 206659 [ 5] 48.00-49.00 sec 285 MBytes 2.39 Gbits/sec 206452 [ 5] 49.00-50.00 sec 285 MBytes 2.39 Gbits/sec 206271 [ 5] 50.00-51.00 sec 285 MBytes 2.39 Gbits/sec 206343 [ 5] 51.00-52.00 sec 285 MBytes 2.39 Gbits/sec 206516 [ 5] 52.00-53.00 sec 285 MBytes 2.39 Gbits/sec 206392 [ 5] 53.00-54.00 sec 285 MBytes 2.39 Gbits/sec 206273 [ 5] 54.00-55.00 sec 285 MBytes 2.39 Gbits/sec 206523 [ 5] 55.00-56.00 sec 285 MBytes 2.39 Gbits/sec 206201 [ 5] 56.00-57.00 sec 285 MBytes 2.39 Gbits/sec 206410 [ 5] 57.00-58.00 sec 285 MBytes 2.39 Gbits/sec 206433 [ 5] 58.00-59.00 sec 285 MBytes 2.39 Gbits/sec 206317 [ 5] 59.00-60.00 sec 285 MBytes 2.39 Gbits/sec 206401 - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-60.00 sec 16.7 GBytes 2.39 Gbits/sec 0.000 ms 0/12384601 (0%) sender [ 5] 0.00-60.01 sec 10.0 GBytes 1.44 Gbits/sec 0.014 ms 0/12379713 (0%) receiver CPU Utilization: local/sender 90.9% (18.3%u/72.5%s), remote/receiver 26.0% (2.8%u/23.1%s) iperf Done.The option
-uis using UDP rather than TCP.
When I runiperf3without-u, i.e. with TCP, I still the speed decreased to 1.2 Gbps (but if if the test takes longer, the speed converges to 1.9 GBps)❯ iperf3 -c 192.168.18.200 -t 60 -V ─╯ iperf 3.16 Darwin roxys-mbp.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64 Control connection MSS 1448 Time: Wed, 03 Apr 2024 19:33:50 UTC Connecting to host 192.168.18.200, port 5201 Cookie: oybbl36xu7jrh7vmufngybr4arqe6fdymo7m TCP MSS: 1448 (default) [ 5] local 10.0.2.21 port 57879 connected to 192.168.18.200 port 5201 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 283 MBytes 2.37 Gbits/sec [ 5] 1.00-2.00 sec 282 MBytes 2.35 Gbits/sec [ 5] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 4.00-5.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 5.00-6.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 10.00-11.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 11.00-12.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 12.00-13.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 13.00-14.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 14.00-15.00 sec 279 MBytes 2.34 Gbits/sec [ 5] 15.00-16.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 16.00-17.00 sec 281 MBytes 2.35 Gbits/sec [ 5] 17.00-18.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 18.00-19.00 sec 183 MBytes 1.53 Gbits/sec [ 5] 19.00-20.00 sec 246 MBytes 2.07 Gbits/sec [ 5] 20.00-21.00 sec 176 MBytes 1.48 Gbits/sec [ 5] 21.00-22.00 sec 154 MBytes 1.29 Gbits/sec [ 5] 22.00-23.00 sec 134 MBytes 1.12 Gbits/sec [ 5] 23.00-24.00 sec 138 MBytes 1.16 Gbits/sec [ 5] 24.00-25.00 sec 146 MBytes 1.22 Gbits/sec [ 5] 25.00-26.00 sec 147 MBytes 1.24 Gbits/sec [ 5] 26.00-27.00 sec 150 MBytes 1.26 Gbits/sec [ 5] 27.00-28.00 sec 150 MBytes 1.26 Gbits/sec [ 5] 28.00-29.00 sec 141 MBytes 1.19 Gbits/sec [ 5] 29.00-30.00 sec 155 MBytes 1.30 Gbits/sec [ 5] 30.00-31.00 sec 134 MBytes 1.12 Gbits/sec [ 5] 31.00-32.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 32.00-33.00 sec 154 MBytes 1.29 Gbits/sec [ 5] 33.00-34.01 sec 154 MBytes 1.29 Gbits/sec [ 5] 34.01-35.00 sec 154 MBytes 1.30 Gbits/sec [ 5] 35.00-36.01 sec 153 MBytes 1.28 Gbits/sec [ 5] 36.01-37.01 sec 155 MBytes 1.30 Gbits/sec [ 5] 37.01-38.00 sec 149 MBytes 1.26 Gbits/sec [ 5] 38.00-39.00 sec 148 MBytes 1.23 Gbits/sec [ 5] 39.00-40.00 sec 155 MBytes 1.30 Gbits/sec [ 5] 40.00-41.00 sec 155 MBytes 1.30 Gbits/sec [ 5] 41.00-42.00 sec 154 MBytes 1.29 Gbits/sec [ 5] 42.00-43.00 sec 152 MBytes 1.27 Gbits/sec [ 5] 43.00-44.00 sec 155 MBytes 1.29 Gbits/sec [ 5] 44.00-45.00 sec 178 MBytes 1.50 Gbits/sec [ 5] 45.00-46.00 sec 178 MBytes 1.50 Gbits/sec [ 5] 46.00-47.00 sec 192 MBytes 1.61 Gbits/sec [ 5] 47.00-48.01 sec 214 MBytes 1.79 Gbits/sec [ 5] 48.01-49.00 sec 200 MBytes 1.69 Gbits/sec [ 5] 49.00-50.01 sec 221 MBytes 1.85 Gbits/sec [ 5] 50.01-51.00 sec 209 MBytes 1.76 Gbits/sec [ 5] 51.00-52.01 sec 190 MBytes 1.59 Gbits/sec [ 5] 52.01-53.01 sec 188 MBytes 1.58 Gbits/sec [ 5] 53.01-54.00 sec 205 MBytes 1.73 Gbits/sec [ 5] 54.00-55.01 sec 206 MBytes 1.72 Gbits/sec [ 5] 55.01-56.00 sec 208 MBytes 1.75 Gbits/sec [ 5] 56.00-57.01 sec 211 MBytes 1.76 Gbits/sec [ 5] 57.01-58.00 sec 189 MBytes 1.58 Gbits/sec [ 5] 58.00-59.00 sec 199 MBytes 1.67 Gbits/sec [ 5] 59.00-60.00 sec 208 MBytes 1.74 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] 0.00-60.00 sec 12.0 GBytes 1.72 Gbits/sec sender [ 5] 0.00-60.01 sec 12.0 GBytes 1.72 Gbits/sec receiver CPU Utilization: local/sender 7.9% (0.1%u/7.8%s), remote/receiver 4.5% (0.2%u/4.3%s) rcv_tcp_congestion cubic iperf Done.Let me describe my setup.
I have two Intel N100 :-) machines. Both have 4 x 2.5 Gbps Intel NIC i226V . Both from TopTon ;-)
One is quite fresh, and I see diferent BIOS-es on both of them. First machine (older one) does have Performance in bios Advanced tab. Second machine (newer on) does not :-(I have installed Proxmox on first (older) machine, and some time ago I experimented with pfsense running in VM.
But now, I have installed pfsense on bare metal second machine (newer one).The pfsense NICs are physical ports and I named them as follows: (WAN) ISP, LAN, DMZ and SERVICES.
The proxmox ports are as follows- vmbr1 (SERVICES) : enp4s0
- vmbr2 (LAN) : enp2s0
- vmbr3 (DMZ): enp1s0, enp3s0 - yeas, I have bridged two NICs
enp1s0 is connected to pfsense's DMZ
enp3s0 is connected to my MacBook's eth card (usb 2.5 Gbps)I have started two Linux Containerx in Proxmox (ubuntu 22.04 + iperf3 running):
- Container1: connected with vmbr3 (DMZ)
- Container2: connected with vmbr1 (SERVICES)
When I run
iperf3 -c <ip>from my Macboox I am in the DMZ network, so the traffic goes to Proxmox's enp3s0 and directly to vmb3 and to theiperf3server running in VM (Container1).
Then I see no traffic in pfsense. This is ok to me. I would expect such behaviour, since this looks like connecting two devices in the same network (connected with vmbr - switch).
But when I askiperf3 -c <ip>withipfromSERVICESnetwork, then I see ..... what I presented above.
The traffic goes to proxmox first, but there is no bridge, and the traffic has to go from Proxmox machine to my pfsense through
Macbookx -> proxmox enp3s0 : vmbr3 : enp1s0 -> pfsense DMZ --- firewall rules --- pfsense SERVICES -> proxmox enp4s0 : vmbr1 --> Container 2I think this really tests if my pfsense is able to handle 2.5 Gbps
I tried to simply my pfsense rules onDMZinterface, but I didn't see any difference if there was only one rulePass Any Traffic -
@roxy said in Topton N100 Reporting 402 MHz:
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-60.00 sec 16.7 GBytes 2.39 Gbits/sec 0.000 ms 0/12384601 (0%) sender
[ 5] 0.00-60.01 sec 10.0 GBytes 1.44 Gbits/sec 0.014 ms 0/12379713 (0%) receiver
CPU Utilization: local/sender 90.9% (18.3%u/72.5%s), remote/receiver 26.0% (2.8%u/23.1%s)That's the important part there. When testing with UDP there is no reply traffic so the sender just sends at whatever speed you tell it to. BUT the receiver only actually saw 1.4Gbps. Unclear why it doesn't show a large loss % there unless it didn't in fact actually send.
What does the server end show for that test? -
I would be very thankful, if you could share the link to bios and the ISO file with instruction how to upgrade the bios.
My machine with pfsense has no option about Power Performance in the BIOS.
-
Here is the output of
iperfon server sideServer listening on 5201 ----------------------------------------------------------- Time: Wed, 03 Apr 2024 20:07:28 GMT Accepted connection from 10.0.2.21, port 58347 Cookie: kmci352mh4io6chfwtgjqwhuehic7m62q6hd Target Bitrate: 2500000000 [ 5] local 192.168.18.200 port 5201 connected to 10.0.2.21 port 63601 Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 279 MBytes 2.34 Gbits/sec 0.008 ms 2966/205296 (1.4%) [ 5] 1.00-2.00 sec 281 MBytes 2.36 Gbits/sec 0.004 ms 2978/206332 (1.4%) [ 5] 2.00-3.00 sec 267 MBytes 2.23 Gbits/sec 0.141 ms 12941/206131 (6.3%) [ 5] 3.00-4.00 sec 280 MBytes 2.36 Gbits/sec 0.002 ms 3544/206604 (1.7%) [ 5] 4.00-5.00 sec 282 MBytes 2.37 Gbits/sec 0.002 ms 1698/206253 (0.82%) [ 5] 5.00-6.00 sec 282 MBytes 2.37 Gbits/sec 0.003 ms 2240/206403 (1.1%) [ 5] 6.00-7.00 sec 283 MBytes 2.37 Gbits/sec 0.003 ms 1406/206361 (0.68%) [ 5] 7.00-8.00 sec 269 MBytes 2.26 Gbits/sec 0.003 ms 11370/206372 (5.5%) [ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec 0.002 ms 3777/206369 (1.8%) [ 5] 9.00-10.00 sec 283 MBytes 2.37 Gbits/sec 0.005 ms 1385/206401 (0.67%) [ 5] 10.00-11.00 sec 281 MBytes 2.36 Gbits/sec 0.002 ms 3063/206405 (1.5%) [ 5] 11.00-12.00 sec 283 MBytes 2.37 Gbits/sec 0.008 ms 1682/206392 (0.81%) [ 5] 12.00-13.00 sec 274 MBytes 2.30 Gbits/sec 0.011 ms 7850/206174 (3.8%) [ 5] 13.00-14.00 sec 279 MBytes 2.34 Gbits/sec 0.002 ms 3002/205331 (1.5%) [ 5] 14.00-15.00 sec 220 MBytes 1.85 Gbits/sec 0.004 ms 46443/205734 (23%) [ 5] 15.00-16.00 sec 176 MBytes 1.48 Gbits/sec 0.004 ms 78575/206230 (38%) [ 5] 16.00-17.00 sec 174 MBytes 1.46 Gbits/sec 0.013 ms 80466/206449 (39%) [ 5] 17.00-18.00 sec 173 MBytes 1.46 Gbits/sec 0.017 ms 80174/205795 (39%) [ 5] 18.00-19.00 sec 170 MBytes 1.43 Gbits/sec 0.014 ms 83626/206832 (40%) [ 5] 19.00-20.00 sec 174 MBytes 1.46 Gbits/sec 0.010 ms 80084/206016 (39%) [ 5] 20.00-21.00 sec 168 MBytes 1.41 Gbits/sec 0.011 ms 84685/206555 (41%) [ 5] 21.00-22.00 sec 175 MBytes 1.46 Gbits/sec 0.012 ms 80001/206417 (39%) [ 5] 22.00-23.00 sec 170 MBytes 1.43 Gbits/sec 0.004 ms 83025/206048 (40%) [ 5] 23.00-24.00 sec 174 MBytes 1.46 Gbits/sec 0.004 ms 80145/206175 (39%) [ 5] 24.00-25.00 sec 174 MBytes 1.46 Gbits/sec 0.012 ms 80689/206907 (39%) [ 5] 25.00-26.00 sec 176 MBytes 1.47 Gbits/sec 0.004 ms 78746/205936 (38%) [ 5] 26.00-27.00 sec 175 MBytes 1.47 Gbits/sec 0.004 ms 79499/206541 (38%) [ 5] 27.00-28.00 sec 174 MBytes 1.46 Gbits/sec 0.008 ms 80868/206754 (39%) [ 5] 28.00-29.00 sec 175 MBytes 1.47 Gbits/sec 0.004 ms 79228/206089 (38%) [ 5] 29.00-30.00 sec 174 MBytes 1.46 Gbits/sec 0.013 ms 80242/206141 (39%) [ 5] 30.00-31.00 sec 174 MBytes 1.46 Gbits/sec 0.013 ms 80401/206639 (39%) [ 5] 31.00-32.00 sec 171 MBytes 1.44 Gbits/sec 0.004 ms 82607/206593 (40%) [ 5] 32.00-33.00 sec 163 MBytes 1.37 Gbits/sec 0.004 ms 87459/205775 (43%) [ 5] 33.00-34.00 sec 163 MBytes 1.37 Gbits/sec 0.004 ms 88392/206567 (43%) [ 5] 34.00-35.00 sec 164 MBytes 1.38 Gbits/sec 0.004 ms 87857/206668 (43%) [ 5] 35.00-36.00 sec 165 MBytes 1.38 Gbits/sec 0.014 ms 87006/206455 (42%) [ 5] 36.00-37.00 sec 165 MBytes 1.38 Gbits/sec 0.007 ms 86617/205829 (42%) [ 5] 37.00-38.00 sec 161 MBytes 1.35 Gbits/sec 0.005 ms 89475/206368 (43%) [ 5] 38.00-39.00 sec 163 MBytes 1.37 Gbits/sec 0.012 ms 88176/206412 (43%) [ 5] 39.00-40.00 sec 165 MBytes 1.38 Gbits/sec 0.012 ms 87100/206534 (42%) [ 5] 40.00-41.00 sec 163 MBytes 1.37 Gbits/sec 0.013 ms 88299/206425 (43%) [ 5] 41.00-42.00 sec 164 MBytes 1.38 Gbits/sec 0.004 ms 87527/206649 (42%) [ 5] 42.00-43.00 sec 160 MBytes 1.34 Gbits/sec 0.004 ms 90439/206193 (44%) [ 5] 43.00-44.00 sec 164 MBytes 1.37 Gbits/sec 0.009 ms 87612/206096 (43%) [ 5] 44.00-45.00 sec 159 MBytes 1.33 Gbits/sec 0.004 ms 91141/206329 (44%) [ 5] 45.00-46.00 sec 165 MBytes 1.38 Gbits/sec 0.007 ms 87291/206462 (42%) [ 5] 46.00-47.00 sec 163 MBytes 1.37 Gbits/sec 0.013 ms 88283/206609 (43%) [ 5] 47.00-48.00 sec 164 MBytes 1.37 Gbits/sec 0.014 ms 87942/206502 (43%) [ 5] 48.00-49.00 sec 164 MBytes 1.38 Gbits/sec 0.008 ms 87637/206516 (42%) [ 5] 49.00-50.00 sec 165 MBytes 1.38 Gbits/sec 0.006 ms 87192/206386 (42%) [ 5] 50.00-51.00 sec 163 MBytes 1.37 Gbits/sec 0.011 ms 88272/206318 (43%) [ 5] 51.00-52.00 sec 164 MBytes 1.38 Gbits/sec 0.013 ms 86997/205996 (42%) [ 5] 52.00-53.00 sec 162 MBytes 1.36 Gbits/sec 0.004 ms 89590/206568 (43%) [ 5] 53.00-54.00 sec 163 MBytes 1.36 Gbits/sec 0.006 ms 88809/206489 (43%) [ 5] 54.00-55.00 sec 164 MBytes 1.37 Gbits/sec 0.009 ms 88019/206431 (43%) [ 5] 55.00-56.00 sec 165 MBytes 1.38 Gbits/sec 0.004 ms 87189/206516 (42%) [ 5] 56.00-57.00 sec 165 MBytes 1.38 Gbits/sec 0.004 ms 86963/206396 (42%) [ 5] 57.00-58.00 sec 164 MBytes 1.38 Gbits/sec 0.014 ms 87700/206487 (42%) [ 5] 58.00-59.00 sec 163 MBytes 1.36 Gbits/sec 0.004 ms 87945/205709 (43%) [ 5] 59.00-60.00 sec 165 MBytes 1.38 Gbits/sec 0.005 ms 87183/206435 (42%) [ 5] 60.00-60.00 sec 672 KBytes 1.36 Gbits/sec 0.007 ms 716/1191 (60%) - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] (sender statistics not available) [SUM] 0.0-60.0 sec 18746 datagrams received out-of-order [ 5] 0.00-60.00 sec 11.4 GBytes 1.63 Gbits/sec 0.007 ms 3944234/12379986 (32%) receiver iperf 3.9 Linux iperf 6.5.13-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-3 (2024-03-20T10:45Z) x86_64 ----------------------------------------------------------- Server listening on 5201 -----------------------------------------------------------and here is the output on client side (MacOS)
❯ iperf3 -c 192.168.18.200 -t 60 -u -b 2.5G -V ─╯ iperf 3.16 Darwin roxys-mbp.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64 Control connection MSS 1448 Setting UDP block size to 1448 Time: Wed, 03 Apr 2024 20:07:28 UTC Connecting to host 192.168.18.200, port 5201 Cookie: kmci352mh4io6chfwtgjqwhuehic7m62q6hd Target Bitrate: 2500000000 [ 5] local 10.0.2.21 port 63601 connected to 192.168.18.200 port 5201 Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 60 second test, tos 0 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 288 MBytes 2.41 Gbits/sec 208278 [ 5] 1.00-2.00 sec 285 MBytes 2.39 Gbits/sec 206327 [ 5] 2.00-3.00 sec 285 MBytes 2.39 Gbits/sec 206373 [ 5] 3.00-4.00 sec 285 MBytes 2.39 Gbits/sec 206285 [ 5] 4.00-5.00 sec 285 MBytes 2.39 Gbits/sec 206448 [ 5] 5.00-6.00 sec 285 MBytes 2.39 Gbits/sec 206320 [ 5] 6.00-7.00 sec 285 MBytes 2.39 Gbits/sec 206484 [ 5] 7.00-8.00 sec 285 MBytes 2.39 Gbits/sec 206329 [ 5] 8.00-9.00 sec 285 MBytes 2.39 Gbits/sec 206472 [ 5] 9.00-10.00 sec 285 MBytes 2.39 Gbits/sec 206409 [ 5] 10.00-11.00 sec 285 MBytes 2.39 Gbits/sec 206371 [ 5] 11.00-12.00 sec 285 MBytes 2.39 Gbits/sec 206434 [ 5] 12.00-13.00 sec 285 MBytes 2.39 Gbits/sec 206296 [ 5] 13.00-14.00 sec 285 MBytes 2.39 Gbits/sec 206422 [ 5] 14.00-15.00 sec 285 MBytes 2.39 Gbits/sec 206395 [ 5] 15.00-16.00 sec 285 MBytes 2.39 Gbits/sec 206292 [ 5] 16.00-17.00 sec 285 MBytes 2.39 Gbits/sec 206444 [ 5] 17.00-18.00 sec 285 MBytes 2.39 Gbits/sec 206422 [ 5] 18.00-19.00 sec 285 MBytes 2.39 Gbits/sec 206325 [ 5] 19.00-20.00 sec 285 MBytes 2.39 Gbits/sec 206399 [ 5] 20.00-21.00 sec 285 MBytes 2.39 Gbits/sec 206316 [ 5] 21.00-22.00 sec 285 MBytes 2.39 Gbits/sec 206491 [ 5] 22.00-23.00 sec 285 MBytes 2.39 Gbits/sec 206291 [ 5] 23.00-24.00 sec 285 MBytes 2.39 Gbits/sec 206453 [ 5] 24.00-25.00 sec 285 MBytes 2.39 Gbits/sec 206358 [ 5] 25.00-26.00 sec 285 MBytes 2.39 Gbits/sec 206349 [ 5] 26.00-27.00 sec 285 MBytes 2.39 Gbits/sec 206477 [ 5] 27.00-28.00 sec 285 MBytes 2.39 Gbits/sec 206393 [ 5] 28.00-29.00 sec 285 MBytes 2.39 Gbits/sec 206454 [ 5] 29.00-30.00 sec 285 MBytes 2.39 Gbits/sec 206262 [ 5] 30.00-31.00 sec 285 MBytes 2.39 Gbits/sec 206515 [ 5] 31.00-32.00 sec 285 MBytes 2.39 Gbits/sec 206328 [ 5] 32.00-33.00 sec 285 MBytes 2.39 Gbits/sec 206272 [ 5] 33.00-34.00 sec 285 MBytes 2.39 Gbits/sec 206456 [ 5] 34.00-35.00 sec 285 MBytes 2.39 Gbits/sec 206449 [ 5] 35.00-36.00 sec 285 MBytes 2.39 Gbits/sec 206259 [ 5] 36.00-37.00 sec 285 MBytes 2.39 Gbits/sec 206438 [ 5] 37.00-38.00 sec 285 MBytes 2.39 Gbits/sec 206453 [ 5] 38.00-39.00 sec 285 MBytes 2.39 Gbits/sec 206382 [ 5] 39.00-40.00 sec 285 MBytes 2.39 Gbits/sec 206296 [ 5] 40.00-41.00 sec 285 MBytes 2.39 Gbits/sec 206490 [ 5] 41.00-42.00 sec 285 MBytes 2.39 Gbits/sec 206334 [ 5] 42.00-43.00 sec 285 MBytes 2.39 Gbits/sec 206429 [ 5] 43.00-44.00 sec 285 MBytes 2.39 Gbits/sec 206423 [ 5] 44.00-45.00 sec 285 MBytes 2.39 Gbits/sec 206342 [ 5] 45.00-46.00 sec 285 MBytes 2.39 Gbits/sec 206319 [ 5] 46.00-47.00 sec 285 MBytes 2.39 Gbits/sec 206422 [ 5] 47.00-48.00 sec 285 MBytes 2.39 Gbits/sec 206468 [ 5] 48.00-49.00 sec 285 MBytes 2.39 Gbits/sec 206198 [ 5] 49.00-50.00 sec 285 MBytes 2.39 Gbits/sec 206460 [ 5] 50.00-51.00 sec 285 MBytes 2.39 Gbits/sec 206363 [ 5] 51.00-52.00 sec 285 MBytes 2.39 Gbits/sec 206525 [ 5] 52.00-53.00 sec 285 MBytes 2.39 Gbits/sec 206222 [ 5] 53.00-54.00 sec 285 MBytes 2.39 Gbits/sec 206451 [ 5] 54.00-55.00 sec 285 MBytes 2.39 Gbits/sec 206297 [ 5] 55.00-56.00 sec 285 MBytes 2.39 Gbits/sec 206428 [ 5] 56.00-57.00 sec 285 MBytes 2.39 Gbits/sec 206497 [ 5] 57.00-58.00 sec 285 MBytes 2.39 Gbits/sec 206403 [ 5] 58.00-59.00 sec 285 MBytes 2.39 Gbits/sec 206335 [ 5] 59.00-60.00 sec 285 MBytes 2.39 Gbits/sec 206447 - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-60.00 sec 16.7 GBytes 2.39 Gbits/sec 0.000 ms 0/12385070 (0%) sender [ 5] 0.00-60.00 sec 11.4 GBytes 1.63 Gbits/sec 0.007 ms 0/12379986 (0%) receiver CPU Utilization: local/sender 91.6% (18.2%u/73.3%s), remote/receiver 24.8% (2.8%u/21.9%s) iperf Done.I have also started
netstat -iwhen server was saying about1.4 Gbpsigc2 1500 <Link#3> 00:d0:b4:02:1c:b8 889310806 0 0 28809147 0 0 igc2 - fe80::%igc2/64 fe80::2d0:b4ff:fe02:1cb8%igc2 0 - - 1 - - igc2 - 10.0.2.0/24 10.0.2.1 27 - - 45202 - - igc3 1500 <Link#4> 00:d0:b4:02:1c:b9 28898629 0 0 877373098 0 0 igc3 - fe80::%igc3/64 fe80::2d0:b4ff:fe02:1cb9%igc3 0 - - 0 - - igc3 - 192.168.18.0/24 192.168.18.1 5279 - - 5249 - - -
@roxy said in Topton N100 Reporting 402 MHz:
[ 5] 0.00-60.00 sec 11.4 GBytes 1.63 Gbits/sec 0.007 ms 3944234/12379986 (32%) receiver
Yup so 36% packet loss. I guess that's a bug in the earlier iperf version at the sender end that it doesn't show the loss value correctly.
Try testing in reverse (
-R) to make sure it behaves the same both ways.Try sending to/from pfSense directly to see if he throttling is obvious in one direction or on one NIC there.
-
Note it's actually pretty much the same as TCP so that rules out TCP windowing as an issue.