Poor 10gbps WAN throughput
-
Hmm, nothing jumps out there. What throughput were you seeing when that was taken?
-
@stephenw10 2000 ish there. The ISP engineer came out today actually and used a thunderbolt OWC branded rj45 adapter on OSX. He was able to get closer to 8000. So I got no clue what the issue is. Would you expect 5000 to be reasonable?
-
I would expect to see something in the 3-4Gbps range with a TCP test using full sized packets. You can see from the values there that it scales to that sort of range until CPU cores start to hit 100%.
Was the ISP engineer testing against the same servers? 'Special' ISP test site?
-
@stephenw10 Same test site but it does seem weird. Would it be worth setting jumbo packets on the MTU?
-
Only if the WAN supports an MTU >1500 which I doubt it does.
Are you seeing any errors or collisions on the interfaces?
Are you able to reassign the WAN NIC to something internal and test between the 10G NICs directly?
Or test against a local 10G device on the WAN side? -
Actually, I have the same problem. It caps at 2.5Gbps with iperf3 (regardless of number of threads) and at. 3.5Gbps with speedest. Both ways, up and down, numbers are the same.
Wiring something directly to ONT gives full 8gbps, which are supposed to be there.Also netgate 6100, attached to ONT via 10GBase-T.
I was expecting to get at least 6 with iperf, looking at specs.
Are device specs for standard frames or jumbo frames?
-
@antst said in Poor 10gbps WAN throughput:
Also netgate 6100, attached to ONT via 10GBase-T.
Via an RJ-45 SFP+ module?
How where you running speedtest? From a client behind the 6100?
-
@stephenw10
I tried all possible ways :)but then looked at the web, and it turned out that this is actual cap of 6100.
Despite all my love for pfSense, flashed vyos on 6100 (took me hours to covert config) and throughput doubled.
Sad that TNSR is not available anymore for homelab. -
Mmm, I'd expect to see something in the 3-4Gbps range for most setups like that. There are a lot of factors though, WAN speed, latency, speedtest server etc.
-
yep, in the best attempts, I was able to push 3.5Gbps.
vyos pushes 5.5 easily (sometimes higher) with exactly the same infra and speedtest server.
Still need to get new router though. direct connect to the WAN from decent hardware pushes to 8Gbps. -
Mmm, TNSR will push that easily on the 6100. You might email our sales guys and see what they can do for you. At least a trial to prove it out.
-
So I've been testing various things over the past few weeks.
I've changed my NIC to an X550-t2 and reinstalled win11. I was previously using an ASUS 10gbe card but many reviews said that it's flakey. I tend to agree.
I saw a number of forum posts from various sources outside of here, who suggested to disable any NIC onboard from the bios before installing, install the 10gb NIC and re-enable onboard NICs.
I have 2 onboard Intel 1gb NICs and to my surprise this actually worked. I'm literally using the same drivers for the 10gbe NIC and the speed is double. I'm now getting 4000 instead of 2000.
4000 is still below what it should be so I'm inclined to believe I'm hitting hardware limitations on the 6100. I may take the offer of trialling tnsr to see where that takes me. Failing that, I'm going to create my own pfsense box.
-
Hmm, that was just changes on the Windows host running the test client?
-
I am pretty sure TNSR will do :)
Trial is no go, just wasting time for configuration. Is there something to assist in conversion of pfSense config to TNSR one? -
Yes that's right. Same drivers too.
It seems a lot of users of win11 were having the same slow speeds as me. The only change I made was jumbo frames on pfsense and on the NIC itself, as well as upping buffers to their max on win11 NIC settings. That seemed to give small differences in speed but nothing earth shattering. Disabling onboard NICs made the most difference.
-
@vertigo8
Asus 10Gbe card, if I recall right, is on AQC107 chip or something. It hardly can push up to 10Gbps :)
It is just cheap chip made for mainstream, where 5G is supposed to be enough for most.
to get max from AQC107, is I recall correctly my experience, requires to have decent card on other side and tune both sides. AQC107 is tricky and sensitive to many things.
If you have choice, better use intel, even if it is more expensive.
I had to go with it only because I didn't have much choice in terms of TB4 10GbE nic. -
@antst said in Poor 10gbps WAN throughput:
@vertigo8
Asus 10Gbe card, if I recall right, is on AQC107 chip or something. It hardly can push up to 10Gbps :)
It is just cheap chip made for mainstream, where 5G is supposed to be enough for most.
to get max from AQC107, is I recall correctly my experience, requires to have decent card on other side and tune both sides. AQC107 is tricky and sensitive to many things.
If you have choice, better use intel, even if it is more expensive.
I had to go with it only because I didn't have much choice in terms of TB4 10GbE nic.I get 8+ with the same chip on a TPLink card, but as I wrote earlier, only after doing a driver repair using the Marwell installer. And I have to do that after every off/on of the PC, otherwise it caps out at ~2Gbit.
-
@Gblenn Did you knwo you can do this:
https://answers.microsoft.com/en-us/windows/forum/all/how-can-i-prevent-automatic-updating-a-specific/9967b1cf-dc6f-495d-82be-4ab3f3207ff1 -
@Gblenn
yep, seems that throughput in range 7-8Gbps is about maximum for this chip.
occasionally I see up to 9, but rare. -
@antst said in Poor 10gbps WAN throughput:
@Gblenn
yep, seems that throughput in range 7-8Gbps is about maximum for this chip.
occasionally I see up to 9, but rare.Well that was a test through, using speedtest and not so much testing max throughput of the card itself. I think I tested with iperf, back when I got the card, and saw maximum possible of around 9.4, if I remember correctly. So I don't think it is such a bad product actually...
[EDIT]
Just tested again and got this result
