10gb wan/lan setup tweaks?

zennb1

Hi.
So I'm running an i5 12600h which has an intel sfp connection to my 10gb switch. I have a few 10gb clients on my lan which testing with iperf3 gives around 9.40/9.50GB on transfer tests.

I just wondered if there are any tweaks I should be doing on the pfsense side which I'm not aware of?

My isp uses a 10gb ont but my profile is currently 2gb/2gb, 8gb/8gb is available.

Thanks

stephenw10

There is some tuning you can try to the buffers/queues on the NICs etc. But the main thing is to use the best NICs you can.

zennb1

@stephenw10 ok thanks.

Im using intel x540 on the wan side as its rj45 and intel x710 onboard sfp+ on the lan side to my zyxel switch which also has sfp and using a custom dac cable.

I have found with iperf tests under windows I need to use the -P 4/8 to achieve over 9.0gb
[SUM] 0.00-10.01 sec 11.1 GBytes 9.49 Gbits/sec sender
[SUM] 0.00-10.01 sec 11.0 GBytes 9.47 Gbits/sec receiver

is this acceptable?

thanks

stephenw10

Check the per usage shown in the output of top -HaSP at the command line while testing.

Make sure no CPU core is pegged at 100%.

Also check: https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#general-tuning

zennb1

@stephenw10 thanks will check it out. I think the cpu's look ok.

Gblenn

@zennb1 said in 10gb wan/lan setup tweaks?:

@stephenw10 ok thanks.

Im using intel x540 on the wan side as its rj45 and intel x710 onboard sfp+ on the lan side to my zyxel switch which also has sfp and using a custom dac cable.

I have found with iperf tests under windows I need to use the -P 4/8 to achieve over 9.0gb
[SUM] 0.00-10.01 sec 11.1 GBytes 9.49 Gbits/sec sender
[SUM] 0.00-10.01 sec 11.0 GBytes 9.47 Gbits/sec receiver

is this acceptable?

thanks

It's quite normal to need to run parallel streams to max out a 10 Gbit connection. And ~9.5 Gbit/s is actually max what you would see. Similarly on a 1 Gig connection you would get around 950 Mbit/s... So that is definitely more than just acceptable!

But what you end up getting in a WAN to LAN connection will depend on what services you run as well. If you run Suricata in Inline mode you would probably see something in the range of 3.5-4.5 Gbit perhaps.

I'm running pfsense virtualized on an i5 11400 and get around 8.2 Gbit max when running speedtest as well as iperf testing across two firewalls connected at the same switch (WAN side). I get those speeds with Suricata in Legacy mode though, which I think is excellent!

zennb1

@Gblenn thanks for your comment. Really helpful. I do see occasional entries in the "retries" under iperf. I hoping this is to be expected from time to time?

Thanks

Gblenn

@zennb1 Hmm not sure that is the case... At least I have not seen any retries. It either works or it doesn't for me, and if it doesn't it's my fault typically, like incorrect IP, not opened the port in pfsense (when testing over WAN) etc.

On LAN at least, I think you should expect quite clean runs...

What NIC are you using on the PC?

Bob.Dig

@Gblenn said in 10gb wan/lan setup tweaks?:

I'm running pfsense virtualized on an i5 11400 and get around 8.2 Gbit max when running speedtest as well as iperf testing across two firewalls connected at the same switch (WAN side).

What Hypervisor? I can't get more than 3-4 Gbit with pfSense on a vSwitch.

zennb1

@Gblenn

Pfsense ms01
Lan side X710 with dac to zxyel xs1930 switch
Wan side 10gtek 10gb x540 rj45 x1 port

Pc 10gtek x540 rj45

Not masses of retries but they happen from time time.

stephenw10

Seeing a few retries at the limit of the hardware is expected.

Gblenn

@Bob-Dig I'm running Proxmox and have the NIC's passed through (IOMMU).
In the test between two firewalls, both are virtualized and one is Sophos XG running on an i5 10400 (4 cores assigned which incidentally I also have for pfsense on the i5 11400).

I have x520 NIC's for both Firewalls, and they connect on WAN via DAC's to a Mikrotik 10G switch where the fiber comes in.
My ISP has been kind enough to give me two IP's for this setup...

So the "external" test using iperf was :
PC client on Sophos LAN->WAN to (FQDN) WAN->pfsenseLAN to iperf Server on a Linux VM running on a third Proxmox machine (10 Gig internal network).

Otherwise I get similar numbers as @zennb1 when running iperf on LAN only.

Gblenn

@stephenw10 Would that be limit of the NIC, or the CPU? I don't think I have ever seen any retries actually...

Gblenn

@zennb1 said in 10gb wan/lan setup tweaks?:

@Gblenn

Pfsense ms01
Lan side X710 with dac to zxyel xs1930 switch
Wan side 10gtek 10gb x540 rj45 x1 port

Pc 10gtek x540 rj45

Not masses of retries but they happen from time time.

Aha, never tried the 10GTek. I have an TPLink TX-401 (Marwell chipset). And it's behaving a bit strange when running speedtest, capping out at 3 Gbit roughly in DL. Only way to solve that is to run driver repair, which fixes the problem until next time i shut down and start the PC (just restart seems ok though).

zennb1

@Gblenn I found the 10gtek does the same but by forcing windows settings to 10g full duplex for the card it's cured it on reboot.
My 10gb ms01

Gblenn

@zennb1 Thanks, but that did not work with my card. It seems though that any change I do to the settings, like shifting from Automatic to 10G full duplex, triggers som initiation of the device which resolves the issue. Disabling and enabling for example, has the same effect. But nothing that I do seems to "stick"... And since I have the driver SW as a shortcut it's quite simple to just run it...