Can PFsense handle 10/10 Gbe Internet?

[remi_imer]

Hello All,

Wondering if the Pfsense gods can help a noob (me) identify the potential root cause of my problem.

I managed to get PFSense working but I am not sure where the limitations are coming from. I have a 10/10 GBe FTTH Internet. But I am only getting these speed below.

I got Pfsense installed bare metal on an i5 4570 (quad core) box with 4 GB ram. Its running Mellanox ConnectX-4 pci 3.0 x8 25GB SFP28 dual port. Port 1 is WAN, Port 2 is LAN and connected to Mikrotik CRS309-1G-8S+IN switch (switch mode only running SWOS) via DAC cable.
My PC is connected to the mikrotik switch via SFP+ 10Gbe port via single LC cable.

When do iperf3 to a localhost i get these speeds below.

[  4] local ::1 port 50053 connected to ::1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  2.68 GBytes  23.1 Gbits/sec
[  4]   1.00-2.01   sec  3.57 GBytes  30.3 Gbits/sec
[  4]   2.01-3.00   sec  3.67 GBytes  31.9 Gbits/sec
[  4]   3.00-4.00   sec  3.98 GBytes  34.2 Gbits/sec
[  4]   4.00-5.00   sec  4.07 GBytes  35.0 Gbits/sec
[  4]   5.00-6.00   sec  4.17 GBytes  35.8 Gbits/sec
[  4]   6.00-7.00   sec  4.21 GBytes  36.2 Gbits/sec
[  4]   7.00-8.00   sec  4.00 GBytes  34.3 Gbits/sec
[  4]   8.00-9.00   sec  4.10 GBytes  35.2 Gbits/sec
[  4]   9.00-10.00  sec  4.14 GBytes  35.6 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  38.6 GBytes  33.2 Gbits/sec                  sender
[  4]   0.00-10.00  sec  38.6 GBytes  33.2 Gbits/sec                  receiver

I looked Pfsense resources when running test and CPU only goes to 28% CPU utilization and 11% Memory usage.

Would love to hear your input on where the potential bottleneck could be.

Thanks in advance.

[stephenw10]

Run top -HaSP in a separate SSH session while you're testing so you can see how that load is spread across the cores.

Check the CPU is running at the expected frequency when under load.

Make sure the NICs are using the expected number of queues. Check the boot log. I'd expect to see 4 Tx and 4 Rx queues on each NIC.

Steve

[Dobby_]

I managed to get PFSense working but I am not sure
where the limitations are coming from.

• Your hardware could be.
  But here I think it is not pending on your local test
• The internet line and all involved knots or wires itself
  Try another time to test out and/or another server
  for the speed test please
• The using of PPPoE could be involved!
  One CPU core = one queues, but with PPPoE you are using only one CPU core!

I have a 10/10 GBe FTTH Internet. But I am only getting these speed below.

This is only what the test server was answering, how
many users were doing a test at the same time is not
written there! So the MTU could also be in game.

If you download something (a larger file) such a DVD iso
and you get the speed shown then you can often
multiplicate with 8 (*8) to get the real life throughput.

It is better then getting the numbers from one
speed test only in my eyes! If you have on the
other sites (Download) also not the given speed,
you will never archive that fully 10 GBit/s,
present on your site or not!

[RobbieTT]

@remi_imer said in Can PFsense handle 10/10 Gbe Internet?:

I have a 10/10 GBe FTTH Internet. But I am only getting these speed below.

Wow - that is the highest upload speed I have seen on speedtest.net. Here in the UK it usually chokes at around 4000 to 5000 Mbps. A year ago it was only around 1700 Mbps upload (if it was really quiet), so they have made major improvements to their server infrastructure to test at > 1Gbit.

We still find that speedtest.net offers the highest bandwidth for testing 10 Gbps links but never to saturation point. Other online testers were much more bandwidth limited or, in the case of fast.com, pretty much random number generators at higher bandwidths.

There was an article 18 months ago that painted a good picture of this issue. Things have improved since but it gives you a hint at the problems of stressing a 10 Gbps link via a single test site:

Benchmarking Broadband ISP Speed Testers On a 10Gbps Line

️

[stephenw10]

Yes, that's a very good point. Have you tested the line without pfSense connected? Can you see the expected rates against speedtest.net then?

[remi_imer]

@stephenw10 hi there,

Thanks for your input. I will try this when I get home later.

What do you mean by this? "Check the CPU is running at the expected frequency when under load" Does it mean if it's rated at 3.6 Ghz it should be reaching that during testing?

Thanks again. I will also check the boot log as per your suggestion.

[remi_imer]

@Dobby_ , hi there,

Thanks for your input.

Do you reckon if I put PFsense on a modern PC could have an impact? Perhaps I could also do this, I have another PC running Ryzen 5 6 core with 16GB ram. I was meant to configure this as Proxmox Server but I have not time to do it due to busy time at work.

Going to your MTU comment. My pfsense has it blank at present, which I believe defaults to 1500. Do I have to change this value if running 10 Gbit?

As for the PPPoE, I think I am not using this one. I don't remember selecting PPPoE when I installed Pfsense.

WIll also try other speed test servers and see what result I get. But i think Ookla is by far the most reliable one.

[remi_imer]

@RobbieTT

Thanks a bunch for your input and also for sharing the article. Perhaps that was true at the time of publication, however, things have significantly changed at present.

Speedtest.net definitely made some improvements to its server, such that it can give you close to 25 gigabit download and upload.

Check out this guy's article on how he achieved this. Link below. PS: he is using a Linux Router which he wrote himself called router 7

Router Build Blog

Test Results

This guy is using the same provider as I am, but he opted for the fastest tier available, which is 25 Gigabit. I am eligible for 25 Gigabit also, but I wanted to test the 10 Gigabit first since the upgrade path was for free

So yeah I am definitely convinced that there are some hardware limitations of some sort.

[remi_imer]

@stephenw10 hi there, I will test with just my PC and see how it goes. Will keep everyone posted.

[stephenw10]

@remi_imer said in Can PFsense handle 10/10 Gbe Internet?:

Does it mean if it's rated at 3.6 Ghz it should be reaching that during testing?

Yes. We have seen systems that didn't enable turbo mode or default to the lowest CPU speed for example.
The CPU speed is usually shown on the dashboard but you can also check:
sysctl dev.cpu.0

But check the per-core usage with top, make sure no core is at 100%

Steve

[NollipfSense]

Dang, I am moving to where ever this place is...
login-to-view

[RobbieTT]

@NollipfSense said in Can PFsense handle 10/10 Gbe Internet?:

Dang, I am moving to where ever this place is...
login-to-view

Well at least in the UK I can glance over and check the season with Stonehenge. None of that Swiss Watch precision needed; that tech will never catch on.

Anyway, I am sporting a new imported abacus. You won't believe how thin and light it is.

[Dobby_]

@remi_imer said in Can PFsense handle 10/10 Gbe Internet?:

Do you reckon if I put PFsense on a modern PC could have an impact?

For sure but if I am in your situation I would try out that
with the actual given hardware once more.

Perhaps I could also do this, I have another PC running Ryzen 5 6 core with 16GB ram. I was meant to configure this as Proxmox Server but I have not time to do it due to busy time at work.

Could be nice or also not!

Going to your MTU comment. My pfsense has it blank at present, which I believe defaults to 1500. Do I have to change this value if running 10 Gbit?

If the MTU is not the same on all device in that row,
you may be seeing other numbers as a result then!

As for the PPPoE, I think I am not using this one. I don't remember selecting PPPoE when I installed Pfsense.

If so you will be nailed to one CPU core "only"!
And that means also only one queue for the entire
wan traffic, would good to be knowing about.

Will also try other speed test servers and see what result I get. But i think Ookla is by far the most reliable one.

Ok, but perhaps you may be trying it out at other
or different time frames!

[remi_imer]

@Dobby_

Am curious about this one. How do I ensure I move away from this?

If so you will be nailed to one CPU core "only"!
And that means also only one queue for the entire
wan traffic, would good to be knowing about.

[stephenw10]

I'd be amazed if you're using PPPoE with 10G.

But it's caused by the fact that PPPoE is not IP and cannot take advantage of hardware/driver hashing to divide traffic across queues:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Very unlikely to apply here but you should check the per-core usage to be sure.

Steve

[remi_imer]

@stephenw10 hi there,

I did exactly as you said and this is what I found during testing. I dont know what am i suppose to be looking at here

last pid: 52273;  load averages:  0.24,  0.23,  0.18                                                                up 3+12:58:31  01:27:23
204 threads:   7 running, 162 sleeping, 35 waiting
CPU 0:  0.0% user,  0.0% nice,  0.0% system, 58.8% interrupt, 41.2% idle
CPU 1:  0.0% user,  0.0% nice,  0.0% system, 50.0% interrupt, 50.0% idle
CPU 2:  0.0% user,  0.0% nice,  0.4% system, 12.2% interrupt, 87.4% idle
CPU 3:  0.0% user,  0.0% nice,  0.0% system,  6.1% interrupt, 93.9% idle
Mem: 21M Active, 47M Inact, 404M Wired, 82M Buf, 3363M Free
Swap: 3656M Total, 3656M Free

[remi_imer]

@stephenw10

I also did what you suggested here.

/root: sysctl dev.cpu.0
dev.cpu.0.temperature: 57.0C
dev.cpu.0.coretemp.throttle_log: 0
dev.cpu.0.coretemp.tjmax: 100.0C
dev.cpu.0.coretemp.resolution: 1
dev.cpu.0.coretemp.delta: 43
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc
dev.cpu.0.cx_usage_counters: 43323265 0
dev.cpu.0.cx_usage: 100.00% 0.00% last 277us
dev.cpu.0.cx_lowest: C1
dev.cpu.0.cx_supported: C1/1/1 C2/2/148
dev.cpu.0.freq_levels: 3201/84000 3200/84000 3000/76349 2900/73036 2700/66595 2500/60408 2300/53689 2200/50804 2000/45218 1800/39849 1700/37248 1500/31535 1300/26734 1100/22147 1000/19936 800/15661
dev.cpu.0.freq: 3201
dev.cpu.0.%parent: acpi0
dev.cpu.0.%pnpinfo: _HID=none _UID=0 _CID=none
dev.cpu.0.%location: handle=\_PR_.CPU0
dev.cpu.0.%driver: cpu
dev.cpu.0.%desc: ACPI CPU

[remi_imer]

Here's an update.

I connected my PC directly to the FTTH OTO Socket without pfsense router and perform the test.

Unfortunately the result is the same with or without pfsense router. Perhaps there is an issue with my 10 Gb NIC. Or there is an issue at the fiber cable somewhere.. or at my ISP. Will call them tomorrow to find out.

[stephenw10]

Ah, well that's mostly good then.

No CPU core is at 0% idle (100% used). The loading could be spread better.

The CPU is running at 3201MHz. The extra 1 there implies turbo is enabled.

So it should be capable of more if there is more to be had.

You might try a local test between two 10G interfaces to confirm that.

Steve

[remi_imer]

Latest Update.

Goal of closer to 10 Gigabit reached So to answer my own question. Yes PFsense can indeed handle 10 Gigabit internet.

Thank you all for your inputs. I highly appreciate it.

Next goal is the 25 Gigabit. I hope TNSR is polished and have GUI interface by then