Low bandwidth on initial install

twoj

So, I'm poking around my system.
And I ran this command:

pciconf -l -BbcevV igb0@pci0:1:0:0

to check the PCIe bus capability of the IGB INTEL NIC. The NIC should capable of handling 1GB speeds. However it is connected via PCIe 2.0 x 1???

This would mean that the MAX bus throughput speed is 500MB/s which would be in line with my tests.

Can someone please confirm this?
Thanks

igb0@pci0:1:0:0:	class=0x020000 card=0x0000ffff chip=0x15338086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xdf500000, size 524288, enabled
    bar   [18] = type I/O Port, range 32, base 0xe000, size 32, enabled
    bar   [1c] = type Memory, range 32, base 0xdf580000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 5 messages
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected
    ecap 0003[140] = Serial 1 00e05cffff681859
    ecap 0017[1a0] = TPH Requester 1

DaddyGo

@twoj

look elsewhere for the reason for the value obtained....
the Intel I210 is a particularly good NIC for pfSense:
https://www.freebsd.org/cgi/man.cgi?igb(4)

perhaps:
https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html
https://docs.netgate.com/pfsense/en/latest/config/advanced-setup.html#Networking

edit: BTW: we know nothing else about its configuration, so we would welcome a more detailed description
(ISP, WAN type, installed packages, cable connections, etc.)

twoj

@DaddyGo
Thanks for the reply, before even getting into the NIC and drivers:

Per lane (each direction):
PCI-E v1.x: 250 MB/s (2.5 GT/s)
PCI-E v2.x: 500 MB/s (5 GT/s)
PCI-E v3.0: 1 GB/s (8 GT/s)
PCI-E v4.0: 2 GB/s (16 GT/s)

16 lane slot (each direction):
PCI-E v1.x: 4 GB/s (40 GT/s)
PCI-E v2.x: 8 GB/s (80 GT/s)
PCI-E v3.0: 16 GB/s (128 GT/s)

[https://serverfault.com/questions/399866/what-is-the-maximum-supported-data-rate-for-pcie]

With pciconfig, it reports

   cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)

PCI-Express 2 x1(x1)
So the theoritical max based on the PCI-E speed is only 500MB/s regardless what the card can do.

My setup is:

ISP: xfinity; I can get consistent 900+MB/s with speedtest on multiple computers.
pfSense - vanilla install, I just downgraded to v 2.4.4 p3 and it seems more consistent with bandwidth. No more run-away 'pf purge' process taking 10-15% CPU when the box is idle.
Also, I applied all the tweaks from:
https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html
The only ones I did not do is the chkecksum offload; that is still default setting.
WAN cable connected to the pfSense box (20 feet)
LAN cable connected between pfSense box and my Laptop (3 feet)

When using the WAN cable in my laptop, I get 900+MB/s download consistently.
When I add pfSense and route traffic through it, I only gets ~420-450MB/s download.

I will set up iperf on another local computer to further isolate the problem.

I also noticed the igb driver is quite old, anybody tried to get the latest intel driver for FreeBDS installed?

Thanks!

DaddyGo

what kind of MOBO is this, that has a 2.0 pcie version?
pfSense is always current for the FreeBSD version (this is the basis) not as easy to replace a driver as under windows

FYI:
https://forum.netgate.com/topic/154337/pfsense-2-4-5-release-p1-now-available

older versions are not recommended

twoj

@DaddyGo
https://forum.netgate.com/topic/154337/pfsense-2-4-5-release-p1-now-available
Nice! Hot off the press!

"What kind of MOBO is this, that has a 2.0 pcie version?"

I know, I could not believe it either... But if we are to trust pciconf then it reports it as PCI-Express 2 link x1

Would you know any other way to confirm this in FreeBSD?

This is the box:
https://www.aliexpress.com/snapshot/0.html?spm=a2g0s.9042647.0.0.1f5b4c4d5KCYT1&orderId=5004244627695702&productId=4000618318595

I figure'd I give it a try, but if PCIE 2.0 turns out to be true... then its going back.

Thanks!

DaddyGo

@twoj Would you know any other way to confirm this in FreeBSD?
try install a plain linux on it first and see what your box knows

and

This is the box:
https://www.aliexpress.com/snapshot/0.html?spm=a2g0s.9042647.0.0.1f5b4c4d5KCYT1&orderId=5004244627695702&productId=4000618318595

huhuhuhu these are Chinese wonders, rather get this (either
can be good): https://www.pfsense.org/products/

NOCling

@twoj said in Low bandwidth on initial install:

PCI-Express 2 x1(x1)
So the theoritical max based on the PCI-E speed is only 500MB/s regardless what the card can do.

You have to distinguish between bits and bytes.
Your Card have 2,5GT/s PCIe, its round about 500MB/s, but its LAN Speed ist round about 112MB/s.
Your bottleneck is somewhere else.

twoj

@NOCling
NoCling... I believe you are right.

I just tested pfSense with ipef (over a static route) and I'm getting 920Mbits/sec transfer rate with IS 1GB/s

So back to square one. Ahhh... I wished this was hardware problem.

In my current setup I'm my xfinity router is connected to the WAN interface of pfSense; my laptop is connected to the LAN interface of pfSense.

My intention is to throw out the xFinity router/modem and use Arris Modem instead, which would then connect to pfSense WAN interface.

Is the existing DHCP setup much different than the PPOE in terms of performance?

Thanks!

NOCling

Try 2.4.5p1, its there.

Both NICs (LAN/WAN) Intel I210?

twoj

@NOCling
Yes. All 6 are I210. :-)

I'm running 2.4.5.p1; I'm applying all harware tweaks again, still no joy. iPerf going 920MB/s regardless.

DaddyGo

@twoj

I think it's unthinkable, even if it's Chinese hardware, to install 6 pieces of I210 and they don't perform.

What do you want to achieve?

@twoj "Is the existing DHCP setup much different than the PPOE in terms of performance?"

Yes, PPPOE has its drawbacks, but it depends on what you use the box for,.....in normal use this is not significant.
Just think of MTU:
https://www.sonicwall.com/support/knowledge-base/how-can-i-optimize-pppoe-connections/170505851231244/

do you want to run IPS (netmap) things on the WAN interface?
although Bill does not describe the goal (IPS setup) as having the IPS on the WAN interface
(it prefers this to the LAN interface, if you just don't want to observe what's happening on the WAN)

twoj

@DaddyGo said in Low bandwidth on initial install:

I think it's unthinkable, even if it's Chinese hardware, to install 6 pieces of I210 and they don't perform.

I'm very confused.

iperf is good at ~920MB/s: testing from within LAN over to WAN then then to the local server.
CPU utilization is very low in 2.4.5p1; way better than in their older version; so its not the CPU problem. I'd say its about 3-5% for short burst times.
I tried most of the HW teaks and still no change. What the teaks do change is how quickly I can get to top speed of ~400MB/s with speed test.
I have replaced psSense setup with Zyxcel USG210 firewall, this one too tops off at ~400MB/s

Any suggestions even where I should look at next?
a) Could it be something else like the SSD that is slow and making things lag?
b) Is speedtest appropriate test for bandwidth? Or is there something else that I can use to reverify the Internet speeds?

Thanks

DaddyGo

@twoj

the case is starting to get a little complicated

so I asked what you want to achieve and where exactly
can you make a system drawing and draw exactly what and where you are measuring?

you say that Zyxcel get exactly the same speed, then - this is not a case of the pfSense box and setup

twoj

@DaddyGo

the case is starting to get a little complicated

I thought it would be a slam dunk. It never fails. I may try load up pfsense on a real fast BOX, with higher end hardware just to see if I can get up to the 900MB/s range.

so I asked what you want to achieve and where exactly

I want to get rig of the xfinity cable modem/router/wifi device. I used to be able to manage it, but now they are getting more and more restricting. I have Arris modem that I plan to use instead, and I want to couple that with a real firewall setup. I could take the easy way out with just a router, but in today's times I feel I need something more.

My setup is pretty simple actually:
Current:
(Internet From ISP) -> xFinity Router -> WAN of pfSense -> LAN of pfSense -> Test Laptop. Here pfSense WAN is in DHCP mode.

Future:
(Internet From ISP) -> Arris modem -> WAN of pfSense -> LAN of pfSense -> 16Port Netgear Switch-> Home computers/devices. Here pfSense WAN will be in DHCP mode as well. No PPPoe.

exactly what and where you are measuring?

In 'Current' setup: SpeedTest -> from Test Laptop running speedtest to the internet.
In 'Current' setup: iperf -> from Test Laptop running speedtest to another computer that is connected to the xFinity Router (internal LAN, but on the 'WAN pfSense' side)

you say that Zyxcel get exactly the same speed, then - this is not a case of the pfSense box and setup

I did not run the iperf over the Zyxcel yet. It would be interesting to see what the numbers are.

Any other things I should be looking into in the pfSense box?

Thanks!

DaddyGo

so you have a double -NAT here:

@twoj
Current:
(Internet From ISP) -> xFinity Router -> WAN of pfSense ->

try it with Zyxcel too, it will be a good starting point
what did you say your exact ISP speed (subscription)

+++++++
https://forum.netgate.com/topic/100945/how-much-throughput-lost-using-pfsense
https://docs.netgate.com/pfsense/en/latest/interfaces/low-throughput-troubleshooting.html

twoj

@DaddyGo said in Low bandwidth on initial install:

What did you say your exact ISP speed (subscription)

1GB/s roughly..... 900MB/s Down and 40MB/s up. I have no problem with hitting 40MB/s upload of course.

try it with Zyxcel too, it will be a good starting point

I did, similar result of about 350-425MB/s. Which makes me think that is the connection between:
xFinity Router -> WAN of (whatever firewall ) is where problem can exists. Possible MTU missmatch?

DaddyGo

@twoj

we have talked about so much already, about a lot of things

the fact is that you get a bad result with the other firewall too, I understand right?

to me this, definitely seems to be the maximum coming out of the xFinity Router (not normal operation, because your subscription is higher, but you need to check this device)

do one thing more please:

connect directly to the xFinity Router with a laptop and measure one

PS:
this is not an MTU problem, but this is how you can test
https://kb.netgear.com/19863/Ping-Test-to-determine-Optimal-MTU-Size-on-Router

twoj

@DaddyGo

we have talked about so much already, about a lot of things

Yes we have! I appreciate all the good info.

the fact is that you get a bad result with the other firewall too, I understand right?

Yes.

to me this, definitely seems to be the maximum coming out of the xFinity Router (not normal operation, because your subscription is higher, but you need to check this device)

Yea... it seems right now that the normal computer connected to xFinity can negotiate at top speed and pfSense and other hardware firewall appliance can not. I'd love to swap out the xFinity modem/router for the Arris right now, but I have to wait until school is over, since my wife teaches from home. I'd hate to be down to some unforeseen problem from xfitiny's side.

I may try running pfSense on beefy computer with enterprise nics in it; just to see what is what.

connect directly to the xFinity Router with a laptop and measure one

Any computer connect to the local LAN is getting top speed ~900MB/s down, but I will retest by directly plugging in.

I WILL GET TO THE BOTTOM OF THIS. :-)

DaddyGo

@twoj

it is clear what you need:

xFinity Router in bridge mode, if it exists for this type and your ISP allows it
or you mention a modem (Arris modem) that does not contain NAT per se and you get a public IP directly

the difference between the measurements is very large approx. 900 and 400

we didn't get ahead professionally, because this difference is not justified by the dual -NAT throughput, so there is still a cat hiding somewhere in the bag

if you have the opportunity to exchange, please come back to us afterwards (the curiosity moves the whole world )