Slow Throughput Gigabit Lan
-
Greetings to all.
First off, a quick and well deserved thanks and goodwork to all involved in this project. I'm really enjoying Pfsense and it's community.My issue is related to the throughput on gigabit lan in pfsense 2.2.2 .
My Hardware:Pfsense box:
Intel Core 2 duo 1 gig ram / dual Intel Pro 1000 nic (igb0-1)Network
dmesg | grep pci
igb0: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.4.0="">at device 0.0 on pci2
igb1: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.4.0="">at device 0.1 on pci2
rl0: <realtek 10="" 8139="" 100basetx="">port 0x1000-0x10ff mem 0x60900000-0x609000ff irq 21 at device 2.0 on pci3Linux Server:
Intel Core 2 duo 2 gig ram / dual Intel Pro 1000 nic e1000eI've been using iperf3 to test my local lan's speed after aquiring a new switch. I noticed that i've only been able to achieve about 422 Mbits/sec on average and 940 Mbits/sec on CentOS 7.
Initially on Pfsense i was getting aroung 281 Mbits/sec. I've search around the web for a solution, but could not find anything related to my issue.
I tried tuning my network cards as per https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards and https://calomel.org/network_performance.html to fiddle with some options, with some benefit, i guess. ( at one point I tried too many things and had to reinstall pfsense)I've tried using older kernel modules, i've tried compiling the windows driver in virtual machine w/ freebsd 10, with no luck.
In Advanced:Networking I currently have all options unticked (Disable hardware checksum offload, Disable hardware TCP segmentation offload, Disable hardware large receive offload) and seem to get better performance - 422 Mbits/sec.
If tried with Lagg fec,roundrobin,failover, same results, with firewall and without.Now, just for fun I have an old ibm x3650 m1 with dual bce and I installed pfsense 2.2.2 on it. I wanted to see if it was my hardware failing or low specs on my pfsensebox causing slow speeds, however I get even less throughput on server grade hardware. results : 138 Mbits/sec and this is a broadcom nic.
Strange. I'm stumped.
I'm new (sort of) to pfsense and freebsd, so i'm wondering if i'm doing something wrong here.
Is anybody else having these issues?
Thank you for your time.</realtek></intel(r)></intel(r)> -
Intel Core 2 duo 1 gig ram / dual Intel Pro 1000 nic (igb0-1)Network
This is the so called recommended Hardware for using pfSense, not more and not less!
What GHz your CPU comes with and how old is this CPU?rl0:<realtek 10="" 8139="" 100basetx=""></realtek>
Was this 100 MBit/s involved in the test?
server grade hardware. results : 138 Mbits/sec and this is a broadcom nic.
How actual is this hardware? And really it is a server grade hardware, but how good it is
supported by pfSense? CPU, RAM, HDD, Board, NICs,…..Here is a quite small list, by that you are able to indicate
_CPU Selection
The numbers stated in the following sections can be increased slightly for quality NICs, and decreased (possibly substantially) with low quality NICs. All of the following numbers also assume no packages are installed.
10-20 Mbps We recommend a modern (less than 4 year old) Intel or AMD CPU clocked at at least 500MHz.
21-100 Mbps We recommend a modern 1.0 GHz Intel or AMD CPU.
101-500 Mbps No less than a modern Intel or AMD CPU clocked at 2.0 GHz. Server class hardware with PCI-e network adapters, or newer desktop hardware with PCI-e network adapters.
501+ Mbps Multiple cores at > 2.0GHz are required. Server class hardware with PCI-e network adapters.Remember if you want to use your pfSense installation to protect your wireless network, or segment
multiple LAN segments, throughput between interfaces must be taken into account. In environments
where extremely high throughput through several interfaces is required, especially with gigabit interfaces,
PCI bus speed must be taken into account. When using multiple interfaces in the same system,
the bandwidth of the PCI bus can easily become a bottleneck._ -
Hi Frank.
To answer your first question:
What GHz your CPU comes with and how old is this CPU?
Intel(R) Celeron(R) D CPU 3.33GHz
Current: 1249 MHz, Max: 3333 MHzSecond question :
Was this 100 MBit/s involved in the test?
No, the 100 Mbit/s network card is my WAN.
I can see where the bottleneck could be in the corelation between the CPU and PCI bus speeds.
The server hardware in question "x3650 CPU 2x e-5405 2.0ghz 8 cores 10 gigs ram 2 x PCI-x broadcom network "I'm just wondering why my linux server gives me the full 940 Mbits/s with the same Motherboard and Network Card, and why I get 138Mbits/s on a totally different machine when installing pfsense 2.2.2 and 940 Mbits/s on the same machine with Centos Linux.
Maybe my modules are not configure properly?
Thanks for your help.
-
No, the 100 Mbit/s network card is my WAN.
Ok this is not counting then.
I can see where the bottleneck could be in the corelation between the CPU and PCI bus speeds.
The server hardware in question "x3650 CPU 2x e-5405 2.0ghz 8 cores 10 gigs ram 2 x PCI-x
broadcom network "How good is the broadcom card supported under pfSense?
I'm just wondering why my linux server gives me the full 940 Mbits/s with the same Motherboard and Network Card, and why I get 138Mbits/s on a totally different machine when installing pfsense 2.2.2 and 940 Mbits/s on the same machine with Centos Linux.
Linux is not pfSense and also not FreeBSD!
Maybe my modules are not configure properly?
Use iPerf from one PC to another PC through the pfSense Firewall,
so you will get the best results from those tests. -
I installed Freebsd 10.1 64bit. I tested bce with Iperf3 default options and got
[ ID] Interval Transfer Bandwidth Retr Cwnd [ 5] 0.00-1.00 sec 108 MBytes 906 Mbits/sec 0 255 KBytes [ 5] 1.00-2.00 sec 111 MBytes 930 Mbits/sec 0 215 KBytes [ 5] 2.00-3.00 sec 111 MBytes 935 Mbits/sec 0 238 KBytes [ 5] 3.00-4.00 sec 111 MBytes 935 Mbits/sec 0 249 KBytes [ 5] 4.00-5.00 sec 112 MBytes 936 Mbits/sec 0 253 KBytes [ 5] 5.00-6.00 sec 112 MBytes 936 Mbits/sec 14 253 KBytes [ 5] 6.00-7.00 sec 111 MBytes 934 Mbits/sec 0 255 KBytes [ 5] 7.00-8.00 sec 112 MBytes 936 Mbits/sec 0 255 KBytes [ 5] 7.00-8.00 sec 112 MBytes 936 Mbits/sec 0 255 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 5] 0.00-8.00 sec 983 MBytes 1.03 Gbits/sec 14 sender [ 5] 0.00-8.00 sec 0.00 Bytes 0.00 bits/sec receiver iperf3: the client has terminatedThis is on the same machine I install pfsense on today [x3650] The machine hasn't moved it's been plugged in the same switch, with the same cat6 cable, everything is the same except Pfsense 2.2.2 and freebsd 10.1 and I also installed Centos 7 and got 940 Mbits/s.
I copied the kernel modules from /boot/kernel/ if_bce.ko and if_bce.ko.symbols to my laptop then winscp'd everything back to a new install but still nothing more than 164 Mbits/s.
How good is the broadcom card supported under pfSense?
I'm not sure I comprehend.
Linux is not pfSense and also not FreeBSD!
I'm starting to see that.
-
Are there any steps I can take to start troubleshooting and norrowing down where the issue might be?
-
Please read my lips something is wrong with your testing method or the hardware otherwise the configuration!
And this is a must be! This SOHO hardware shown under the link will be able to bring ~435 MBit/s with
cheap Realtek NICs and a smaller CPU as you are using! Alix APU- Enable PowerD on your system
- High up the MBuffer for the NIC
This is on the same machine I install pfsense on today [x3650] The machine hasn't moved it's been plugged in the same switch, with the same cat6 cable, everything is the same except Pfsense 2.2.2 and freebsd 10.1 and I also installed Centos 7 and got 940 Mbits/s.
Once more again you can´t compare those systems against!
Linux is Linux and FreeBSD is FreeBSD and this is not the same!
And on top pfSense is based on FreeBSD but not all from FreeBSD is inside of the pfSense!!!!
CentOS is a Linux OS! And FreeBSD is a BSD OS they both do not firewalling with ruels and pf
and not SPI/NAT! But pfSense does this![ 5] 7.00-8.00 sec 112 MBytes 936 Mbits/sec 0 255 KBytes
This is for FreeBSD and this is a good value as I think.
I copied the kernel modules from /boot/kernel/ if_bce.ko and if_bce.ko.symbols to my laptop then winscp'd everything back to a new install but still nothing more than 164 Mbits/s.
Can be that it will matching often but also that it is not matching it in your case!
I'm not sure I comprehend.
Hm, comparing things against is not a solution in your case!
What is it interesting you what CentOS is able to get through or FreeBSD?
What pfSense is able to do is right interesting you!
This would be the best options for you as I see it right.- enable PowerD
- high up the MBuffer of the NICs
- don´t do a test on the pfSense machine it selfs!!!!!!!! Do it from one PC to another PC through pfSense! (LAN-LAN) throughput
-
Thanks Frank.
CentOS is a Linux OS! And FreeBSD is a BSD OS they both do not firewalling with ruels and pf
and not SPI/NAT! But pfSense does this!I've tried with Lagg fec,roundrobin,failover, same results, with firewall and without.
Even when I disabled the firewall I still could not get it to go full speed when pfsense is the iperf3 listening server.
Please read my lips something is wrong with your testing method or the hardware otherwise the configuration!
Since I'm new to Pfsense, where should i begin to look at my configurations if i'm not getting any errors?
You are definately right about Pfsense being different, and I get that, what I'm trying to see by installing Freebsd and Centos is to rule out that this is a malfunctioning piece of hardware.
From what I can see the hardware is functioning as it should on other systems, so the next step in my mind, is to look at the configuration or my testing method.This would be the best options for you as I see it right.
- enable PowerD
- high up the MBuffer of the NICs
- don´t do a test on the pfSense machine it selfs!!!!!!!! Do it from one PC to another PC through pfSense! (LAN-LAN) throughput
PowerD is enabled.
I have raise kern.ipc.nmbclusters="1000000" but this seems to make my system hangs so I raised it from 8000 to 16000.
Do you mean to test lan subnet to lan subnet.I have tested it from the pfsense machine itself. I start iperf3 -s on my other machine, login to pfsense execute iperf3 -c tohost I get :
[ ID] Interval Transfer Bandwidth [ 4] 0.00-1.01 sec 100 MBytes 832 Mbits/sec [ 4] 1.01-2.01 sec 112 MBytes 940 Mbits/sec [ 4] 2.01-3.01 sec 112 MBytes 940 Mbits/sec [ 4] 3.01-4.01 sec 112 MBytes 940 Mbits/sec [ 4] 4.01-5.01 sec 112 MBytes 941 Mbits/sec [ 4] 5.01-6.00 sec 111 MBytes 940 Mbits/sec [ 4] 6.00-7.00 sec 112 MBytes 941 Mbits/sec [ 4] 7.00-8.00 sec 112 MBytes 940 Mbits/sec [ 4] 8.00-9.00 sec 112 MBytes 941 Mbits/sec [ 4] 9.00-10.01 sec 113 MBytes 939 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.01 sec 1.08 GBytes 929 Mbits/sec sender [ 4] 0.00-10.01 sec 1.08 GBytes 929 Mbits/sec receiverWhen I reverse the roles, login to pfsense execute iperf3 -s and then login to my other machine and iperf3 -c tohost
I get :[ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 35.9 MBytes 301 Mbits/sec 0 192 KBytes [ 4] 1.00-2.00 sec 58.8 MBytes 494 Mbits/sec 0 199 KBytes [ 4] 2.00-3.00 sec 66.5 MBytes 558 Mbits/sec 0 199 KBytes [ 4] 3.00-4.00 sec 47.3 MBytes 397 Mbits/sec 0 199 KBytes [ 4] 4.00-5.00 sec 52.2 MBytes 438 Mbits/sec 0 201 KBytes [ 4] 5.00-6.00 sec 60.1 MBytes 504 Mbits/sec 0 202 KBytes [ 4] 6.00-7.00 sec 45.9 MBytes 385 Mbits/sec 0 202 KBytes [ 4] 7.00-8.00 sec 56.5 MBytes 474 Mbits/sec 0 204 KBytes [ 4] 8.00-9.00 sec 49.8 MBytes 417 Mbits/sec 0 204 KBytes [ 4] 9.00-10.00 sec 45.3 MBytes 380 Mbits/sec 0 204 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 518 MBytes 435 Mbits/sec 0 sender [ 4] 0.00-10.00 sec 518 MBytes 434 Mbits/sec receiverDo you have any suggestions for me concerning the configurations.
From what i've searched i do not get any errors in dmesg.I have been looking at this for a while now. I searched on the net for a week trying a multitude of options.
I gave up and decided to ask on the forum for some help.cat /boot/loader.conf.local
hw.igb.rxd="2048" hw.igb.txd="2048" hw.igb.buf_ring_size="4096" hw.igb.max_interrupt_rate="16000" hw.igb.enable_aim="1" net.link.ifqmaxlen="1024" kern.ipc.nmbclusters="131072" kern.sched.interact="5" kern.sched.slice="3" net.inet.tcp.tcbhashsize="512"System tunables:
net.inet.ip.fastforwarding 1 net.isr.dispatch direct dev.igb.0.fc 0 # but that did not help much. -
Do you mean to test lan subnet to lan subnet.
PC with iPerf server –---- pfSense ------ PC with iPerf host
Can be in the same subnet but must not be.So I think it would be really came more to the point of true throughput
because the pfSense is only passive acting in the test. -
When you enabled powerD, did you set it to maximum or hiadaptive?
-
Hi.
I set it to maximum.
# sysctl -a | egrep -i 'hw.machine|hw.model|hw.ncpu' hw.machine: amd64 hw.model: Intel(R) Celeron(R) D CPU 3.33GHz hw.ncpu: 1 hw.machine_arch: amd64 # cat /var/log/dmesg.boot | grep CPU: CPU: Intel(R) Celeron(R) D CPU 3.33GHz (3333.54-MHz K8-class CPU)I have done the same to my test machine x3650. I'll be looking over a packet capture in the near futur and post back.
Thanks for your interrest.
I'm still looking for a solution.