[solved] significant problem with throughput through our pfsense
-
Dear all,
Currently we have a significant problem with throughput through our pfsense.
Throughput is very small if we send data from one local segment to another local segment and also,
if we send data over the WAN-Connection, but we have full bandwidth available if we send from any host to one of the pfsense interfaces.
Throughput through the pfsense firewall is 12.7 Mbits/s, compared to the expected bandwidth of approx.. 9 Gbit/s, which is achieved when testing to the pfsense interfaces.
We use the pfsense as the central router in our network. All hosts in our network are connected with 10GBE Nic's.
The pfsense also have a 10GBE WAN P2P connection to a second location. The latency of this connection is 13ms.
In an attempt to reach full speed over the 10GBE WAN connection we have made some adjustments in the TCP windows size on the hosts and on the pfsense (incremented to 16B).Also, to help find the error we disabled the entire firewall functionality on the pfsense but without any result.
Please can you investigate and help resolve why throughput through the pfsense is significantly reduced.
Please see below for information about the configurations.With kind regards,
Teste scenarios with iperf 2
host a 10.100.4.2/24
host b 10.100.5.8/24pfsense
10.100.4.1/24
10.100.5.1/2410.100.5.8 -> 10.100.4.1 = 8.60 Gbits/sec
10.100.4.2 -> 10.100.5.1 = 8.88 Gbits/sec
10.100.5.8 -> 10.100.4.2 = 12.7 Mbits/sec
10.100.4.2 -> 10.100.5.8 = 12.7 Mbits/secPFSense Version:
2.3.2-RELEASE-p1 (amd64)
10.100.5.1 -> vlan on bxe2
10.100.4.1 -> vlan on bxe2tunables
net.inet.tcp.recvspace 16777216
net.inet.tcp.sendspace 16777216
net.raw.sendspace 16777216
net.raw.recvspace 16777216
kern.ipc.maxsockbuf 33554432
kern.ipc.nmbclusters 1000000firewall & nat
disable firewall true
disable firewall scrubnetworking
Disable hardware checksum offload false
Enable device polling false
Disable hardware TCP segmentation offload false
Disable hardware large receive offload falsehardware
Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
32 CPUs: 2 package(s) x 8 core(s) x 2 SMT threads
130GB RAMHP FlexFabric 10Gb 2-port 533FLR-T Adapter
Location Embedded
Firmware 7.14.79
2c:44:fd:91:0d:50
2c:44:fd:91:0d:54HP Ethernet 10Gb 2-port 530T Adapter
Location Slot 3
Firmware 7.14.79
14:02:ecbd:28
14:02:ecbd:2cHP Ethernet 1Gb 4-port 331i Adapter - NIC
Location Embedded
Firmware 17.4.41
1c:98:ec:15:69:2c
1c:98:ec:15:69:2d
1c:98:ec:15:69:2e
1c:98:ec:15:69:2fifconfig
bxe0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=507bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 2c:44:fd:91:0d:50 inet6 fe80::2e44:fdff:fe91:d50%bxe0 prefixlen 64 scopeid 0x1 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active bxe1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=507bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 2c:44:fd:91:0d:54 inet6 fe80::2e44:fdff:fe91:d54%bxe1 prefixlen 64 scopeid 0x2 inet 192.168.11.3 netmask 0xffffff00 broadcast 192.168.11.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active bge0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500 options=c019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate>ether 1c:98:ec:15:69:2c nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect bge1: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500 options=c019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate>ether 1c:98:ec:15:69:2d nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect bge2: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500 options=c019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate>ether 1c:98:ec:15:69:2e nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect bge3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500 options=c019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate>ether 1c:98:ec:15:69:2f nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect bxe2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=507bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 14:02:ec:cd:bd:28 inet6 fe80::1602:ecff:fecd:bd28%bxe2 prefixlen 64 scopeid 0x7 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active bxe3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500 options=507bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 14:02:ec:cd:bd:2c nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (none) pflog0: flags=100 <promisc>metric 0 mtu 33160 pfsync0: flags=0<> metric 0 mtu 1500 syncpeer: 224.0.0.240 maxupd: 128 defer: on syncok: 1 enc0: flags=0<> metric 0 mtu 1536 nd6 options=21 <performnud,auto_linklocal>lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc nd6 options=21 <performnud,auto_linklocal>bxe2_vlan1016: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28 inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1016 prefixlen 64 scopeid 0xd inet 10.100.1.1 netmask 0xfffffff0 broadcast 10.100.1.15 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 1016 vlanpcp: 0 parent interface: bxe2 bxe2_vlan1017: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28 inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1017 prefixlen 64 scopeid 0xe inet 10.100.4.1 netmask 0xffffff00 broadcast 10.100.4.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 1017 vlanpcp: 0 parent interface: bxe2 bxe2_vlan1018: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28 inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1018 prefixlen 64 scopeid 0xf inet 10.100.9.1 netmask 0xffffff00 broadcast 10.100.9.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 1018 vlanpcp: 0 parent interface: bxe2 bxe2_vlan1019: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28 inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1019 prefixlen 64 scopeid 0x10 inet 10.100.10.1 netmask 0xffffff00 broadcast 10.100.10.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 1019 vlanpcp: 0 parent interface: bxe2 bxe2_vlan1501: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28 inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1501 prefixlen 64 scopeid 0x11 inet 10.100.5.1 netmask 0xffffff00 broadcast 10.100.5.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 1501 vlanpcp: 0 parent interface: bxe2 bxe2_vlan1502: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28 inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1502 prefixlen 64 scopeid 0x12 inet 10.100.6.1 netmask 0xffffff00 broadcast 10.100.6.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 1502 vlanpcp: 0 parent interface: bxe2 bxe2_vlan2518: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28 inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan2518 prefixlen 64 scopeid 0x13 inet 10.100.2.1 netmask 0xfffffff0 broadcast 10.100.2.15 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 2518 vlanpcp: 0 parent interface: bxe2 bxe2_vlan2519: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28 inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan2519 prefixlen 64 scopeid 0x14 inet 10.100.3.1 netmask 0xffffff00 broadcast 10.100.3.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 2519 vlanpcp: 0 parent interface: bxe2 bxe2_vlan3015: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28 inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan3015 prefixlen 64 scopeid 0x15 inet 10.100.7.1 netmask 0xffffff00 broadcast 10.100.7.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 3015 vlanpcp: 0 parent interface: bxe2 bxe0_vlan3016: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 2c:44:fd:91:0d:50 inet6 fe80::2e44:fdff:fe91:d50%bxe0_vlan3016 prefixlen 64 scopeid 0x16 inet 10.100.8.1 netmask 0xffffff00 broadcast 10.100.8.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 3016 vlanpcp: 0 parent interface: bxe0 bxe0_vlan3017: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=303 <rxcsum,txcsum,tso4,tso6>ether 2c:44:fd:91:0d:50 inet6 fe80::2e44:fdff:fe91:d50%bxe0_vlan3017 prefixlen 64 scopeid 0x17 inet 10.100.11.1 netmask 0xffffff00 broadcast 10.100.11.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active vlan: 3017 vlanpcp: 0 parent interface: bxe0</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,auto_linklocal></promisc></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></broadcast,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate></broadcast,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate></broadcast,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate></broadcast,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate></broadcast,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast>
-
Is this a new setup or working setup that has failed?
-
It looks like you tested from a Device on Subnet A to PFSense's IP on Subnet B and vice-versa, and got in the 8Gb/s range. Then you tried to test between the two devices in the different subnets and got only in the 13Mb/s range.
Have you tried placing the two devices on the same subnet and iperf'd between them that way to rule-out an issue between the two devices?
-
Hey Guys,
thank you for responding, we appreciate that you spend your time.
Of course i will answer your questions to get some more suggestions from you.@CC:
Is this a new setup or working setup that has failed?
Its a new setup which we are testing here.
It looks like you tested from a Device on Subnet A to PFSense's IP on Subnet B and vice-versa, and got in the 8Gb/s range. Then you tried to test between the two devices in the different subnets and got only in the 13Mb/s range.
Have you tried placing the two devices on the same subnet and iperf'd between them that way to rule-out an issue between the two devices?
Of course we did.
The two machines are virtual ones, so when they are on the same virtual host we get a throughput about 26Gbit/s. When the machines are on different hosts we get a throughput about 8-9Gbit/s with iperf.Here are some more infos
hostname 10.100.5.8 -> app0016 Linux app0016 3.10.0-327.36.3.el7.x86_64 # tuning für wan: net.ipv6.conf.all.disable_ipv6 = 1 net.core.wmem_max=16777216 net.core.rmem_max=16777216 net.ipv4.tcp_rmem= 10240 16777216 16777216 net.ipv4.tcp_wmem= 10240 16777216 16777216 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 net.ipv4.tcp_no_metrics_save = 1 net.core.netdev_max_backlog = 5000 host 10.100.4.2 -> adm0001 Linux adm0001 3.10.0-327.36.3.el7.x86_64 tuning für wan net.ipv6.conf.all.disable_ipv6 = 1 net.core.wmem_max=16777216 net.core.rmem_max=16777216 net.ipv4.tcp_rmem= 10240 16777216 16777216 net.ipv4.tcp_wmem= 10240 16777216 16777216 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 net.ipv4.tcp_no_metrics_save = 1 net.core.netdev_max_backlog = 5000
iperf server 10.100.4.2 client 10.100.5.8 [root@adm0001 ~]# iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 16.0 MByte (default) ------------------------------------------------------------ [ 4] local 10.100.4.2 port 5001 connected with 10.100.5.8 port 46600 [root@app0016 ~]# iperf -c 10.100.4.2 ------------------------------------------------------------ Client connecting to 10.100.4.2, TCP port 5001 TCP window size: 16.0 MByte (default) ------------------------------------------------------------ [ 3] local 10.100.5.8 port 46600 connected with 10.100.4.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 15.2 MBytes 12.7 Mbits/sec iperf server 10.100.4.2 client 10.100.5.8 - kleinere window size [root@app0016 ~]# iperf -c 10.100.4.2 -w 64KB ------------------------------------------------------------ Client connecting to 10.100.4.2, TCP port 5001 TCP window size: 128 KByte (WARNING: requested 64.0 KByte) ------------------------------------------------------------ [ 3] local 10.100.5.8 port 46606 connected with 10.100.4.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-15.0 sec 215 KBytes 117 Kbits/sec iperf client 10.100.5.8 -> server 10.100.4.1 (pfsense) [root@app0016 ~]# iperf -c 10.100.4.1 ------------------------------------------------------------ Client connecting to 10.100.4.1, TCP port 5001 TCP window size: 16.0 MByte (default) ------------------------------------------------------------ [ 3] local 10.100.5.8 port 32782 connected with 10.100.4.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 9.97 GBytes 8.57 Gbits/sec
-
disable firewall true
disable firewall scrubnetworking
Disable hardware checksum offload false
Enable device polling false
Disable hardware TCP segmentation offload false
Disable hardware large receive offload falseHave you tried this:
disable firewall false
disable firewall scrub falsenetworking
Disable hardware checksum offload true
Enable device polling false
Disable hardware TCP segmentation offload true
Disable hardware large receive offload trueWith performance problems on certain NICs disabling the offloads is probably what you want, at least when you're looking at pitiful performance like 12.7 Mbits/sec through the firewall. Something is certainly off there. That almost looks like the problem with checksumming going through XenServer.
It looks like you are "hairpinning" traffic between those 2 subnets on bxe2_vlan1017 and bxe2_vlan1501 so you do realize you will never see the same throughput you see when that is not the case right? Obviously should be seeing better than 12Mbit/s though. Out of curiosity does it perform the same between a VLAN on bxe2 and a VLAN on bxe0? bxe2 and bgeX? bgeX & bgeY?
-
We will check your suggestions.
As far we have some results you will here from me.We activated the TSO and LRO to get a better performance and that work actually fine. I am not sure if this causes the side effects.
But i keep you informed after trying. -
We activated the TSO and LRO to get a better performance and that work actually fine.
Yeah something there is obviously not fine.
-
So… it seems to be the LRO that is decreasing the throughput.
I'll keep you informed.
*edit:
i think that and some performancetuning helped.