[solved] significant problem with throughput through our pfsense

um3n

Dear all,

Currently we have a significant problem with throughput through our pfsense.
Throughput is very small if we send data from one local segment to another local segment and also,
if we send data over the WAN-Connection, but we have full bandwidth available if we send from any host to one of the pfsense interfaces.
Throughput through the pfsense firewall is 12.7 Mbits/s, compared to the expected bandwidth of approx.. 9 Gbit/s, which is achieved when testing to the pfsense interfaces.
We use the pfsense as the central router in our network. All hosts in our network are connected with 10GBE Nic's.
The pfsense also have a 10GBE WAN P2P connection to a second location. The latency of this connection is 13ms.
In an attempt to reach full speed over the 10GBE WAN connection we have made some adjustments in the TCP windows size on the hosts and on the pfsense (incremented to 16B).

Also, to help find the error we disabled the entire firewall functionality on the pfsense but without any result.

Please can you investigate and help resolve why throughput through the pfsense is significantly reduced.
Please see below for information about the configurations.

With kind regards,

Teste scenarios with iperf 2

host a 10.100.4.2/24
host b 10.100.5.8/24

pfsense

10.100.4.1/24
10.100.5.1/24

10.100.5.8 -> 10.100.4.1 = 8.60 Gbits/sec
10.100.4.2 -> 10.100.5.1 = 8.88 Gbits/sec
10.100.5.8 -> 10.100.4.2 = 12.7 Mbits/sec
10.100.4.2 -> 10.100.5.8 = 12.7 Mbits/sec

PFSense Version:

2.3.2-RELEASE-p1 (amd64)

10.100.5.1 -> vlan on bxe2
10.100.4.1 -> vlan on bxe2

tunables

net.inet.tcp.recvspace 16777216
net.inet.tcp.sendspace 16777216
net.raw.sendspace 16777216
net.raw.recvspace 16777216
kern.ipc.maxsockbuf 33554432
kern.ipc.nmbclusters 1000000

firewall & nat

disable firewall true
disable firewall scrub

networking

Disable hardware checksum offload false
Enable device polling false
Disable hardware TCP segmentation offload false
Disable hardware large receive offload false

hardware

Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
32 CPUs: 2 package(s) x 8 core(s) x 2 SMT threads
130GB RAM

HP FlexFabric 10Gb 2-port 533FLR-T Adapter

Location Embedded

Firmware 7.14.79

2c:44:fd:91:0d:50
2c:44:fd:91:0d:54

HP Ethernet 10Gb 2-port 530T Adapter

Location Slot 3

Firmware 7.14.79

14:02:ecbd:28
14:02:ecbd:2c

HP Ethernet 1Gb 4-port 331i Adapter - NIC

Location Embedded

Firmware 17.4.41

1c:98:ec:15:69:2c
1c:98:ec:15:69:2d
1c:98:ec:15:69:2e
1c:98:ec:15:69:2f

ifconfig

bxe0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=507bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 2c:44:fd:91:0d:50

inet6 fe80::2e44:fdff:fe91:d50%bxe0 prefixlen 64 scopeid 0x1

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

bxe1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=507bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 2c:44:fd:91:0d:54

inet6 fe80::2e44:fdff:fe91:d54%bxe1 prefixlen 64 scopeid 0x2

inet 192.168.11.3 netmask 0xffffff00 broadcast 192.168.11.255

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

bge0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500

options=c019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate>ether 1c:98:ec:15:69:2c

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect

bge1: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500

options=c019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate>ether 1c:98:ec:15:69:2d

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect

bge2: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500

options=c019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate>ether 1c:98:ec:15:69:2e

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect

bge3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500

options=c019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate>ether 1c:98:ec:15:69:2f

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect

bxe2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=507bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 14:02:ec:cd:bd:28

inet6 fe80::1602:ecff:fecd:bd28%bxe2 prefixlen 64 scopeid 0x7

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

bxe3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500

options=507bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether 14:02:ec:cd:bd:2c

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (none)

pflog0: flags=100 <promisc>metric 0 mtu 33160

pfsync0: flags=0<> metric 0 mtu 1500

syncpeer: 224.0.0.240 maxupd: 128 defer: on

syncok: 1

enc0: flags=0<> metric 0 mtu 1536

nd6 options=21 <performnud,auto_linklocal>lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384

options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000

inet6 ::1 prefixlen 128

inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc

nd6 options=21 <performnud,auto_linklocal>bxe2_vlan1016: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28

inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1016 prefixlen 64 scopeid 0xd

inet 10.100.1.1 netmask 0xfffffff0 broadcast 10.100.1.15

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 1016 vlanpcp: 0 parent interface: bxe2

bxe2_vlan1017: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28

inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1017 prefixlen 64 scopeid 0xe

inet 10.100.4.1 netmask 0xffffff00 broadcast 10.100.4.255

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 1017 vlanpcp: 0 parent interface: bxe2

bxe2_vlan1018: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28

inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1018 prefixlen 64 scopeid 0xf

inet 10.100.9.1 netmask 0xffffff00 broadcast 10.100.9.255

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 1018 vlanpcp: 0 parent interface: bxe2

bxe2_vlan1019: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28

inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1019 prefixlen 64 scopeid 0x10

inet 10.100.10.1 netmask 0xffffff00 broadcast 10.100.10.255

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 1019 vlanpcp: 0 parent interface: bxe2

bxe2_vlan1501: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28

inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1501 prefixlen 64 scopeid 0x11

inet 10.100.5.1 netmask 0xffffff00 broadcast 10.100.5.255

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 1501 vlanpcp: 0 parent interface: bxe2

bxe2_vlan1502: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28

inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan1502 prefixlen 64 scopeid 0x12

inet 10.100.6.1 netmask 0xffffff00 broadcast 10.100.6.255

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 1502 vlanpcp: 0 parent interface: bxe2

bxe2_vlan2518: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28

inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan2518 prefixlen 64 scopeid 0x13

inet 10.100.2.1 netmask 0xfffffff0 broadcast 10.100.2.15

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 2518 vlanpcp: 0 parent interface: bxe2

bxe2_vlan2519: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28

inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan2519 prefixlen 64 scopeid 0x14

inet 10.100.3.1 netmask 0xffffff00 broadcast 10.100.3.255

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 2519 vlanpcp: 0 parent interface: bxe2

bxe2_vlan3015: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 14:02:ec:cd:bd:28

inet6 fe80::1602:ecff:fecd:bd28%bxe2_vlan3015 prefixlen 64 scopeid 0x15

inet 10.100.7.1 netmask 0xffffff00 broadcast 10.100.7.255

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 3015 vlanpcp: 0 parent interface: bxe2

bxe0_vlan3016: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 2c:44:fd:91:0d:50

inet6 fe80::2e44:fdff:fe91:d50%bxe0_vlan3016 prefixlen 64 scopeid 0x16

inet 10.100.8.1 netmask 0xffffff00 broadcast 10.100.8.255

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 3016 vlanpcp: 0 parent interface: bxe0

bxe0_vlan3017: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500

options=303 <rxcsum,txcsum,tso4,tso6>ether 2c:44:fd:91:0d:50

inet6 fe80::2e44:fdff:fe91:d50%bxe0_vlan3017 prefixlen 64 scopeid 0x17

inet 10.100.11.1 netmask 0xffffff00 broadcast 10.100.11.255

nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (10Gbase-T <full-duplex>)

status: active

vlan: 3017 vlanpcp: 0 parent interface: bxe0</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,auto_linklocal></promisc></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></broadcast,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate></broadcast,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate></broadcast,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate></broadcast,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwtso,linkstate></broadcast,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast>

CC

Is this a new setup or working setup that has failed?

Harvy66

It looks like you tested from a Device on Subnet A to PFSense's IP on Subnet B and vice-versa, and got in the 8Gb/s range. Then you tried to test between the two devices in the different subnets and got only in the 13Mb/s range.

Have you tried placing the two devices on the same subnet and iperf'd between them that way to rule-out an issue between the two devices?

um3n

Hey Guys,

thank you for responding, we appreciate that you spend your time.
Of course i will answer your questions to get some more suggestions from you.

@CC:

Is this a new setup or working setup that has failed?

Its a new setup which we are testing here.

@Harvy66:

It looks like you tested from a Device on Subnet A to PFSense's IP on Subnet B and vice-versa, and got in the 8Gb/s range. Then you tried to test between the two devices in the different subnets and got only in the 13Mb/s range.

Have you tried placing the two devices on the same subnet and iperf'd between them that way to rule-out an issue between the two devices?

Of course we did.
The two machines are virtual ones, so when they are on the same virtual host we get a throughput about 26Gbit/s. When the machines are on different hosts we get a throughput about 8-9Gbit/s with iperf.

Here are some more infos

hostname
10.100.5.8 -> app0016
Linux app0016 3.10.0-327.36.3.el7.x86_64 #

tuning für wan:

net.ipv6.conf.all.disable_ipv6 = 1
net.core.wmem_max=16777216
net.core.rmem_max=16777216
net.ipv4.tcp_rmem= 10240  16777216  16777216
net.ipv4.tcp_wmem= 10240  16777216  16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 5000

host
10.100.4.2 -> adm0001
Linux adm0001 3.10.0-327.36.3.el7.x86_64

tuning für wan

net.ipv6.conf.all.disable_ipv6 = 1
net.core.wmem_max=16777216
net.core.rmem_max=16777216
net.ipv4.tcp_rmem= 10240  16777216  16777216
net.ipv4.tcp_wmem= 10240  16777216  16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_no_metrics_save = 1
net.core.netdev_max_backlog = 5000

iperf server 10.100.4.2  client 10.100.5.8

[root@adm0001 ~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 16.0 MByte (default)
------------------------------------------------------------
[  4] local 10.100.4.2 port 5001 connected with 10.100.5.8 port 46600

[root@app0016 ~]# iperf -c 10.100.4.2
------------------------------------------------------------
Client connecting to 10.100.4.2, TCP port 5001
TCP window size: 16.0 MByte (default)
------------------------------------------------------------
[  3] local 10.100.5.8 port 46600 connected with 10.100.4.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  15.2 MBytes  12.7 Mbits/sec

iperf server 10.100.4.2  client 10.100.5.8 - kleinere window size

[root@app0016 ~]# iperf -c 10.100.4.2 -w 64KB
------------------------------------------------------------
Client connecting to 10.100.4.2, TCP port 5001
TCP window size:  128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  3] local 10.100.5.8 port 46606 connected with 10.100.4.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-15.0 sec   215 KBytes   117 Kbits/sec

iperf client 10.100.5.8 -> server 10.100.4.1 (pfsense)

[root@app0016 ~]# iperf -c 10.100.4.1
------------------------------------------------------------
Client connecting to 10.100.4.1, TCP port 5001
TCP window size: 16.0 MByte (default)
------------------------------------------------------------
[  3] local 10.100.5.8 port 32782 connected with 10.100.4.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  9.97 GBytes  8.57 Gbits/sec

Derelict

disable firewall true
disable firewall scrub

networking

Disable hardware checksum offload false
Enable device polling false
Disable hardware TCP segmentation offload false
Disable hardware large receive offload false

Have you tried this:

disable firewall false
disable firewall scrub false

networking

Disable hardware checksum offload true
Enable device polling false
Disable hardware TCP segmentation offload true
Disable hardware large receive offload true

With performance problems on certain NICs disabling the offloads is probably what you want, at least when you're looking at pitiful performance like 12.7 Mbits/sec through the firewall. Something is certainly off there. That almost looks like the problem with checksumming going through XenServer.

It looks like you are "hairpinning" traffic between those 2 subnets on bxe2_vlan1017 and bxe2_vlan1501 so you do realize you will never see the same throughput you see when that is not the case right? Obviously should be seeing better than 12Mbit/s though. Out of curiosity does it perform the same between a VLAN on bxe2 and a VLAN on bxe0? bxe2 and bgeX? bgeX & bgeY?

um3n

We will check your suggestions.
As far we have some results you will here from me.

We activated the TSO and LRO to get a better performance and that work actually fine. I am not sure if this causes the side effects.
But i keep you informed after trying.

Derelict

We activated the TSO and LRO to get a better performance and that work actually fine.

Yeah something there is obviously not fine.

um3n

So… it seems to be the LRO that is decreasing the throughput.
I'll keep you informed.
*edit:
i think that and some performancetuning helped.