Change in performance from 1.2.x, 2.0 BETA, and 2.0 RC

zackb

I've been using pfSense for a couple of years on the same hardware, and have had some download performance issues with 2.0 BETA and now the RC on single flows over Comcast Cable. On 1.2.3, I would consistently get ~50Mbit down and ~10Mbit up. I moved recently and switched to 2.0 BETA, got poor perf and then changed setting in sysctl.conf and got ~20Mbit down and ~4Mbit up, which isn't as good, but was acceptable (I had been on 1.5 Mbit DSL for 6 months). I decided to upgrade to the RC and now I get ~4Mbit down/~5Mbit up. Comcast allows bursting and then throttles down to whatever speed tier you are paying for ( I am paying for 22/5). If I connect my Win7 machine to the modem, I get 60+Mbit down / 5+ up. When multiple connections are active through pfSense, the aggregate bandwidth will reach the cap of 22 Mbit down. I'd like to get the same performance when using pfSense as I get from Windows ;)

I've tried quite a few things on the RC, some of which reduced perf, and none of which helped:
Hard set Speed/duplex (no collisions, no errors in auto or manual)
Disable offload
Disable PowerD
Disable net.inet.ip.random_id
Enable net.inet.ip.fastforwarding
Disable net.inet.tcp.tso
Disable net.inet.tcp.tso
Enable net.inet.tcp.delayed_ack (this helped a bit)
Change the ethernet MTU to 1490 on the Windows machine and the LAN interface on pfSense, since the WAN had an MTU of 1490

I performed packet captures on both pfSense(WAN) and the Win7 box. The pfSense box truncated the capture at ~10MB, and corrupted the contents. If I only captured the first 44bytes, it was able to capture without corruption, but the capture was less useful. The capture on the Windows machine showed DUP ACKs and retransmissions and the experts showed:


TCP:  Previous segment lost (common at capture start)	133
TCP:  Fast retransmission (suspected)			103
TCP:  Out-Of-Order segment				4
TCP:  Window is full					5
TCP:  Zero window					19
TCP:  Retransmission (suspected)			59
TCP:  Duplicate ACK ()					~500
TCP:  Zero window probe					14
TCP:  Zero window probe ACK				14

I also changed net.inet.ip.portrange.first to 49152, since this is the IANA specified starting ephemeral port.

Hardware:
pfSense

Atom 230 1.6 GHz
1GB RAM
3xJetway <realtek 8169="" 8169s="" 8169sb(l)="" 8110s="" 8110sb(l)="" gigabit="" ethernet="">-LAN
1 x  <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">-WAN (using MAC cloning)
1 x</realtek></realtek>

Modem DOCSIS 3 Surfboard SB6120

sysctl.conf from BETA (tested this with RC, worse performance):

# set to at least 16MB
kern.ipc.maxsockbuf=16777216
# set autotuning maximum to at least 16MB too
net.inet.tcp.sendbuf_max=16777216 
net.inet.tcp.recvbuf_max=16777216
# enable send/recv autotuning
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
# increase autotuning step size 
net.inet.tcp.sendbuf_inc=16384 
net.inet.tcp.recvbuf_inc=524288 
# turn off inflight limitting
net.inet.tcp.inflight.enable=0
# set this on test/measurement hosts
net.inet.tcp.hostcache.expire=1

It looks like it could be a driver issue, but I'm really at a loss. Any ideas?

zackb

Forgot one important thing– when I downloaded the capture, it downloaded at ~4Mbit. Probably not a coincidence.

cmb

Most of those settings have no relevance to performance through the firewall, they only apply to traffic initiated by that system. Does seem like it's probably NIC driver-related, which network cards are you using? Most drivers didn't change at all between beta and RC.

zackb

Realtek.

I am using one of the Jetway AD3RTLANG 3 x Gigabit LAN Port with <realtek 8169="" 8169s="" 8169sb(l)="" 8110s="" 8110sb(l)="" gigabit="" ethernet="">driver for the LAN. It looks like the chipset is 8110S. I'm using the onboard port from the Jetway NC92-N230 mini-ITX (RTL8111C) which loads the <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">driver for the WAN.</realtek></realtek>

zackb

Used 2.0 RC LiveCD on another box which has Intel NICs - no repro, so it is almost certainly a driver issue.

cmb

The Realtek drivers are stock FreeBSD's. Those 3 port Jetway cards seem to be hit and miss on every OS from what I've seen and heard from several others, not one of the better Realtek implementations.

zackb

Changing to the Intel NIC on the original box resolved the performance issues as well. Anyone want to buy a 3-port realtek gigabit NIC?

zackb

Oh, and I forgot to mention, NIC setup was invoked automatically on the command line after boot wiht the new NICs and the configuration was applied to the new interfaces without any issues… nice work, guys!