Change in performance from 1.2.x, 2.0 BETA, and 2.0 RC
I've been using pfSense for a couple of years on the same hardware, and have had some download performance issues with 2.0 BETA and now the RC on single flows over Comcast Cable. On 1.2.3, I would consistently get ~50Mbit down and ~10Mbit up. I moved recently and switched to 2.0 BETA, got poor perf and then changed setting in sysctl.conf and got ~20Mbit down and ~4Mbit up, which isn't as good, but was acceptable (I had been on 1.5 Mbit DSL for 6 months). I decided to upgrade to the RC and now I get ~4Mbit down/~5Mbit up. Comcast allows bursting and then throttles down to whatever speed tier you are paying for ( I am paying for 22/5). If I connect my Win7 machine to the modem, I get 60+Mbit down / 5+ up. When multiple connections are active through pfSense, the aggregate bandwidth will reach the cap of 22 Mbit down. I'd like to get the same performance when using pfSense as I get from Windows ;)
I've tried quite a few things on the RC, some of which reduced perf, and none of which helped:
Hard set Speed/duplex (no collisions, no errors in auto or manual)
Enable net.inet.tcp.delayed_ack (this helped a bit)
Change the ethernet MTU to 1490 on the Windows machine and the LAN interface on pfSense, since the WAN had an MTU of 1490
I performed packet captures on both pfSense(WAN) and the Win7 box. The pfSense box truncated the capture at ~10MB, and corrupted the contents. If I only captured the first 44bytes, it was able to capture without corruption, but the capture was less useful. The capture on the Windows machine showed DUP ACKs and retransmissions and the experts showed:
TCP: Previous segment lost (common at capture start) 133 TCP: Fast retransmission (suspected) 103 TCP: Out-Of-Order segment 4 TCP: Window is full 5 TCP: Zero window 19 TCP: Retransmission (suspected) 59 TCP: Duplicate ACK () ~500 TCP: Zero window probe 14 TCP: Zero window probe ACK 14
I also changed net.inet.ip.portrange.first to 49152, since this is the IANA specified starting ephemeral port.
Atom 230 1.6 GHz 1GB RAM 3xJetway <realtek 8169="" 8169s="" 8169sb(l)="" 8110s="" 8110sb(l)="" gigabit="" ethernet="">-LAN 1 x <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">-WAN (using MAC cloning) 1 x</realtek></realtek>
Modem DOCSIS 3 Surfboard SB6120
sysctl.conf from BETA (tested this with RC, worse performance):
# set to at least 16MB kern.ipc.maxsockbuf=16777216 # set autotuning maximum to at least 16MB too net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 # enable send/recv autotuning net.inet.tcp.sendbuf_auto=1 net.inet.tcp.recvbuf_auto=1 # increase autotuning step size net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_inc=524288 # turn off inflight limitting net.inet.tcp.inflight.enable=0 # set this on test/measurement hosts net.inet.tcp.hostcache.expire=1
It looks like it could be a driver issue, but I'm really at a loss. Any ideas?
Forgot one important thing– when I downloaded the capture, it downloaded at ~4Mbit. Probably not a coincidence.
Most of those settings have no relevance to performance through the firewall, they only apply to traffic initiated by that system. Does seem like it's probably NIC driver-related, which network cards are you using? Most drivers didn't change at all between beta and RC.
I am using one of the Jetway AD3RTLANG 3 x Gigabit LAN Port with <realtek 8169="" 8169s="" 8169sb(l)="" 8110s="" 8110sb(l)="" gigabit="" ethernet="">driver for the LAN. It looks like the chipset is 8110S. I'm using the onboard port from the Jetway NC92-N230 mini-ITX (RTL8111C) which loads the <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">driver for the WAN.</realtek></realtek>
Used 2.0 RC LiveCD on another box which has Intel NICs - no repro, so it is almost certainly a driver issue.
The Realtek drivers are stock FreeBSD's. Those 3 port Jetway cards seem to be hit and miss on every OS from what I've seen and heard from several others, not one of the better Realtek implementations.
Changing to the Intel NIC on the original box resolved the performance issues as well. Anyone want to buy a 3-port realtek gigabit NIC?
Oh, and I forgot to mention, NIC setup was invoked automatically on the command line after boot wiht the new NICs and the configuration was applied to the new interfaces without any issues… nice work, guys!