Change in performance from 1.2.x, 2.0 BETA, and 2.0 RC



  • I've been using pfSense for a couple of years on the same hardware, and have had some download performance issues with 2.0 BETA and now the RC on single flows over Comcast Cable.  On 1.2.3, I would consistently get ~50Mbit down and ~10Mbit up.  I moved recently and switched to 2.0 BETA, got poor perf and then changed setting in sysctl.conf and got ~20Mbit down and ~4Mbit up, which isn't as good, but was acceptable (I had been on 1.5 Mbit DSL for 6 months).  I decided to upgrade to the RC and now I get ~4Mbit down/~5Mbit up.  Comcast allows bursting and then throttles down to whatever speed tier you are paying for ( I am paying for 22/5).  If I connect my Win7 machine to the modem, I get 60+Mbit down / 5+ up.  When multiple connections are active through pfSense, the aggregate bandwidth will reach the cap of 22 Mbit down.  I'd like to get the same performance when using pfSense as I get from Windows  ;)

    I've tried quite a few things on the RC, some of which reduced perf, and none of which helped:
    Hard set Speed/duplex (no collisions, no errors in auto or manual)
    Disable offload
    Disable PowerD
    Disable net.inet.ip.random_id
    Enable net.inet.ip.fastforwarding
    Disable net.inet.tcp.tso
    Disable net.inet.tcp.tso
    Enable net.inet.tcp.delayed_ack (this helped a bit)
    Change the ethernet MTU to 1490 on the Windows machine and the LAN interface on pfSense, since the WAN had an MTU of 1490

    I performed packet captures on both pfSense(WAN) and the Win7 box.  The pfSense box truncated the capture at ~10MB, and corrupted the contents.  If I only captured the first 44bytes, it was able to capture without corruption, but the capture was less useful.  The capture on the Windows machine showed DUP ACKs and retransmissions and the experts showed:

    
    TCP:  Previous segment lost (common at capture start)	133
    TCP:  Fast retransmission (suspected)			103
    TCP:  Out-Of-Order segment				4
    TCP:  Window is full					5
    TCP:  Zero window					19
    TCP:  Retransmission (suspected)			59
    TCP:  Duplicate ACK ()					~500
    TCP:  Zero window probe					14
    TCP:  Zero window probe ACK				14
    
    

    I also changed net.inet.ip.portrange.first to 49152, since this is the IANA specified starting ephemeral port.

    Hardware:
    pfSense

    Atom 230 1.6 GHz
    1GB RAM
    3xJetway <realtek 8169="" 8169s="" 8169sb(l)="" 8110s="" 8110sb(l)="" gigabit="" ethernet="">-LAN
    1 x  <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">-WAN (using MAC cloning)
    1 x</realtek></realtek> 
    

    Modem DOCSIS 3 Surfboard SB6120

    sysctl.conf from BETA (tested this with RC, worse performance):

    # set to at least 16MB
    kern.ipc.maxsockbuf=16777216
    # set autotuning maximum to at least 16MB too
    net.inet.tcp.sendbuf_max=16777216 
    net.inet.tcp.recvbuf_max=16777216
    # enable send/recv autotuning
    net.inet.tcp.sendbuf_auto=1
    net.inet.tcp.recvbuf_auto=1
    # increase autotuning step size 
    net.inet.tcp.sendbuf_inc=16384 
    net.inet.tcp.recvbuf_inc=524288 
    # turn off inflight limitting
    net.inet.tcp.inflight.enable=0
    # set this on test/measurement hosts
    net.inet.tcp.hostcache.expire=1
    

    It looks like it could be a driver issue, but I'm really at a loss.  Any ideas?



  • Forgot one important thing– when I downloaded the capture, it downloaded at ~4Mbit.  Probably not a coincidence.



  • Most of those settings have no relevance to performance through the firewall, they only apply to traffic initiated by that system. Does seem like it's probably NIC driver-related, which network cards are you using? Most drivers didn't change at all between beta and RC.



  • Realtek.

    I am using one of the Jetway AD3RTLANG 3 x Gigabit LAN Port with <realtek 8169="" 8169s="" 8169sb(l)="" 8110s="" 8110sb(l)="" gigabit="" ethernet="">driver for the LAN.  It looks like the chipset is 8110S.  I'm using the onboard port from the Jetway NC92-N230 mini-ITX (RTL8111C) which loads the <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">driver for the WAN.</realtek></realtek>



  • Used 2.0 RC LiveCD on another box which has Intel NICs - no repro, so it is almost certainly a driver issue.



  • The Realtek drivers are stock FreeBSD's. Those 3 port Jetway cards seem to be hit and miss on every OS from what I've seen and heard from several others, not one of the better Realtek implementations.



  • Changing to the Intel NIC on the original box resolved the performance issues as well.  Anyone want to buy a 3-port realtek gigabit NIC?



  • Oh, and I forgot to mention, NIC setup was invoked automatically on the command line after boot wiht the new NICs and the configuration was applied to the new interfaces without any issues… nice work, guys!


Locked