Need help tuning throughput for 100Mbps NICs on 100Mbps Link



  • I am using the 2.1 release (nanobsd) on a CF card, mounted in a Soekris 5501-70 board. The 4 VIA VT6105M 10/100 Mbit Auto MDIX Ethernet ports are rated for 100Mbps max.

    I don't expect to get 100Mbps throughput, as there must be overhead and inefficiencies. But it would be nice to get 80Mbps from a 100Mbps NIC.

    The WAN interface gets its IP by DHCP from the TDS router. There are no VPN's configured, and no longer any www or mail servers.

    All was well…UNTIL... I switched ISP's. The new provider had run fiber to the curb. So, I opted for the 100/50 plan.

    The problem is that the Soekris board is not getting anywhere near that kind of throughput. And, oddly, upload speed seems to average almost 50% faster than download speed. Posting results from speedtest, or speakeasy, or testmy.net seems pointless, as they are inconsistent, and all over the board.

    I have read many accounts of folks tuning their pfSense install for 1Gig and 10Gig throughput. But, I have not been able to find any reference that discusses the finer points of tuning for 100Mbps or less.

    The relevant changes I have made to system tunables are:

    net.inet.tcp.recvspace=405440
    net.inet.tcp.sendspace=405440
    net.inet.ip.fastforwarding=1
    net.inet.ip.intr_queue_maxlen=1000
    net.inet.tcp.mssdflt=1448
    kern.ipc.nmbclusters=65536
    net.link.ifqmaxlen=2048
    net.inet.tcp.sendbuf_inc=16384
    net.inet.tcp.recvbuf_inc=32768

    Your experiences/suggestions would be helpful.



  • The 5501-70 is basically the same as most Alix boards.  It's going to top out at about 85Mbit/s TOTAL (both directions combined) when doing just FW+NAT.  If you've got any intensive packages installed, use traffic shaping, VPN, etc., then it will drop…  A lot.

    You haven't mentioned what kind of speeds you are seeing, so it's tough to say more.



  • So… I've been testing my speed every day for the last monthl.

    When i plug directly into the TDS router from the patch panel, I am getting an average speed of 75Mbps down and 40Mbps up.

    When I plug the pfsense box into the TDS router, I am getting 15Mbps down and 30Mbps up.

    Details that may/may not be relevant:

    1. The ISP router hands out a WAN address via DHCP. That address is in the 192.168.0.x space. No worries, I have the pfSense WAN interface configured to use DHCP. HOWEVER...when I un-check the "block private networks" and "block bogon" boxes. I can ping outside, but lose 60% or more of the packets.

    2. If I un-plug the WAN interface of the pfSense box FROM the ISP router, and then plug it back in to make sure I've got a snug connection, it takes a reboot, and 5 minutes of "settling in" before I get connectivity back.

    I'd really like to get a bit more speed out of my pfsense box. It runs on a Soekris 5501-70, so the theoretical maximum is 100Mbps. I don't expect to saturate the pipe, but I would at least like to get 80%.

    Any suggestions?




  • @Jones:

    1. The ISP router hands out a WAN address via DHCP. That address is in the 192.168.0.x space. No worries, I have the pfSense WAN interface configured to use DHCP. HOWEVER…when I un-check the "block private networks" and "block bogon" boxes. I can ping outside, but lose 60% or more of the packets.

    2. If I un-plug the WAN interface of the pfSense box FROM the ISP router, and then plug it back in to make sure I've got a snug connection, it takes a reboot, and 5 minutes of "settling in" before I get connectivity back.

    Well that seems odd, doesn't it? Have you tried doing a packet dump on the WAN while or right after plugging it in? It would also be interesting to see where the packet loss is occurring. For example, try a traceroute to 8.8.8.8 or some internet server and then ping each hop separately on the way. See which hop is dropping the packets.

    You may want to check your cables and try swapping your WAN and LAN NICs with some others on the board to eliminate a faulty NIC.

    Try a different power supply. Soekris boards are notorious for strange behaviour when the power supply isn't operating within spec.

    Oh yes, and the CF card too. My 5501 acts funny with certain CF cards and fine with others. Not that it won't boot or run, it just acts really strange in all sorts of ways, kind of like you are describing. It could even be that it was fine on your old provider, but only started showing weirdness when you switched.



  • @clarknova:

    Well that seems odd, doesn't it? Have you tried doing a packet dump on the WAN while or right after plugging it in? It would also be interesting to see where the packet loss is occurring. For example, try a traceroute to 8.8.8.8 or some internet server and then ping each hop separately on the way. See which hop is dropping the packets.

    You may want to check your cables and try swapping your WAN and LAN NICs with some others on the board to eliminate a faulty NIC.

    Try a different power supply. Soekris boards are notorious for strange behaviour when the power supply isn't operating within spec.

    Oh yes, and the CF card too. My 5501 acts funny with certain CF cards and fine with others. Not that it won't boot or run, it just acts really strange in all sorts of ways, kind of like you are describing. It could even be that it was fine on your old provider, but only started showing weirdness when you switched.

    How do you to a packet dump?

    The dropped packets have been occurring between my desktop (inside the firewall) and the LAN interface of the router (outside the firewall).

    I'll have a look at the power supply.

    And the CF card… HA! When I first built this box in 2011, I paid $30 for an industrial CF card. Couldn't get that silly thing to work at all. So I went down to Walmart and got a $12 CF card that goes in a camera. Worked like a charm the first time.



  • @Jones:

    How do you to a packet dump?

    Go to Diagnostics: Packet Capture. Watch on the LAN interface and then the WAN as you ping each host upstream. You should see all the ICMP echo requests go out and the responses come back. You can see packet loss this way and figure out where it's happening. You could also watch for strange traffic that could be causing problems.

    And the CF card… HA! When I first built this box in 2011, I paid $30 for an industrial CF card. Couldn't get that silly thing to work at all. So I went down to Walmart and got a $12 CF card that goes in a camera. Worked like a charm the first time.

    Yeah, that doesn't surprise me at all. They can be finicky.