Metro Ethernet p2p link speed issues, TCP needs tuning?

  • I have a 100mbps metro ethernet eWAN link from the UK to the US. I am using pfsense, with QinQ enabled, and everything works fine…
    However, with a single connection using iperf i can only get a maximum of 14mbps, i have to use many simultaneous connections to saturate the line.

    I have changed the send and receive buffer to 256kb, and disabled tcp inflight mode form the system tunables with no change.
    I am using a standard mtu of 1500.

    Would the latency of around 130ms affect the speed with one single connection?
    Do i need to tune the tcp stack?

    thanks in advance

  • Latency that high will definitely limit TCP throughput. But you have to change your windowing and related settings on the source and destination hosts, not the firewall, the firewall doesn't handle that except for traffic it initiates itself.

  • I understand this, thats why im testing on the router/firewalls on each end to try to get 100mbps initially.

    Im running iperf from each firewall, these are routing. this is what i see.

    Client connecting to, TCP port 5001
    TCP window size:   257 KByte (default)
    [  3] local port 29753 connected with port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.1 sec  17.3 MBytes  14.3 Mbits/sec
    [  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

  • A link from the UK to the US? Are you sure its truly a point-to-point link or is it getting aggregated in with other traffic somewhere? I know the common links between countries can be in the multi-gigabit range but they can also be fairly saturated at times. I know this may seem like a "duh" question, but sometimes it's the little things that get overlooked (and usually aren't something you did).

  • Its a logical layer 2 "dedicated" circuit, so it runs across an mpls network, we have to tag our traffic with a vlan tag they specify, and do QinQ.
    It seems like it could be something todo with them, as i can only saturate it with many simultaneous connections, using iperf and not just using a single connection.
    One way from UK to US i get 114mbps according to iperf, but from US to UK, i only get 14mbps, unless i use many connections.
    Both sides have the same hardware spec.
    On the uk side it plugs into a media converter, and the US side plugs into a switch with a SM fiber sfp module.

    It doesent make sense not to the same speeds both ways?

  • An MPLS network would seem to imply it rides over their own infrastructure. Presumably they have their own connectivity across the ocean but that's likely only the case if they have a business presence in both countries to terminate that transport. It is weird though that you can get full speed in one direction but not another. Do you know if it's the same company (or at least a subsidiary) on both ends? Since you can saturate the link with simultaneous connections it seems like some kind of rate limiting is turned on somewhere.

    On the UK side, the output of the media converter goes to a NIC on the firewall? Why not try the same thing on the US side? Seems like it would be better to have it go straight to the firewall than to a switch first unless you wanted to hang other devices off it in parallel to the firewall.

  • hmm, the thing that i dont understand is that even though i get 100mbps one way with iperf, point to point on the pfsense boxes… if i repeat the test with a test machine at either end, routing through the boxes, i only get 300kb/sec one way, and 64kb/sec the other way.
    doesent make sense to me.

  • Tests on the other boxes could be a TCP tuning issue. What OS is running on the machines behind pfSense? Assuming it is Windows, if you are using Vista/7 you shouldn't need to do anything as it generally does a good job at dynamically adjusting it's settings as needed. XP and older though don't have the dynamically adjusted settings and can benefit from this: and then this: I know that tool won't work for non-windows OSes but the tweak test is Java based and should run on any OS, of course making the adjustments it recommends would require a skillset I don't have :P