Asymmetrical throughput (measured by Speedtest and similar) on symmetric link
-
pfSense users:
I have a roughly symmetrical GigE FTTH connection through AT&T. Measured with speedtest.net and similar sites, I get near "line rate" down (930+Mb/s) and over 600Mb/s up if I remove my pfSense router from behind AT&T's required router (I have voice with them and haven't figured out how to put it behind my router).
My dual-core atom pfSense router is only good for about 600Mbit. That I understand well.
My speeds for download, behind my pfSense router, drop to just over 600Mb/s. But my upload drops by an even greater degree, 50%, down to 300MBb/s.
My setup includes NAT, OpenVPN (although nobody using it during my testing), but no traffic shaping.
I also have a DMZ using a VLAN sharing the same physical Ethernet port as LAN.
Is this "normal" to see slower "outbound" throughput, not from DSL or other asymmetric services, but from the interaction with the router? I'm wondering if this is just a byproduct of stacking latencies along with the way Speedtest works.
I tried putting a laptop on the "outside" between my router and AT&T's to measure through the router with iperf, but there's something weird there. My router is picking up a passthrough IP, not a private IP, on my WAN link. Using iperf results in 10's to 100's of Kbit/s. I think it has to go from pfsense->AT&T->host and the AT&T router (while it's full line rate capable) isn't liking that.
Thoughts? Ways to measure? I could always tear down my WAN settings, try it with a more simplified setup to eliminate the passthrough strangeness above, but are there other thoughts or has this been talked through before?
-
One more tidbit: Comparing two SpeedTest results: Ping time without pfSense: 3ms. With pfSense, approx 30ms.
Hardware is Intel D2500CCE http://www.intel.com/content/www/us/en/motherboards/desktop-motherboards/desktop-board-d2500cc.html
-
Can you get a snapshot of system activity during your speedtests so we can see CPU usage of the different parts of the system.
That much higher ping is a bit troublesome. What does your traceroutes look like and what's your ping to PFSense on the LAN?
-
Let me know what I should trace while running it.
Here are the pings (I am sitting at AUS airport, thank goodness for pfSense, OpenVPN and Viscosity!)
$ ping 172.17.0.1
PING 172.17.0.1 (172.17.0.1): 56 data bytes
64 bytes from 172.17.0.1: icmp_seq=0 ttl=64 time=0.305 ms
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=0.207 ms
64 bytes from 172.17.0.1: icmp_seq=2 ttl=64 time=0.213 ms -
Speedtest with bare AT&T (Motorola NVG589)
http://www.speedtest.net/my-result/4086297037
Speedtest with pfSense router behind AT&T router
http://www.speedtest.net/my-result/4084692443
-
What should I run on the router to trace activity? I'd like to figure this out :)
-
Hold it!
I've tracked it down to my iMac… all of my "slow ping" tests were done from my desktop, and I would carry my laptop to the other room to do the tests directly connected to AT&T.
Turns out my laptop gets just over 500mbit, symmetrical, with 3ms pings, from the same switch in my office, using a Thunderbolt Ethernet adapter. Using that SAME adapter on my iMac gives me the SAME speeds (and slow ping) as the built-in Ethernet.
I don't recall setting any strange configs on my iMac (like sysctls) but I am going to check that.
This doesn't appear to be a pfSense issue. Moving on!
-
WOW….
It's Firefox.
I found some sysctls that I had set a long time ago for networking (probably on 1 or 2 major OS releases back), removed those and rebooted. Tried Firefox and Chrome, and Chrome is low latency and symmetric, Firefox is high latency and asymmetric.
See, I use Chrome almost exclusively on my work laptop, and Firefox almost exclusively on home desktop. Talk about variables stacked on variables!
-
No! Its got to be pfsense…. (kidding)
Well - I'm glad thats sorted out...
-
Next time you have an issue with performance with anything, make sure you use don't go about changing variables, like computers and browsers.
-
Harvy66:
Thanks for the pointers. Believe it or not, I'm a successful systems engineer by trade. So I get it.
I had no reason to suspect a browser would be the reason for bad latency.
I've found a more interesting problem: I can put pfsense in a virtual machine (on ESXi) and move that between 4 servers and it gets different Up and Down speeds (and different ratios between those) based on the server. Yes, the servers are all different performance, but you'd expect the 3.2GHz Westmere to be faster than the 2.4GHz Westmere, but that's not the case. And the 2.4 does 900/700 while the 3.2 does 590/600.
Lots of variables there. But its funny how a VM running in a Xeon (in the slower example above) is not much faster than my Atom running bare metal.
-
have you ever played with ipertf?
-
Unless you have a VM host with some good pass-through and good hardware and drivers to back that up and a guest to take advantage, baremetal can be a lot faster for IO related stuff. For now, you pretty much need to plan out your VM system if you want decent IO performance. Hand select your hardware, host and guest.
We all have our moments, I know I have them. I wasn't trying to being belittlingly mean, but a lot of people are repeat offenders of changing variables when testing issues. Next time they have an issue, my hope is they think "last time I had a problem to solve, some jerk pointed out my simple mistake". People have made me this way! I am a monster :'(
I also would be interested in iperf performance. Don't just test to and from the firewall, but also through the firewall.
-
The challenge with trying iperf is that I have to reconfigure some things to test it. I am currently router-behind-router with a twist.
The Motorola NVG589 in front of my pfSense system has a hybrid NAT as well as public IP, because I am paying for multiple IP addresses. So I have 5 IPs on the public subnet and then a private 192.x.x.x subnet. PFSense sits on one of the public IPs and I can use VIPs for the additional and NAT them in to a given host.
The issue is putting iperf out on one of those 192.x.x.x IP addresses, between pfSense and the AT&T router (actually sitting next to pfSense, but "outside" my firewall).
iperf can generate only 10's to 100Kbit/s in that situation, from inside my LAN to that immediate WAN before the AT&T router. I can get better iperf performance to a system I have at a colocation than a system sitting on my Motorola router just outside pfSense!
So to test my pfSense router I'd have to reconfigure it entirely, do the test, and then put it back so my family can get their internet back :)