ALIX - can't get more than ~10Mbit throughput OTW
-
Lots of problems with this setup regarding throughput. It used to work well, ie at the 20Mbit+ I had at the time through the ISP. Then I got a new (Cisco) modem from the ISP and changed to a new pf box, which has since died; I've gone back to the older ALIX 2D13.
Now, I should get 35Mbit, but I'm getting around 10Mbit. Per connection throughput is no more than 20-40kbps per connection. pfsense itself isn't maxing on state tables, CPU bandwidth (barely anything) or memory (about half, which remains pretty idle). Nothing on pf indicates a problem. This pf platform is known being able to do 80Mbit over the wire, synchronous. I've tried pfsense 1.2.3 and 2.0.1 with the same result.
If I connect via ethernet with another device (eg. laptop) to the cable modem, I'll get the 35/5Mbit I'm supposed to be getting. With pfsense in between, I'll sometimes only get 1Mbit, but usually it's at about 10Mbit down and 4-5Mbit up. Latency goes up to 30-60ms from the 10-15 I get directly connected.
Things I've tried:
- spoofing the MAC to different values - get different IP but marginal/negligible difference (gives me different upstream gateways and the like - pulled the MACs from 3 different old ethernet cards - 3COM and 2 different rtl8139s)
- changing MTU down to 1480 and manually setting it at 1500 (same result either way, with different MACs)
- new ethernet between me and (cisco) cable modem (no difference)
- a different, shorter ethernet cable, relocating pfsense to be directly next to modem (no difference)
- changing to different interface on ALIX for LAN as well as WAN (no difference)
- toggling checksum etc. offloading (no difference)
- manually setting 100MbitTX full duplex - no difference
- verifying all ALIX interfaces autoMDX on my switch to 100 full (they do).
Speedtest.net returns consistent throughput with each of the above changes - about 10Mbps, often less (seemingly arbitrarily).
I know this platform can perform better than it is, because there are benchmarks out there demonstrating it (and I've experienced it with this same device). So the problem has to be with the device in this specific arrangement, but I'm not seeing the problem.
The thing that miffs me is that per-connection throughput gets chopped down (20-30Kbps per download, usually) from known-fast sites - and I have no problem downloading quickly while directly connected to the modem with my laptop.
Any help with either the throughput or latency would be greatly appreciated (primarily the throughput, because that's horrible).
Thanks.
-
Anyone have any ideas on this? Anything at all!
-
Have you checked the pfSense error counters for the interfaces?
What download speed do you get to the Alix compared with download sped to a system downstream of the Alix?
-
wallabybob, it doesn't matter: regardless of whether I'm downloading eg. pfSense updates, going over a system on wireless (3 devices from the modem), or on a hardwired system (2 devices from the modem), the results are the same. For instance, I'm 'maxing out' my connection as we speak (at 5Mbit), downloading a half dozen files from a work site that has a gigE connection and that I can pull 35M on with a direct connection - http://tinypic.com/view.php?pic=jidqf9&s=6
Notice the CPU, memory, MBUF, state table, etc. - these are all pretty damn static.
Everything screams "ISP problem" to me, but it's quite apparently specific to this pfSense device, and I can't figure out why.
In/out error counters on all interfaces is 0/0. I've very rarely seen an error on the counter.
I should note: I've changed out the switch now, an older rack 10/100 Linksys for a late model HP Procurve. I'm still not seeing any difference.
-
Is the CPU heavily loaded?
Is the ALIX power supply delivering suitable voltage and current?
Does your modem have the ability to ping the ALIX? If so, what are the response times like? What are the response times like when you ping the ALIX from a system on the LAN interface? Any losses reported in (say) 100 packets?
-
The CPU is just sitting idle. It never peaks - eg I can sit there in console with vmstat and not see anything significant.
I'm using a power brick from an old Linksys modem - I forget what it's delivering, exactly, but it's right/above what the ALIX was designed for (I double checked when I started using it).
I don't have access to the modem. I've not been able to get into the interface yet, like I have been able to with different modems in the past.
Latency isn't really a "problem" - it's "normal" for what can be expected for around here, maybe even a little on the good side (30ms average for in-state type connectivity, 50-80ms for elsewhere).
Statistics on the WAN interface right now (9 day uptime - I have it scheduled to reboot twice a month on the 1st and 15th) are:
Media 100baseTX <full-duplex>In/out packets 67722571/67602120 (53.00 GB/46.54 GB)
In/out packets (pass) 67602020/67949144 (52.98 GB/46.54 GB)
In/out packets (block) 120551/100 (14.90 MB/7 KB)
In/out errors 0/0
Collisions 0I've never seen an in/out error that I can recall.
I can ping eg. google.com all day without any packet loss and 60-65ms latency. Even with a larger packet size (eg ping -s 512 google.com) - I still have low enough jitter for VoIP to be usable.
54 byte packets:
–- google.com ping statistics ---
137 packets transmitted, 137 received, 0% packet loss, time 136181ms
rtt min/avg/max/mdev = 62.516/68.732/155.162/17.354 ms512 byte packets:
--- google.com ping statistics ---
145 packets transmitted, 145 received, 0% packet loss, time 144199ms
rtt min/avg/max/mdev = 62.938/65.415/138.578/6.573 ms(Ironically, the larger packets have a shorter round trip... which makes sense due to prioritization I suppose.)</full-duplex>