Slow internet connection behind pfsense
I have an issue with some of my existing pfsense boxes that are onsite. These boxes were built and put in place prior to me coming into this role and they have been all fine. Up until last week when we trialed Fiber in one of our sites. Now the connection when you plug straight into the BT openreach box is great. 45-50MB download, 13-15Mb upload.
But when I look behind the pfsense box I am getting a lot less then that. I am getting only 6-7Mb download, and 0.2-0.4Mb upload.
I have looked and there are no limiters in place on the machine. I have tried playing around with the MTU on it, but still no success.
I wondered if anyone could point me in the right directions. I have also checked the hardware and it is hardly taxing the system. Not even 15% CPU usages and less then 50% memory usage even when I try push the system and network.
Any help is appreciated.
That much restriction I would look at a speed/duplex mismatch between the vdsl modem (which model do you have?) and the pfSense WAN interface. If both ends are not set to 'auto' then they need to be set the same fixed setting. Usually (but seemingly not always!) you would see collisions or errors on the WAN interface in that situation.
What hardware is pfSense running on? Anything in the logs?
We are using a Xyzel NBG4604 router which then connects to the pfsense box. Both are set to Auto. Also, no collisions on the WAN interface on the pfsense.
Nothing unusual in the logs.
All my Pfsense boxes have Ipsec tunnels which push all internet traffic back to my HQ office to then go through a firewall filter here. But nothing on that end should limit it either from what I can see
Ah ok. So you have a double NAT setup? You should try connecting the pfSense box directly to the vdsl modem and letting pfSense handle the PPPoE connection if you can. If you look at Status: Interfaces: WAN: it shows both no collisions and no in/out errors? If you run ifconfig (your wan interface) at the command line does it show that it has connected via autoselect:
$ ifconfig fxp5 fxp5: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=4219b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwtso>ether 00:90:7f:67:56:f6 inet6 fe80::290:7fff:fe68:56f6%fxp5 prefixlen 64 scopeid 0xa nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>) status: active</full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,wol_magic,vlan_hwtso></up,broadcast,running,simplex,multicast>
Sometimes two bits of hardware just don't play nicely together, try connecting a switch in between the pfSense box and the router.
If all of your traffic is being routed via an ipsec tunnel how are you running the test?
Yes. It shows Autoselect.
media: Ethernet autoselect (100baseTX <full-duplex>)
I might try the switch to see if that helps.
I initially connected a laptop directly to the BT Openreach box and setup a connection on there to the ISP, then ran the speed test. I got the types of speeds I would expect from the line. Then I plugged the Zyxel router back into the BT box and then the laptop into the Zyxel, same result.
Then the Zyxel plugged into the WAN socket and the laptop into the LAN network of the pfsense and I got the reduced speeds.
Even if all the traffic is being routed, I should still be getting decent speeds no? They are being routed from a fiber line onsite to a 100MB fiber line at my HQ.
My next test I am doing is getting a clean build of Pfsense without any Ipsec tunnels and going to test that. But if you have any other suggestions in the meantime I would gladly hear them.
Oh and the test I am doing is via the speedtest.btwholesale.com site.
I assume you don't have any conflicting subnets, you must quite a few private IPs in the chain there?
You can run a test directly from the pfSense box to test the speed at that point:
[firstname.lastname@example.org]/root(2): fetch -o /dev/null http://download.thinkbroadband.com/50MB.zip /dev/null 100% of 50 MB 1961 kBps 00m00s
Thinkbroadband have a number of file sizes to choose from and have always proved a high bandwidth source for me.
If that shows a good number then it could be speed/duplex mismatch between the laptop and pfSense box? What hardware are you using for pfSense?
We only have 3 Interfaces on our Pfsense boxes. 1 Wan, which uses the 2nd usable external IP address for that service (zyxel being the first usable), then 2 LAN cards. 192.168.x.1/24 and 172.16.x.1/24. There could be about 20-40 private addresses on the 192 range, maybe about 10/20 addresses tops on the 172 range.
Here is what i am getting from your command.
/dev/null 37% of 50 MB 960 kBps 00m34s
960 is what it topped out at.
The machines themselves are older types.
Intel(R) Celeron(R) CPU 2.66GHz with 512mb memory.
Ok, well that hardware should be capable of well over 8Mbps, like >500Mbps.
So the Zyxel box is not NATing the connection between the modem and pfSense box? I assume you must have configured it to do that since it would be setup for NAT by default. When you connected the laptop directly to the Zyxel box did you also give that that a public IP address? It seems like you might have a routing issue through the the Zyxel box.
Yes, the Zyxel box has NAT disabled. Also DHCP. The Pfsense does the NAT'ing for us on the sites.
And when I plug a laptop directly into the Zyxel box I have to set the laptop up with a manual IP address as though it was the WAN interface on the pfsense.
Hmm. Well I'd definitely try a switch between the devices to make sure it's not a low level glitch before anything else.
Has this pfSense box been proven anywhere else?
This is the first time we have tried to use the existing Pfsense box on a fiber line in this way. All my other boxes like this are on a normal ADSL line and they seem to be ok.
Hmm, well try the switch. Try removing the Zyxel box completely.
I have a pfSense box here connected directly to the Openreach modem and it connects via PPPoE and gets full line speeds no problem. It's a home connection though. I've never dealt with BT's business hookups personally but there have been other users here who have.
Can you post your RTT (pings should be fine) between those offices? I have noticed that IPSec is especially sensitive to congestion when there are long round trip times and bandwidth is higher (you can read a little about it if you Google "Bandwidth Delay Product"). There is a whitepaper that sheds a bit of light on the IPSec part here: http://www.academia.edu/694268/_TCP_in_the_IPSEC_environment
To work around this, you may seriously consider implementing an OpenVPN solution to replace IPSec. pfSense has an excellent implementation of OpenVPN point to point and can handle failover (via CARP).