PLEASE HELP!!! pfSense is slowing down my internet.

[kovon]

Good Day to all

I need someone to assist me in troubleshooting why pfSense is slowing down my internet.

I have pfSense set up in Hyper-V and when I do a speed test I get 7-23 Mbps down and 15-59Mbps up. My ISP speeds are usually >100Mbps down and 50-80Mbps up.

My host machine is: i7 processor, 20GB of RAM, onboard PCIe GBe NIC, 1 expansion PCIe GBe NIC and 1 PCI GBe NIC, and nVidia 1060 6GB GPU.

I have 2 VMs. The first VM has pfSense and 2 NIC - 1 for WAN and 1 for LAN and the other VM is a Windows 10 with 1 LAN, the same internal network as the pfSense VM.

This is a test environment that I am setting up as a proof of concept to configure 2 OpenVPN clients in pfSense that I can alternate between. So I only need pfSense for it routing capabilities and the OPenVPN interfaces. When I get this working properly I will install pfSense on a SATA 2.5 HDD and place it in my 4 GBe port mini PC to run my VPN services. I am currently using DD-WRT x86 on my 4 port mini PC but DD-WRT only allows for 1 OpenVPN client via the GUI and I have no background in Linux and while I am comfortable coding and interpreting commands, I can find no clear instructions on how to configure the 2 OpenVPNs in DD-WRT via CLI. Most of what I find, are posts telling you what to do but not how to do it as they assume persons already know the necessary linux commands to configure scripts.

So to my problem at hand...

I am getting internet and can access web pages on the Win 10 VM.

When I do a speed test in the host machine, I get full speed. When I test in the Win 10 VM guest that is receiving internet from the pfSense VM, I get the slow speeds.

I have searched the internet and this forum, and I have tried most if not all the suggestions I have read/found including

1. checking the duplex on the interfaces. - Both were set to the same duplex.
2. Disable the Hardware checksum overload. - No change
3. unticking everything in Network Interfaces under System->Advanced->Netwokring. - No change
4. upgrading the drivers of my NICs. They were dated 2015 before. Now they are dated 2018. This did improve speed. I was original testing 7-10 down now I am testing 7-23 down. etc. Incidentally the dates on the virtual Hyper V adapters say, 2006, but I cannot find any place where there are any drivers for the virtual adapter. However, while the speed is better, it is no where near full capacity which I need it to be to account for the lost in speed on OpenVPN.
5. Disabling Packet filtering but this of course, Disabled NAT so I am unable to speed test to see if the Firewall is the cost. I did see someone post that as an alternative you can set NAT to manual and set the Firewall rules to any. But when I looked at the rules, I was unclear on this so I did not go any further down that possibility. If someone can, explain this further I am more than willing to try this.

Thinking it was Hyper-V, I decided to try an Untangle VM in Hyper-V and that got full speed straight off the bat. So I am not convinced it HYper-V. I mean I could set this all up in VMware player or VirtualBox but I don't want to take the time to do all that and it bares no fruit. I prefer Hyper-V because 1) my background is Windows and 2) Hyper-V vms automatically restart with Windows without any additional configuration to get the other VM types to run as a service.

So please, please, please HELP. Can anyone tell me what is lowering my internet speeds? why is the upload higher than the download? if it is the packet filtering? how exactly can I configure it to stop filtering but keep NAT?

Help...

[stephenw10]

@kovon said in PLEASE HELP!!! pfSense is slowing down my internet.:

unticking everything in Network Interfaces under System->Advanced->Netwokring.

Those offloading options should all be ticked. Ticked boxed there means disabling the hardware offloading which may not be supported correctly.

Steve

[kovon]

@stephenw10

I figured as much. I saw someone post that unticking them improved their internet speed which seemed counter intuitive but I tried it none the less. Did not work. I believe I have already ticked them back or plan to. As I recall the first one was unticked by default and I ticked it after reading many people saying to disable it. but that didn't work. So I willing to try any and everything at this point.

[stephenw10]

You can try running a test from the pfSense VM itself. That will determine if the issue is WAN or LAN side.

At the command line:

pkg install -y py27-speedtest-cli
rehash
speedtest-cli

Or by fetching a test file directly:

[2.4.4-RELEASE][root@8860.stevew.lan]/root: fetch -o /dev/null http://download.thinkbroadband.com/50MB.zip
/dev/null                                     100% of   50 MB 8697 kBps 00m06s

You might want to use a test file more local to you.

Steve

[kovon]

@stephenw10

Thank you Steve for the suggestion. It has shed some much needed light on the situation.

I installed speedtest-cli as you said on pfSense. These are my findings:

The speed test from my Host machine is as follows:

The speed test and test download from pfSense VM is as follows:

The speed test from the Win 10 VM is as follows:

So the issue seems to be while pfSense is getting the full internet speed, it is not pushing out the same speed to the Win 10 client.

I just checked and had not re-enabled hardware checksum offload, hardware TCP segmentation offload and hardware large receive offload on the System->Advanced-Networking page.

I reenabled them, restarted pfSense and redid the speed tests.

From pfSense VM:

From the Win 10 VM:

So that confirms that those offload settings had no effect on the speed.

Any ideas of what could be decreasing the speed going to the Win 10 VM?

Regards.

[Gertjan]

@kovon said in PLEASE HELP!!! pfSense is slowing down my internet.:

Any ideas of what could be decreasing the speed going to the Win 10 VM?

The answer is in the question.
Take out the VM part. I'm pretty sure you'll find numbers that you like. You will also know where to focus on when going back to a VM solution.

[stephenw10]

Is the WAN NIC there pass-through to the VM?

How is the LAN NIC configured in Hyper-V? You might try the legacy NIC type there instead.

I doubt it's an issue with Windows in HyperV but you could throw an Ubuntu client in there on the same subnet just to be sure.

I assume you read this:
https://docs.netgate.com/pfsense/en/latest/virtualization/virtualizing-pfsense-with-hyper-v.html

Steve

[kovon]

@stephenw10

Hey Steve,

Sorry for taking so long to respond. It's been a long headache with this thing.

So basically, you were right the issue was the VM. I tried the legacy adapter but I was getting unidentified network and in trying to troubleshoot that my host machine started blue screening on me. So I had to resolve that.

Anyhow after I was back up and running, I tried a new approach. I took an actual hard drive (the one I plan to install in my Mini PC) and set up a new pfsense VM to pass through to the hard drive. This way I can then take the hard drive and boot straight into pfsense without having to reinstall and reconfigure etc. I then used 4 NICs and bridged the hyper-v virtual switched to each of them so I can connect the NIC to actual switches and computers rather than using VMs internal or private networks. When I speed test in the VM using internal virtual switch the speed is slow. But when I use an actual device connected to an actual bridged network adapter, I get full speed coming off the pfsense. So i am think pfsense was not the issue but some misconfiguration in the hyper-v set up somewhere. So while I have not resolved the slow speed in the VM issue, I am sticking a pin in that for now, so I can get my pfsense up and running.

Before re-configuring the OpenVPN in this new hard drive, I decided to try bridging the 3 interfaces that will be used for the LAN side of things. As I said orignally, this whole exercise is to replace my DD-WRT router with pfSense so I can have 2 OpenVPN so i need pfSense to operate like a SOHO router with 1 WAN port and 3 LAN ports.

I have used the following 3 links (plus a fourth that I can't seem to find now) to help me configure the bridge but it is not working

https://www.cyberciti.biz/faq/how-to-pfsense-configure-network-interface-as-a-bridge-network-switch/
http://centosquestions.com/creating-bridge-lan-opt1-pfsense-firewall/
https://eengstrom.github.io/musings/configure-pfsense-bridge-over-multiple-nics-as-lan

So far I have:

1. configured the system tunables and toggled the two .member and .bridge settings
2. I have created a bridge - some sites said do not include LAN until afterwards and other included it the same time the bridge was created
3. added bridge to interfaces
4. created firewall rules
5. created interface group
6. set ip on bridge and configured dhcp server
   etc

What is happening is that when I disable the IP on the original LAN port, the bridge interface is not coming up on none of the other NICs. it is not assigning an IP. Tried static and that did not work either.

I will keep trying but I wanted to give you an update of the Speed test issue. The VM is definitely causing an issue there but I will return to that later if need be.

I am new to the forum thing. Should I create a new post for this bridging issue or is it okay to continue troubleshooting it here?

[Raffi_]

@kovon it would be best to start a new thread. If a solution does come up, it'll be easier for someone to search for a term in the heading that applies to the actual issue. Whereas, if you continue in here they'll come to a solution to something they were not expecting.

[stephenw10]

Ok, so let me just say that bridging interfaces in pfSense because you want it to behave like a switch is a bad idea.
If you want a switch, like the switch inside a SOHO router, use a real switch. Or run the SG-3100/XG-7100 which actually have an internal switch.

That said you can bridge interfaces and it usually works OK.

Probably going to need screenshots of what you have done. You don't usually need an interface group if you moved filtering off the member interfaces. The rules will all go on the assigned bridge interface.

It's better to try to do this whilst NOT connected to any interface you want to have on the bridge , it's very easy to get locked out. So maybe enable WAN side access or configure another interface to use whilst you do this.

Steve

[kovon]

@stephenw10

Hey Steve, I responded to your bridging comments on my Bridging post. Please see.

Back to the internet speed issue on pfSense.

After installing pfSense on my mini PC and configuring the 2 OpenVPN Clients in a VPN group as Tier 1 and Tier 2, I have noticed that while the speed test with the VPNs off on pfSense itself is basically the same as on the my desktop computer, I am noticing different speeds when the VPN is on.

With my Tier 2 VPN on :

On my Desktop:

On pfSense:

As you can see, pfSense is reporting faster speed than what the client is getting. Can you tell me how to fix that? I want the client to get full speeds to whatever pfSense is capable of.

Another strange issue, When my Tier 1 VPN is on and I try to run speedtest-cli on pfSense, it is picking up my ISP internet as if the VPN is off. So the Client sees the VPN and Tier 1 speed is slower than Tier 2 but pfSense is testing as if the VPN is off. Any ideas as to what I have misconfigured?

Thanks in advance

Kevin

[Gertjan]

What probably happens :
speedtest is using your WAN, not some "VPN WAN".
Clients on your LAN are forced to pas through the/a VPN.

Run

speedtest -h

and you'll see

...
  --source SOURCE       Source IP address to bind to
...

So, bind speedtest the IP of your VPN to force using that interface.
I couldn't test this myself - I have no outgoing VPN.

Btw : it's normal that VPN connections are slower then non VPN connections.
Also : speedtest running on your pfSense does not pas through your LAN interfaces (you bridged them).

[kovon]

@gertjan

Thanks for your response.

I think I understand what you are saying but this is my challenge:

I had my Tier 2 VPN configured on my DD-WRT router and the speed test on my desktop was around the same 60Mbps down 50 Mbps up when the VPN was enabled which is the same speed I am getting on pfSense itself using speedtest-cli. But now the very same VPN configured on pfSense using the exact same hardware is testing barely in the 20s from the very same desktop. The only thing that has changed is the firmware.

So based on what you are saying, on DD-WRT, the VPN traffic my desktop got was using the WAN interface on the router (since I guess there was only one WAN on the routet) and pfSense speedtest-cli is using the same WAN, which is why their speed tests are the same. But now, my desktop is using a different WAN because of these VPN gateway interfaces that had to be created on pfSense.

But the service I am using the VPN for requires a minimum of 20Mbps in order for it to work properly. On DD-WRT, it worked but now on pfSense it is poor.

So my question is how can I reconfigure my VPN to use the WAN interface like pfSense is using and like how my dd-wrt did? And if it cannot, because pfSense is a firewall and everything is now different on a firewall, what changes can I make on the VPN WAN, to make it faster?

The reason I switched to pfSense was to use the 2 VPNs to toggle between them seemlessly and for failover without having to reconfigure a new VPN whenever I needed to switch since I could not have multiple VPNs on DD-WRT via the gui but I never imagined that I would lose speed in the process. I can't lose speed for what I am doing. So how can I get this speed back?

Also, I want to be clear, I know the speed on the VpN is slower than the speed on non-vpn. That's not my issue. The issue is my clients are getting a slower VPN speed on pfSense than they got on DD-WRT. If I can't get back the same speed I got before when the VPN is running, I will have to go back to DD-wrt and forget about having multiple VPNs configured which at this point I was starting to warm up to the idea of having a firewall protect my home network and was prepared to learn it inside out to even help improve my firewall skills at work. So it would be a bummer if I have to switch back.

And also, I have not bridged the LAN interfaces. I abandoned that plan.

I really hoping I can resolve this favourably.

[stephenw10]

What CPU do you have in that firewall? What ping times do you see to the VPN gateway? What encryption settings are you using?

If you are using UDP, which is the default for OpenVPN, then make sure you have fastIO enabled. Increase the send/recv buffers to 512K.

Steve

[kovon]

@stephenw10

Good Morning Steve

The CPU:

Ping Times and Speed Tests

Ping Times and speeds on desktop at home with all VPNs off:

Ping times and speeds on desktop at home with VPN Tier 1 on pfSense:

Ping times on desktop at home with VPN Tier 2 using pfSense (WOW!! I have never seen this speeds before. On DD-wrt, max was 65-69 Mbps and last night it was maxing out at 23Mbps. Don't know what is happening this morning.):

Tested Tier 2 again to be certain (Wow.):

As a comparison, I did speed test at work using VPN clients running on my work desktop.

Ping Times of desktop at work with no VPN running:

Ping times on desktop at work with VPN Tier 1 using SoftEther VPN CLient:

Ping times on desktop at work with VPN Tier 2 using vendor's VPN client (The vendor must have made a change because I never got this sort of speeds at work before. I hope this keeps up):

Tested Tier 2 at work again to verify (Speechless):

With the sudden jump in Tier 2 speeds, I don't know what to make of it.

Anyhow to answer the rest of your questions:
My Tier 2 vendor provided guides for configuring openvpn on DD-WRT and pfSense. On their DD-WRT guide, they said to use Blowfish CBC but on their pfSense guide they said to use AES-128-CBC. Last night, I tried changing the encryption of the pfSense to BF-CBC but there was not much difference, now this morning it has boosted up significantly. Other differences in the vendor's configuration was DD-WRT said LZO-Comp No while pfSense was Adaptive. Also, custom options were completely different. I email my Vendor's support last night to ask them about the differences in their configuration guides but they said they forwarded it to their technical department so I am yet to hear back from them. I wonder if the boost in the VPN on pfSense is as a result of my query or if it coincidental?

My Tier 1 VPN is a VPN I built myself on a VPS I recently got. So that configuration is basically me "trial-and-error-ing" until I got it to work. I did how configured it exactly like my Tier 2 vendor's configuration and then tweaked until it worked.

On the Tier 2, I enabled FastIO and increased the buffer size to 512KiB. But the speed test on my desktop at home went back to last night's speeds:

I disabled FastIO and set the buffer size back to Default and the steed test on my desktop at home was this:

So I am now uncertain what to do to get back those blistering VPN speeds I got earlier. Or maybe they were indeed flukes.

I did a speedtest again on my desktop at work with the Tier 2's vendor's vpn client and the speeds are still extremely high:

So its definitely some configuration in pfSense that makes the speeds inconsistent. I will leave it as is and see if it goes back high after a few hours.

Regards

[stephenw10]

FastIO will only help if the provider is using UDP, which they probably are.

I would expect to see around 100Mbps from that CPU in a local test. The latency will reduce that and the encryption may not be optimal but that 96Mbps figure doesn't seem unreasonable.

If the tier 1 gateway is completely under your control you should be able to optimise that for best throughput.

100ms is a high latency though for both of those. I assume they are both far away from you.

Steve

[kovon]

@stephenw10

Wow Ok.

Well I never saw those kinds of speeds with the VPN on. I saw them on my WIFI with the VPN off. But via ethernet with the VPN on is new for me. I like though. I would very much like to replicate and make it the standard.

Yes the Tier 1 I created as a proof of Concept to see if it would give me access to some stuff, I am not able to access on my Tier 2. Hence why I want to make it my Tier 1. The only issue is that I suspect I may be limited by the internet speed available to the VPS. I have to contact the vendor to see what speeds should be expected on the VPS. So I don't know if I will get the same speeds as my Tier 2. However, 20Mbps is the minimum of my heaviest bandwidth services so if I can at least get double that at 40Mbps as a standard, I will be content.

I am new to creating your own VPN server, so I will now have to look into things such as throughput etc.

as it relates to the high latency, yes both gateways are over 2,000 Miles from me give or take.