Drastic speed reduction in 2.4.0 vs 2.3.4?

peppersass

Short version of my question: Are there known issues with speed degradation in 2.4, especially when using OpenVPN? If so, are there any solutions?

I'm new to pfSense. I installed 2.3.4 on a Zotac ZBox C1323 and got it to work over IPV4 and native IPV6 with my ISP, and it works over IPV4 with my VPN via OpenVPN at about 90% of full speed, which is what I get on other routers. But I was unable to get a connection to my VPN over IPV6. I posted about that here: https://forum.pfsense.org/index.php?topic=136392.0.. But that's a different issue than the one I'm posting about now.

There's some indication that the parameters sent by my VPN aren't compatible with the pfSense 2.3.4 version of OpenVPN 2.3, so I updated to the experimental version of pfSense 2.4.0. The connection errors I was getting on OpenVPN in 2.3.4 went away, though the connection still doesn't work. But there's a much more serious problem: download speeds via the OpenVPN IPV4 connection are horrendous – anywhere from 20% to 50% of full speed. The non-VPN connections are butter, but they're only running about 80% of normal speed. And I see unusually long times in the RTT and RTTsd fields for the OpenVPN and non-OpenVPN connections -- at least 2x to 3x normal, sometimes as high as 10x, and sometimes some losses, which I don't see on 2.3.4. Following instructions I found here, I downgraded to 2.4.0 RC, but it has the same speed problem.

I'd love to move on to 2.4.0 RC because it looks like I'll have a better chance of getting OpenVPN IPV6 to work with my VPN. But the speed downgrade is a non-starter.

Is this a known problem with 2.4.0? If not, what could possibly be causing such a drastic difference in performance between 2.3.4 and 2.4.0? Are there any logs or diagnostics I can use to get a handle on what's happening?

L068904

When i upgraded to 2.4RC, I struggled with OpenVPN and getting the same speeds as in 2.3.4 ( site resolution is what I am referring to). There were a few things I had to toggle with. Not sure it will help you, but I got to get same "speed" as I had before with 2.3.4. it wasn't a pfsense issue, it was a setup issue for me.

Setup. I ended up using load balance using 3 VPN servers under one gateway and let pfsense serve as round robin. I host my own OpenVPN servers so I can control (to an extent) latency. so I know that a) there aren't a million people on the servers, b) I get to chose the location closest to me for low latency.
Verify Setup of OpenVPN clients. Enable NCP, Adaptive LZO compression, don't pull routes & don't add remove routes (checked), UDP Fast I/O checked, send/receive buffer 512K.
DNS Resolver. Network interfaces (lan and vlans if you have them), Outgoing network interfaces, only your OpenVPN interfaces otherwise you may leak. DNSSEC checked, DNS query checked, DHCP registration checked, static DHCP checked.
DNS forwarder. DNS query forwarding uncheck sequential dns check.
General setup. your DNS servers listed but no assigned gateway. DNS server override uncheck, otherwise you'll leak.

other thoughts,

have you considered hosting pfsense virtually and connecting to it site-to-site and let the virtual site be your gateway? not sure if this setup may bypass a lot of the issues you face with your local provider. There are a ton of youtube videos as to how to do this.

hope this helps

peppersass

Thanks for the reply. Very helpful.

My issue isn't DNS speed, it's download/upload speed. That said, I didn't pay attention to DNS speed on 2.4.0 because the download/upload speeds were so broken.

I did have some issues with DNS config in 2.3.4 but after some experimentation I ended up with pretty much the same parameters you suggested. I wasn't using the DNS Resolver because I'd had some trouble when it was enabled, but I just enabled it with your suggested parameter and DNS is working pretty fast with no leaks. I did have to change the DNS Resolver port from 53 because the DNS Forwarder uses that port. I set it to 54. Any issues with that?

When I go back to 2.4.0 I'll be sure to update the DNS parameters.

Not sure your client setup suggestions are going to work with my VPN. I tried checking don't pull routes and don't add remove routes in 2.3.4, but that caused the connection to fail. Will have to wait 'till I reboot under 2.4.0 to see if NCP, UDP Fast I/O and send/receive buffer 512K help (FWIW, my VPN pushes ncp-disable.)

I'm not very experienced with advanced routing, so I don't understand what you mean by hosting pfsense virtually. Can you elaborate?

L068904

let me take these in sections…

"I did have to change the DNS Resolver port from 53 because the DNS Forwarder uses that port. I set it to 54. Any issues with that?"

I'm used to seeing 5353 as an alternate, but really you should be able to use 54 if u like…

"Not sure your client setup suggestions are going to work with my VPN. I tried checking don't pull routes and don't add remove routes in 2.3.4, but that caused the connection to fail. Will have to wait 'till I reboot under 2.4.0 to see if NCP, UDP Fast I/O and send/receive buffer 512K help (FWIW, my VPN pushes ncp-disable.)"

These will only help you if you have multi-openvpn gateways routed into one gateway, the purpose for that is to prevent leakage …and the buffering setting, you'll have to play with it to find out if something other than the default may work...I'd leave it alone if its causing problems...

I am not really sure what your end goal is, I've been using pfsense for about 5 years now, and the main reasons for me using it are 1) protection from intrusion, 2) privacy in or out of the network. With openvpn 2.4, I know that IPV6 is now a possibility, however, since I started using pfsense and openvpn, ipv6 has been a "no-no", most vanilla vpn services out there ask for you to disable it. I have it disabled in my setup, and don't intend to start using any time soon, besides I host my own openVPN servers, latency and users will certainly undermine your bandwith and througput...i dont know one openvpn that doesnt. My comment to you, and this is nothing new, is that a lot of people use pfsense either on a virtual machine locally (ESXI, hyper-V) or in the cloud. Up until recently, at least this is my experience, it has been an challenge/limited to setup Pfsense in the cloud ( it still is), but now there is a virtual appliance for AWS, MS Azure, and I know Vultr has it as an ISO library, so my thought, in your case, since you are required to do ipv6 that perhaps, instead of using pfsense locally, you could circumvent all those requirements from your ISP and put pfsense on a vps...tht way you can bypass all the issues you are having with your ISP and configuring ipv6 with openvpn. A thought was to setup a site-to-site and use your vps pfsense site as the gateway instead of your local device...there is a ton of instructions in Youtube as to how to set this up with pfsense...unfortunately, I am not that knowledgeable about setting up ipv6 with openvpn and pfsense.

peppersass

I tried your suggestions under 2.4.0 and got the same result – I still get dismal download/upload speeds through the VPN. Unusable. Speeds are fine in 2.3.4. IPV6 does connect on 2.4.0. There are no errors in the OpenVPN logs and the RTT and RTTsd times change. But no traffic flows over the interface. Seems to be a routing issue, but I don't know what it is. Frustrating. Even if I could solve that, or even if I turned off IPV6, IPV4 speed through the VPN is a non-starter.

Not sure what you mean by "multi-openvpn gateways routed into one gateway". I have one gateway for the VPN, which goes through the WAN gateway.

Actually, IPV6 isn't a must-have. I have native IPV6 through my ISP and simply wanted the same capability through the VPN with the real IPV6 address masked. My VPN is one the few that support IPV6. IPV6 works over the VPN fine if I use their client software on Windows or IKEv2 on iOS or ISX -- I can connect to IPV6-only sites and there are no leaks. They support OpenVPN on several platforms, so I need to try that and see if those OpenVPN implementations work with IPV6.

But many devices in my network aren't computers, so I can't route them through the VPN unless I use a VPN router. Also, the humans using the computers in the network aren't technically savvy. I'd rather hide the VPN from them and administer the whole thing from the router. pfSense seems ideal for that application.

FWIW, my VPN has been around for a while and they have first-rate security and features. Their servers are lightly loaded and very fast. One of the more expensive services, but worth it.

My goal is to go as private as technically possible. Not for any nefarious reason, just to see how far I can get with it. I don't want any traffic leaving my network to be in the clear. Don't want my ISP to have any information other than the VPN servers I'm connecting to. I don't see how cloud-based pfSense can do that unless I run encryption software on all the devices in my network, which I can't.

For now I've turned off IPV6 globally in pfSense to prevent leaks. Aside from the fact that IPV6 through the VPN doesn't work, I'm not able to consistently route IPV6 from iOS devices in my network to the VPN (where it won't work, but at least it won't leak.) Turns out the iOS IPV6 address changes every time the device connects to the network, and iOS doesn't let you set a static IPV6(!). I've tried using an alias with the FQDN, but either it doesn't resolve to IPV6 addresses or it doesn't update when the IPV6 address changes.

doktornotor

I think this has exactly zero to do with IPv6. Perhaps the guys should finally figure out what kind of accelleration is needed for OpenVPN instead.

https://redmine.pfsense.org/issues/7810

peppersass

Thanks. I suspect you're correct.

I saw that bug report but didn't make the connection because when I first started using 2.3.4 I had Hardware Acceleration set to none and download/upload speeds through the VPN were fine. I've since set it to BSD but haven't seen any difference.

Last time I booted under 2.4.0 I did notice that I couldn't set BSD hardware acceleration, but figured it didn't matter, like in 2.3.4. Evidently not.

Serious bug for people with affected hardware configurations. Hope it gets into 2.4.1 soon.

peppersass

Turns out the speed reduction has nothing to do with crypto acceleration. First, there's at least a 20% speed reduction in 2.4.0RC when not using the VPN. Second, the non-VPN RTT and RTTsd times are 2x-3x what they are in 2.3.4. Third, I tried turning off BSD Hardware Acceleration in 2.3.4 and it made no difference to the VPN speed.

Could this be an issue with the Zotac ZBOX C1327? Has anyone else used one of those successfully with 2.4.0RC? I would have gotten the more proven C1323 but couldn't find one.

doktornotor

@peppersass:

Third, I tried turning off BSD Hardware Acceleration in 2.3.4 and it made no difference to the VPN speed.

That was not the point of the linked bug. On the contrary, the point was exactly the opposite - to have both AES-NI and cryptodev loaded. Testing this with 2.3.4 is not relevant here.

digitalgimpus

I've noticed something pretty similar to this on my Qotom.

I think there's something in 2.4 that's impacting performance of RealTek NIC's significantly. Previously I didn't have a problem hitting a few hundred Mbits/s… given my uplink more than ok... With 2.4:

$ iperf -c gateway -w 256k
------------------------------------------------------------
Client connecting to gateway, TCP port 5001
TCP window size:  416 KByte (WARNING: requested  256 KByte)
------------------------------------------------------------
[  3] local 10.10.1.42 port 44840 connected with 10.10.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   107 MBytes  89.4 Mbits/sec

Can't seem to get these above 90 Mbits/sec.

It's indeed detected as gigabit:


re0@pci0:1:0:0:	class=0x020000 card=0x012310ec chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
re1@pci0:2:0:0:	class=0x020000 card=0x012310ec chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet

The interface is definitely "1000baseT <full-duplex>".

Very strange… I'm running out of things to look at.</full-duplex>

JeGr

@doktornotor

Why would one want both loaded, aesni and cryptodev? The ticket from your link (https://redmine.pfsense.org/issues/7810) has those numbers at the bottom, right? Isn't the benching of both loaded the worst by almost factor 3.5 or am I misreading something entirely? Just asking because the ticket title and you stating the same makes no sense to me when looking at those speed tests?

Thanks,
Jens

heper

@JeGr:

@doktornotor

Why would one want both loaded, aesni and cryptodev? The ticket from your link (https://redmine.pfsense.org/issues/7810) has those numbers at the bottom, right? Isn't the benching of both loaded the worst by almost factor 3.5 or am I misreading something entirely? Just asking because the ticket title and you stating the same makes no sense to me when looking at those speed tests?

Thanks,
Jens

look at the times next to the numbers …. its confusing, but loading both is significantly fasters, specially if size increases

doktornotor

^^^ That. Yeah, you are misreading entirely.

peppersass

Again, I'm seeing significantly reduced transfer speeds (20%+) and RTT/RTTsd times (2x-3x) on the WAN with no VPN. But speed really goes to hell over the VPN.

My NICs are Realtek 8168/8111. Googling came up with posts about connection and speed issues with these NICs under various OSs that are solved by updating the driver. How do I find out what Realtek driver versions are incorporated into freeBSD for pfSense 2.3.4 and 2.4.0RC? If a driver update is required for 2.0.4RC, is this something I can do or will it have to be incorporated into 2.4.0RC by the developers?

FWIW, Realtek lists their latest freeBSD driver for the 8168/8111 as 1.94 released 9/15/2017. But it says it's for freeBSD 7.x and 8.0. I believe pfSense 2.3.4 is using freeBSD 10.3.

Of course, it may not be the driver, but how the NICs are being initialized. I've seen some posts where people have tried different parameters to resolve connection and speed issues. But I wouldn't know where to begin with that. Is it possible to find out what initialization commands are being issued by 2.3.4 and 2.4.0RC, respectively? If there's a difference, is it possible to send commands to the NICs with shell commands?

JeGr

@heper:

look at the times next to the numbers …. its confusing, but loading both is significantly fasters, specially if size increases

My bad, my ill brain didn't take a look at the timings in the last one. Thanks!

hescominsoon

@digitalgimpus:

I've noticed something pretty similar to this on my Qotom.

I think there's something in 2.4 that's impacting performance of RealTek NIC's significantly. Previously I didn't have a problem hitting a few hundred Mbits/s… given my uplink more than ok... With 2.4:
$ iperf -c gateway -w 256k
------------------------------------------------------------
Client connecting to gateway, TCP port 5001
TCP window size:  416 KByte (WARNING: requested  256 KByte)
------------------------------------------------------------
[  3] local 10.10.1.42 port 44840 connected with 10.10.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   107 MBytes  89.4 Mbits/sec
Can't seem to get these above 90 Mbits/sec.

It's indeed detected as gigabit:
re0@pci0:1:0:0:	class=0x020000 card=0x012310ec chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
re1@pci0:2:0:0:	class=0x020000 card=0x012310ec chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
The interface is definitely "1000baseT <full-duplex>".

Very strange… I'm running out of things to look at.</full-duplex>

Considering realtek is a software nic written in the winmodem model a slowdown isn't surprising. If you can get some Intel I-series nics or broadcoms and try again. If the slowdown is still present then I would worry about there being a general code issue. Even then the new codebase might be cpu hungry until it gets evened out..but usually these kinds of networking slowdowns are due to "win-nics".

peppersass

@hescominsoon:

Considering realtek is a software nic written in the winmodem model a slowdown isn't surprising. If you can get some Intel I-series nics or broadcoms and try again. If the slowdown is still present then I would worry about there being a general code issue. Even then the new codebase might be cpu hungry until it gets evened out..but usually these kinds of networking slowdowns are due to "win-nics".

Alas, my Realtek NICs are on the motherboard of a Zoltac ZBOX C1327, which isn't expandable. :(

peppersass

FWIW, I switched from the Zotac C1327 to a Protectli E3845 Vault with Intel NICs and there's no problem with speed under 2.4.2. I think it's likely an issue with the Realtek driver used in the latest version of FreeBSD, though it could also have something to do with a crypto or BIOS incompatibility.