Drastic speed reduction in 2.4.0 vs 2.3.4?

peppersass

Turns out the speed reduction has nothing to do with crypto acceleration. First, there's at least a 20% speed reduction in 2.4.0RC when not using the VPN. Second, the non-VPN RTT and RTTsd times are 2x-3x what they are in 2.3.4. Third, I tried turning off BSD Hardware Acceleration in 2.3.4 and it made no difference to the VPN speed.

Could this be an issue with the Zotac ZBOX C1327? Has anyone else used one of those successfully with 2.4.0RC? I would have gotten the more proven C1323 but couldn't find one.

doktornotor

@peppersass:

Third, I tried turning off BSD Hardware Acceleration in 2.3.4 and it made no difference to the VPN speed.

That was not the point of the linked bug. On the contrary, the point was exactly the opposite - to have both AES-NI and cryptodev loaded. Testing this with 2.3.4 is not relevant here.

digitalgimpus

I've noticed something pretty similar to this on my Qotom.

I think there's something in 2.4 that's impacting performance of RealTek NIC's significantly. Previously I didn't have a problem hitting a few hundred Mbits/s… given my uplink more than ok... With 2.4:

$ iperf -c gateway -w 256k
------------------------------------------------------------
Client connecting to gateway, TCP port 5001
TCP window size:  416 KByte (WARNING: requested  256 KByte)
------------------------------------------------------------
[  3] local 10.10.1.42 port 44840 connected with 10.10.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   107 MBytes  89.4 Mbits/sec

Can't seem to get these above 90 Mbits/sec.

It's indeed detected as gigabit:


re0@pci0:1:0:0:	class=0x020000 card=0x012310ec chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
re1@pci0:2:0:0:	class=0x020000 card=0x012310ec chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet

The interface is definitely "1000baseT <full-duplex>".

Very strange… I'm running out of things to look at.</full-duplex>

JeGr

@doktornotor

Why would one want both loaded, aesni and cryptodev? The ticket from your link (https://redmine.pfsense.org/issues/7810) has those numbers at the bottom, right? Isn't the benching of both loaded the worst by almost factor 3.5 or am I misreading something entirely? Just asking because the ticket title and you stating the same makes no sense to me when looking at those speed tests?

Thanks,
Jens

heper

@JeGr:

@doktornotor

Why would one want both loaded, aesni and cryptodev? The ticket from your link (https://redmine.pfsense.org/issues/7810) has those numbers at the bottom, right? Isn't the benching of both loaded the worst by almost factor 3.5 or am I misreading something entirely? Just asking because the ticket title and you stating the same makes no sense to me when looking at those speed tests?

Thanks,
Jens

look at the times next to the numbers …. its confusing, but loading both is significantly fasters, specially if size increases

doktornotor

^^^ That. Yeah, you are misreading entirely.

peppersass

Again, I'm seeing significantly reduced transfer speeds (20%+) and RTT/RTTsd times (2x-3x) on the WAN with no VPN. But speed really goes to hell over the VPN.

My NICs are Realtek 8168/8111. Googling came up with posts about connection and speed issues with these NICs under various OSs that are solved by updating the driver. How do I find out what Realtek driver versions are incorporated into freeBSD for pfSense 2.3.4 and 2.4.0RC? If a driver update is required for 2.0.4RC, is this something I can do or will it have to be incorporated into 2.4.0RC by the developers?

FWIW, Realtek lists their latest freeBSD driver for the 8168/8111 as 1.94 released 9/15/2017. But it says it's for freeBSD 7.x and 8.0. I believe pfSense 2.3.4 is using freeBSD 10.3.

Of course, it may not be the driver, but how the NICs are being initialized. I've seen some posts where people have tried different parameters to resolve connection and speed issues. But I wouldn't know where to begin with that. Is it possible to find out what initialization commands are being issued by 2.3.4 and 2.4.0RC, respectively? If there's a difference, is it possible to send commands to the NICs with shell commands?

JeGr

@heper:

look at the times next to the numbers …. its confusing, but loading both is significantly fasters, specially if size increases

My bad, my ill brain didn't take a look at the timings in the last one. Thanks!

hescominsoon

@digitalgimpus:

I've noticed something pretty similar to this on my Qotom.

I think there's something in 2.4 that's impacting performance of RealTek NIC's significantly. Previously I didn't have a problem hitting a few hundred Mbits/s… given my uplink more than ok... With 2.4:
$ iperf -c gateway -w 256k
------------------------------------------------------------
Client connecting to gateway, TCP port 5001
TCP window size:  416 KByte (WARNING: requested  256 KByte)
------------------------------------------------------------
[  3] local 10.10.1.42 port 44840 connected with 10.10.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   107 MBytes  89.4 Mbits/sec
Can't seem to get these above 90 Mbits/sec.

It's indeed detected as gigabit:
re0@pci0:1:0:0:	class=0x020000 card=0x012310ec chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
re1@pci0:2:0:0:	class=0x020000 card=0x012310ec chip=0x816810ec rev=0x06 hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
The interface is definitely "1000baseT <full-duplex>".

Very strange… I'm running out of things to look at.</full-duplex>

Considering realtek is a software nic written in the winmodem model a slowdown isn't surprising. If you can get some Intel I-series nics or broadcoms and try again. If the slowdown is still present then I would worry about there being a general code issue. Even then the new codebase might be cpu hungry until it gets evened out..but usually these kinds of networking slowdowns are due to "win-nics".

peppersass

@hescominsoon:

Considering realtek is a software nic written in the winmodem model a slowdown isn't surprising. If you can get some Intel I-series nics or broadcoms and try again. If the slowdown is still present then I would worry about there being a general code issue. Even then the new codebase might be cpu hungry until it gets evened out..but usually these kinds of networking slowdowns are due to "win-nics".

Alas, my Realtek NICs are on the motherboard of a Zoltac ZBOX C1327, which isn't expandable. :(

peppersass

FWIW, I switched from the Zotac C1327 to a Protectli E3845 Vault with Intel NICs and there's no problem with speed under 2.4.2. I think it's likely an issue with the Realtek driver used in the latest version of FreeBSD, though it could also have something to do with a crypto or BIOS incompatibility.