2.4.0 Upgrade Killed my Speeds

jimp

There is not nearly enough information here to offer any advice.

What hardware? What NICs? Any interface errors? What does the status of the WAN/LAN NICs show? ifconfig output? What packages are you using? What sort of load is the firewall under when you run the test? Any errors in the logs? Anything else you can provide that might be relevant.

CobraGT2000

Intel(R) Core(TM) i3 CPU 550 @ 3.20GHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: No

2 intel Pro GB NIcs
1 Netgear GB NIC

No interface errors.

Load of the firewall is less then 50% - even before at full GB download. Avg is 13-17%.

with Google, there needs to be a vlan tag on the WAN for 1GB upload, only config that is not the normal.

No system errors or anything out of the ordinary.

Wan.PNG_thumb

Packages.PNG_thumb
ifconfig.txt

deadmalc

This may have been fixed now, but I've not heard anything to say it has!
Are you using the "DNS resolver" - if you are try switching to the "DNS Forwarder".

I had awful problems (om my home firewall) when my IP changed, since switching back to the Forwarder I've had no issues at all.

You mention what the load is now, what was it before?
Also what's the load on the box, i.e. if you go to the console and type uptime.

It's worth disabling suricata and see if that makes a difference, to track the issue down.
On my server I've got that on without any issues after a 2.4 upgrade.

Limited info, so blindly suggesting ideas!

JKnott

Are you using the "DNS resolver" - if you are try switching to the "DNS Forwarder".

I don't see any bandwidth issues with the new version. However, what does the DNS method have to do with it? All DNS does is return an IP address for a host name. How the address is obtained should have absolutely no effect on speed.

deadmalc

The dns resolver goes crazy when your wan ip changes causing huge load.
On my sg1000 enough so the console can be unresponsive.

JKnott

Why is your WAN IP changing? Why would that affect the resolver?

kpa

@JKnott:

Why is your WAN IP changing? Why would that affect the resolver?

The way it's done is that every time the WAN DHCP lease gets renewed it counts as a WAN reconnect regardless of whether the dynamic IP changes or not. This reconnect restarts the unbound resolver and can be a rather time costly operation on lower end machines.

I still don't understand why a 'unbound-control reload' isn't enough on simple renewal of the lease where the IP address doesn't change, it's like hitting it with a hammer when a screwdriver is needed.

mhab12

I've experienced the GUI slowdown caused by DNS issues in the past, but it appears the OP is advising that throughput is way down. Likely unrelated to any DNS service issues especially since they are on an i3 machine.

opalmer

I also have Google Fiber I have this exact same problem and I'm running a Netgate SG-4860. On pfsense 2.3 I was getting >800 up and down. I'm less concerned about the download speed and more concerned with the upload speed I'm now getting;
http://beta.speedtest.net/result/6702311671.png

In order to use pfsense to with Google Fiber you have to configure the WAN to operate on vlan2 with priority 3, see this guide: https://www.dslreports.com/forum/r30908033-Bypass-Google-Fiber-Box-How-To

If it's not setup properly then your upload speeds get throttled to <=10mbps which is what I'm now getting even though all I did was upgrade to pfsense 2.4.0. What I've tried to fix the problem:

Different computer (no change)
Different speedtest server and services (no change)
Reverting to factory defaults after the upgrade (no change)
Disconnecting pfsense and reconnecting the Google Fiber router (works, upload speeds return to normal)
Add a WAN firewall rule to tag packets with the proper priority bit (no change, assuming this could actually be a workaround)
Try without vlan/priority and try other priorities (no change)

I know this worked before the upgrade and I'm kind of running out of ideas beyond a downgrade. I'm happy to offer any help I can to help debug/fix this, just tell me what to do. If 2.4 has stopped setting the priority of the vlan then I bet this is a bug that has an impact beyond just those using Google Fiber.

EDIT I'll reply or edit this post a bit later today with some additional data along the lines of what @jimp was asking for. - Done.

What hardware?

Netgate SG-4860
BIOS: coreboot, Version: ADI_RCCVE-01.00.00.12-nodebug
Intel(R) Atom(TM) CPU C2558 @ 2.40GHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active)

What NICs?

6x Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k

What packages are you using?

Only installed package is the default aws-wizard.

What does the status of the WAN/LAN NICs show

All interfaces are up, including vLANs

ifconfig output

?

igb1 - WAN interface


igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=6500bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwfilter,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether REDACTED
	hwaddr REDACTED
	inet6 fe80::208:a2ff:fe0b:b80d%igb1 prefixlen 64 scopeid 0x2 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
igb1_vlan2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>ether REDACTED
	inet6 fe80::208:a2ff:fe0b:b80d%igb1_vlan2 prefixlen 64 scopeid 0x10 
	inet REDACTED netmask 0xffffe000 broadcast REDACTED
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 2 vlanpcp: 3 parent interface: igb1
	groups: vlan</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwfilter,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast>

What sort of load is the firewall under when you run the test?

Zero, no other devices connected.

Any errors in the logs?

None that I can see, dmesg does not appear to have anything relevant either.

Anything else you can provide that might be relevant.

What else can I provide?

w0w

@opalmer:

Disconnecting pfsense and reconnecting the Google Fiber router (works, upload speeds return to normal)

Until reboot?

opalmer

Sorry that should have read, "Disconnecting from pfsense…":

Computer > pfsense > fiber jack > internet (capped to <=10mbps)

Computer > google router > fiber jack > internet (no cap)

opalmer

So after looking around a bit on the bug tracker, this appears to be related: https://redmine.pfsense.org/issues/7748

I don't have a packet capture right now to prove it myself but the behavior appears to be the same:

ifconfig shows the proper priority for pfsense 2.3 and 2.4 (which is what our logs in this post show)
Packet capture in the bug shows the priority bit not being set. Without the proper priority bit, Google Fiber throttles upload speeds (which is the behavior we're seeing)

So this potentially seems like a pretty bad regression that slipped into the 2.4 release. Is there any chance we could get this fixed in 2.4.1 (there's probably more people out there who will have problems with this bug)?

CobraGT2000

@opalmer:

I also have Google Fiber I have this exact same problem and I'm running a Netgate SG-4860. On pfsense 2.3 I was getting >800 up and down. I'm less concerned about the download speed and more concerned with the upload speed I'm now getting;
http://beta.speedtest.net/result/6702311671.png

In order to use pfsense to with Google Fiber you have to configure the WAN to operate on vlan2 with priority 3, see this guide: https://www.dslreports.com/forum/r30908033-Bypass-Google-Fiber-Box-How-To

If it's not setup properly then your upload speeds get throttled to <=10mbps which is what I'm now getting even though all I did was upgrade to pfsense 2.4.0. What I've tried to fix the problem:

Yeah –- I actually created that thread @ DSLREPORTS. Im not sure what the cause was, im sure it has to do with the vlan tagging. I went ahead and reverted back to 2.3.4 and all is well. I will play around with it a little this weekend and see if i can narrow down the cause.

@opalmer:

So after looking around a bit on the bug tracker, this appears to be related: https://redmine.pfsense.org/issues/7748

I don't have a packet capture right now to prove it myself but the behavior appears to be the same:

ifconfig shows the proper priority for pfsense 2.3 and 2.4 (which is what our logs in this post show)

Packet capture in the bug shows the priority bit not being set. Without the proper priority bit, Google Fiber throttles upload speeds (which is the behavior we're seeing)

So this potentially seems like a pretty bad regression that slipped into the 2.4 release. Is there any chance we could get this fixed in 2.4.1 (there's probably more people out there who will have problems with this bug)?

That appears to be the exact issue. Downstream was flaky, upstream was horrid and now that I think about it, it was that way before I worked with GF support on the vlan tags.

Also - looks like they updated the bug-tracker to advise would be fixed in 2.4.1

opalmer

Yeah I'm pretty sure it's a vlan priority tagging issue too. I replied to the bug and posted some output from tcpdump and looks like nothing appears to be tagged with priority 3. I'll try to downgrade this weekend which will probably fix this issue for me too in the short term.

By the way, thanks a lot for that thread over on dslreports. It eliminated a lot of guess work and it's certainly something I continue to point people to when they they run into issues with pfsense and GF :).

CobraGT2000

@opalmer:

By the way, thanks a lot for that thread over on dslreports. It eliminated a lot of guess work and it's certainly something I continue to point people to when they they run into issues with pfsense and GF :).

You're most welcome. I'm actually in the process of getting them to re-enable my account, my daughter was on there clicking around 1 day and poof my account was gone. Ive tried emailing and posting anyon in the forums, but no response. Hopefully they will reinstate my account. Ill be working on the GF TV stuff next, that's going to be a pain in the you know…

jimp

We'll try to get that fixed up for 2.4.1 which should be out in a week or so. Assuming there isn't something crazy complicated breaking it.

CobraGT2000

@jimp:

We'll try to get that fixed up for 2.4.1 which should be out in a week or so. Assuming there isn't something crazy complicated breaking it.

Sweet! Thank you so much!!

chudak

@opalmer:

So after looking around a bit on the bug tracker, this appears to be related: https://redmine.pfsense.org/issues/7748

I don't have a packet capture right now to prove it myself but the behavior appears to be the same:

ifconfig shows the proper priority for pfsense 2.3 and 2.4 (which is what our logs in this post show)

Packet capture in the bug shows the priority bit not being set. Without the proper priority bit, Google Fiber throttles upload speeds (which is the behavior we're seeing)

So this potentially seems like a pretty bad regression that slipped into the 2.4 release. Is there any chance we could get this fixed in 2.4.1 (there's probably more people out there who will have problems with this bug)?

I have Sonic 1gb fiber. @opalmer it sounds it will/may affect me as well?

Thx

Derelict

Does sonic rely on vlan priority tagging like google apparently does? If so, maybe. If not, maybe not.

opalmer

Does sonic rely on vlan priority tagging like google apparently does? If so, maybe. If not, maybe not.

This is the important bit. I know for Google Fiber it makes all the difference but I wouldn't know about other ISPs sorry. I'm normally really hesitant to blame speed issues on a router upgrade and the only reason I'm certain the upgrade is the culprit here is because the missing priority tagging has very clearly defined behavior in this case and I did a factory reset/tcpdump to test it too.

We'll try to get that fixed up for 2.4.1 which should be out in a week or so. Assuming there isn't something crazy complicated breaking it.

Awesome, thank you!!