2.4.0 Upgrade Killed my Speeds

JKnott

Why is your WAN IP changing? Why would that affect the resolver?

kpa

Why is your WAN IP changing? Why would that affect the resolver?

The way it's done is that every time the WAN DHCP lease gets renewed it counts as a WAN reconnect regardless of whether the dynamic IP changes or not. This reconnect restarts the unbound resolver and can be a rather time costly operation on lower end machines.

I still don't understand why a 'unbound-control reload' isn't enough on simple renewal of the lease where the IP address doesn't change, it's like hitting it with a hammer when a screwdriver is needed.

mhab12

I've experienced the GUI slowdown caused by DNS issues in the past, but it appears the OP is advising that throughput is way down. Likely unrelated to any DNS service issues especially since they are on an i3 machine.

opalmer

I also have Google Fiber I have this exact same problem and I'm running a Netgate SG-4860. On pfsense 2.3 I was getting >800 up and down. I'm less concerned about the download speed and more concerned with the upload speed I'm now getting;
http://beta.speedtest.net/result/6702311671.png

In order to use pfsense to with Google Fiber you have to configure the WAN to operate on vlan2 with priority 3, see this guide: https://www.dslreports.com/forum/r30908033-Bypass-Google-Fiber-Box-How-To

If it's not setup properly then your upload speeds get throttled to <=10mbps which is what I'm now getting even though all I did was upgrade to pfsense 2.4.0. What I've tried to fix the problem:

Different computer (no change)
Different speedtest server and services (no change)
Reverting to factory defaults after the upgrade (no change)
Disconnecting pfsense and reconnecting the Google Fiber router (works, upload speeds return to normal)
Add a WAN firewall rule to tag packets with the proper priority bit (no change, assuming this could actually be a workaround)
Try without vlan/priority and try other priorities (no change)

I know this worked before the upgrade and I'm kind of running out of ideas beyond a downgrade. I'm happy to offer any help I can to help debug/fix this, just tell me what to do. If 2.4 has stopped setting the priority of the vlan then I bet this is a bug that has an impact beyond just those using Google Fiber.

EDIT I'll reply or edit this post a bit later today with some additional data along the lines of what @jimp was asking for. - Done.

What hardware?

Netgate SG-4860
BIOS: coreboot, Version: ADI_RCCVE-01.00.00.12-nodebug
Intel(R) Atom(TM) CPU C2558 @ 2.40GHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active)

What NICs?

6x Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k

What packages are you using?

Only installed package is the default aws-wizard.

What does the status of the WAN/LAN NICs show

All interfaces are up, including vLANs

ifconfig output

?

igb1 - WAN interface


igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=6500bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwfilter,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether REDACTED
	hwaddr REDACTED
	inet6 fe80::208:a2ff:fe0b:b80d%igb1 prefixlen 64 scopeid 0x2 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
igb1_vlan2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>ether REDACTED
	inet6 fe80::208:a2ff:fe0b:b80d%igb1_vlan2 prefixlen 64 scopeid 0x10 
	inet REDACTED netmask 0xffffe000 broadcast REDACTED
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 2 vlanpcp: 3 parent interface: igb1
	groups: vlan</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwfilter,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast>

What sort of load is the firewall under when you run the test?

Zero, no other devices connected.

Any errors in the logs?

None that I can see, dmesg does not appear to have anything relevant either.

Anything else you can provide that might be relevant.

What else can I provide?

w0w

@opalmer:

Disconnecting pfsense and reconnecting the Google Fiber router (works, upload speeds return to normal)

Until reboot?

opalmer

Sorry that should have read, "Disconnecting from pfsense…":

Computer > pfsense > fiber jack > internet (capped to <=10mbps)

Computer > google router > fiber jack > internet (no cap)

opalmer

So after looking around a bit on the bug tracker, this appears to be related: https://redmine.pfsense.org/issues/7748

I don't have a packet capture right now to prove it myself but the behavior appears to be the same:

ifconfig shows the proper priority for pfsense 2.3 and 2.4 (which is what our logs in this post show)
Packet capture in the bug shows the priority bit not being set. Without the proper priority bit, Google Fiber throttles upload speeds (which is the behavior we're seeing)

So this potentially seems like a pretty bad regression that slipped into the 2.4 release. Is there any chance we could get this fixed in 2.4.1 (there's probably more people out there who will have problems with this bug)?

CobraGT2000

@opalmer:

I also have Google Fiber I have this exact same problem and I'm running a Netgate SG-4860. On pfsense 2.3 I was getting >800 up and down. I'm less concerned about the download speed and more concerned with the upload speed I'm now getting;
http://beta.speedtest.net/result/6702311671.png

In order to use pfsense to with Google Fiber you have to configure the WAN to operate on vlan2 with priority 3, see this guide: https://www.dslreports.com/forum/r30908033-Bypass-Google-Fiber-Box-How-To

If it's not setup properly then your upload speeds get throttled to <=10mbps which is what I'm now getting even though all I did was upgrade to pfsense 2.4.0. What I've tried to fix the problem:

Yeah –- I actually created that thread @ DSLREPORTS. Im not sure what the cause was, im sure it has to do with the vlan tagging. I went ahead and reverted back to 2.3.4 and all is well. I will play around with it a little this weekend and see if i can narrow down the cause.

@opalmer:

So after looking around a bit on the bug tracker, this appears to be related: https://redmine.pfsense.org/issues/7748

I don't have a packet capture right now to prove it myself but the behavior appears to be the same:

ifconfig shows the proper priority for pfsense 2.3 and 2.4 (which is what our logs in this post show)

Packet capture in the bug shows the priority bit not being set. Without the proper priority bit, Google Fiber throttles upload speeds (which is the behavior we're seeing)

So this potentially seems like a pretty bad regression that slipped into the 2.4 release. Is there any chance we could get this fixed in 2.4.1 (there's probably more people out there who will have problems with this bug)?

That appears to be the exact issue. Downstream was flaky, upstream was horrid and now that I think about it, it was that way before I worked with GF support on the vlan tags.

Also - looks like they updated the bug-tracker to advise would be fixed in 2.4.1

opalmer

Yeah I'm pretty sure it's a vlan priority tagging issue too. I replied to the bug and posted some output from tcpdump and looks like nothing appears to be tagged with priority 3. I'll try to downgrade this weekend which will probably fix this issue for me too in the short term.

By the way, thanks a lot for that thread over on dslreports. It eliminated a lot of guess work and it's certainly something I continue to point people to when they they run into issues with pfsense and GF :).

CobraGT2000

@opalmer:

By the way, thanks a lot for that thread over on dslreports. It eliminated a lot of guess work and it's certainly something I continue to point people to when they they run into issues with pfsense and GF :).

You're most welcome. I'm actually in the process of getting them to re-enable my account, my daughter was on there clicking around 1 day and poof my account was gone. Ive tried emailing and posting anyon in the forums, but no response. Hopefully they will reinstate my account. Ill be working on the GF TV stuff next, that's going to be a pain in the you know…

jimp

We'll try to get that fixed up for 2.4.1 which should be out in a week or so. Assuming there isn't something crazy complicated breaking it.

CobraGT2000

@jimp:

We'll try to get that fixed up for 2.4.1 which should be out in a week or so. Assuming there isn't something crazy complicated breaking it.

Sweet! Thank you so much!!

chudak

@opalmer:

So after looking around a bit on the bug tracker, this appears to be related: https://redmine.pfsense.org/issues/7748

I don't have a packet capture right now to prove it myself but the behavior appears to be the same:

ifconfig shows the proper priority for pfsense 2.3 and 2.4 (which is what our logs in this post show)

Packet capture in the bug shows the priority bit not being set. Without the proper priority bit, Google Fiber throttles upload speeds (which is the behavior we're seeing)

So this potentially seems like a pretty bad regression that slipped into the 2.4 release. Is there any chance we could get this fixed in 2.4.1 (there's probably more people out there who will have problems with this bug)?

I have Sonic 1gb fiber. @opalmer it sounds it will/may affect me as well?

Thx

Derelict

Does sonic rely on vlan priority tagging like google apparently does? If so, maybe. If not, maybe not.

opalmer

Does sonic rely on vlan priority tagging like google apparently does? If so, maybe. If not, maybe not.

This is the important bit. I know for Google Fiber it makes all the difference but I wouldn't know about other ISPs sorry. I'm normally really hesitant to blame speed issues on a router upgrade and the only reason I'm certain the upgrade is the culprit here is because the missing priority tagging has very clearly defined behavior in this case and I did a factory reset/tcpdump to test it too.

We'll try to get that fixed up for 2.4.1 which should be out in a week or so. Assuming there isn't something crazy complicated breaking it.

Awesome, thank you!!

ChristianHaag

Similar case here…

We have two German Telekom DSL, which we run in parallel. After the upgrade from 2.3.4_1 to 2.4 the connection was terribly slow.

We do VLAN tagging in the DSL modem (as it did not work letting pfSense do the VLAN tagging at the time we set it up).

Nothing fancy in our setup...

Reverted back to 2.3.4_1 using VMware snapshot.

After reverting to the snapshot the connection was still slow. After a reboot all was fine, speed as before.

I'll wait for 2.4.1 and try again.

cjdw94

@CobraGT2000:

:(

I was on the 2.3.4_1 version before the 2.4.0 upgrade. Prior to the upgrade my Google Fiber stats were (.8ms ping with a rttd of .2ms) and a download upload of 950/950.

After the upgrade my pings are 300+ with an rttd of 500+, speeds are horrible at best, 200/10….....

Ive tried 9 different speed test sites, pings etc, im 100% its not my connection as it was tested prior to the upgrade.

Any ideas?

Just wanted to chime in and say that I've been banging my head against the wall for the past few days trying to confirm where this problem was coming from. I also have Google Fiber, and prior to 2.4.0, I was getting the down/up speeds that you mentioned (950/950). After 2.4.0, I'm getting like 350/20 (best case scenario). I'm not sure about pings, but speeds are definitely way lower. And like opalmer mentioned, the ~20 Mbps upload rate on all speed tests was the red flag that caused me concern. My family uses WiFi for most everything, so we rarely get to see maximum download speeds in wireless situations, but with Google Fiber, the upload speed has always been nearly matched with whatever download speed we see reported.

My 2.4.0 upgrade went relatively smoothly - first it mentioned that it failed (nearly instantly after trying), but then I just tried again once more and it upgraded without a hitch.

I thought maybe Google Fiber was having some intermittent connection/speed issues (there's been some roadwork/utility maintenance being performed on my street by various companies over the last couple weeks), so I figured I'd give it a day or two and see if it cleared up. No luck. Then I contacted Google just to see if they would volunteer any info (if they were having any issues, a shortage, etc.) They said that they were not aware of any present issues.

This morning (several days post-2.4.0), I finally had a free minute to hook up the Google Fiber Network Box. Speeds were restored to ~1 Gbps up/down. I decided to try and reinstall pfSense fresh, so I backed up my 2.4.0 config and completely reinstalled. Restored from my backup config. Same speed issues returned.

Anyway, thanks again for starting this thread - I'm glad to know that we finally have ruled out most other complications.

If it matters to anyone, I'm running on physical hardware (not a VMWare setup) -

AMD 6400k APU @ 3.9 ghz
MSI A78M-35 Military Class 4 motherboard
4 GB DDR3 PC3 12800 @ 1600 mhz
2x Intel Gigabit PCIe cards
1 physical HDD (I forget the specs, but just a standard platter drive)

Edit(s): Just clearing up some typos.

Kuthedk

This seems to also be affecting me on AT&T gigapower. pfSense box I’m getting 20Mbps down/102ish Mbps up. Directly from the AT&T box, I’m getting 950Mbps/940Mbps Down/Up respectively.

I do not have any VLans setup, so I don't think that is the problem.

The pfSense box hardware:
Gigabyte Built-In Intel Celeron J1900 (2.0 GHz) Dual LAN Mini ITX DDR3 1333 NA Motherboard GA-J1900N-D3V
Crucial 8GB Kit (4GBx2) DDR3 1333

jimp

There is a fix for the VLAN priority issue (when set on Interfaces > Assignments, VLAN tab entries) coming in the next round of snapshots. When you see a snapshot with today's date, it should be OK to test.

Nitrobeast

Question did you install zfs as the file system during install ?? zfs has been a pain for me over the years as the overhead is high to run it.