2.4.0 Upgrade Killed my Speeds



  • :(

    I was on the 2.3.4_1 version before the 2.4.0 upgrade. Prior to the upgrade my Google Fiber stats were (.8ms ping with a rttd of .2ms) and a download upload of 950/950.

    After the upgrade my pings are 300+ with an rttd of 500+, speeds are horrible at best, 200/10….....

    Ive tried 9 different speed test sites, pings etc, im 100% its not my connection as it was tested prior to the upgrade.

    Any ideas?


  • Rebel Alliance Developer Netgate

    There is not nearly enough information here to offer any advice.

    What hardware? What NICs? Any interface errors? What does the status of the WAN/LAN NICs show? ifconfig output? What packages are you using? What sort of load is the firewall under when you run the test? Any errors in the logs? Anything else you can provide that might be relevant.



  • Intel(R) Core(TM) i3 CPU 550 @ 3.20GHz
    4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
    AES-NI CPU Crypto: No

    2 intel Pro GB NIcs
    1 Netgear GB NIC

    No interface errors.

    Load of the firewall is less then 50% - even before at full GB download. Avg is 13-17%.

    with Google, there needs to be a vlan tag on the WAN for 1GB upload, only config that is not the normal.

    No system errors or anything out of the ordinary.





    ifconfig.txt



  • This may have been fixed now, but I've not heard anything to say it has!
    Are you using the "DNS resolver" - if you are try switching to the "DNS Forwarder".

    I had awful problems (om my home firewall) when my IP changed, since switching back to the Forwarder I've had no issues at all.

    You mention what the load is now, what was it before?
    Also what's the load on the box, i.e. if you go to the console and type uptime.

    It's worth disabling suricata and see if that makes a difference, to track the issue down.
    On my server I've got that on without any issues after a 2.4 upgrade.

    Limited info, so blindly suggesting ideas!



  • Are you using the "DNS resolver" - if you are try switching to the "DNS Forwarder".

    I don't see any bandwidth issues with the new version.  However, what does the DNS method have to do with it?  All DNS does is return an IP address for a host name.  How the address is obtained should have absolutely no effect on speed.



  • The dns resolver goes crazy when your wan ip changes causing huge load.
    On my sg1000 enough so the console can be unresponsive.



  • Why is your WAN IP changing?  Why would that affect the resolver?



  • @JKnott:

    Why is your WAN IP changing?  Why would that affect the resolver?

    The way it's done is that every time the WAN DHCP lease gets renewed it counts as a WAN reconnect regardless of whether the dynamic IP changes or not. This reconnect restarts the unbound resolver and can be a rather time costly operation on lower end machines.

    I still don't understand why a 'unbound-control reload' isn't enough on simple renewal of the lease where the IP address doesn't change, it's like hitting it with a hammer when a screwdriver is needed.



  • I've experienced the GUI slowdown caused by DNS issues in the past, but it appears the OP is advising that throughput is way down.  Likely unrelated to any DNS service issues especially since they are on an i3 machine.



  • I also have Google Fiber I have this exact same problem and I'm running a Netgate SG-4860. On pfsense 2.3 I was getting >800 up and down. I'm less concerned about the download speed and more concerned with the upload speed I'm now getting;
    http://beta.speedtest.net/result/6702311671.png

    In order to use pfsense to with Google Fiber you have to configure the WAN to operate on vlan2 with priority 3, see this guide: https://www.dslreports.com/forum/r30908033-Bypass-Google-Fiber-Box-How-To

    If it's not setup properly then your upload speeds get throttled to <=10mbps which is what I'm now getting even though all I did was upgrade to pfsense 2.4.0. What I've tried to fix the problem:

    • Different computer (no change)

    • Different speedtest server and services (no change)

    • Reverting to factory defaults after the upgrade (no change)

    • Disconnecting pfsense and reconnecting the Google Fiber router (works, upload speeds return to normal)

    • Add a WAN firewall rule to tag packets with the proper priority bit (no change, assuming this could actually be a workaround)

    • Try without vlan/priority and try other priorities (no change)

    I know this worked before the upgrade and I'm kind of running out of ideas beyond a downgrade. I'm happy to offer any help I can to help debug/fix this, just tell me what to do. If 2.4 has stopped setting the priority of the vlan then I bet this is a bug that has an impact beyond just those using Google Fiber.

    EDIT I'll reply or edit this post a bit later today with some additional data along the lines of what @jimp was asking for. - Done.

    What hardware?

    Netgate SG-4860
    BIOS: coreboot, Version: ADI_RCCVE-01.00.00.12-nodebug
    Intel(R) Atom(TM) CPU C2558 @ 2.40GHz
    4 CPUs: 1 package(s) x 4 core(s)
    AES-NI CPU Crypto: Yes (active)

    What NICs?

    6x Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k

    What packages are you using?

    Only installed package is the default aws-wizard.

    What does the status of the WAN/LAN NICs show

    All interfaces are up, including vLANs

    ifconfig output

    ?

    igb1 - WAN interface

    
    igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    	options=6500bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwfilter,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether REDACTED
    	hwaddr REDACTED
    	inet6 fe80::208:a2ff:fe0b:b80d%igb1 prefixlen 64 scopeid 0x2 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    igb1_vlan2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    	options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>ether REDACTED
    	inet6 fe80::208:a2ff:fe0b:b80d%igb1_vlan2 prefixlen 64 scopeid 0x10 
    	inet REDACTED netmask 0xffffe000 broadcast REDACTED
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    	vlan: 2 vlanpcp: 3 parent interface: igb1
    	groups: vlan</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwfilter,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast> 
    

    What sort of load is the firewall under when you run the test?

    Zero, no other devices connected.

    Any errors in the logs?

    None that I can see, dmesg does not appear to have anything relevant either.

    Anything else you can provide that might be relevant.

    What else can I provide?



  • @opalmer:

    • Disconnecting pfsense and reconnecting the Google Fiber router (works, upload speeds return to normal)

    Until reboot?



  • Sorry that should have read, "Disconnecting from pfsense…":

    Computer > pfsense > fiber jack > internet (capped to <=10mbps)

    Computer > google router > fiber jack > internet (no cap)



  • So after looking around a bit on the bug tracker, this appears to be related: https://redmine.pfsense.org/issues/7748

    I don't have a packet capture right now to prove it myself but the behavior appears to be the same:

    • ifconfig shows the proper priority for pfsense 2.3 and 2.4 (which is what our logs in this post show)

    • Packet capture in the bug shows the priority bit not being set. Without the proper priority bit, Google Fiber throttles upload speeds (which is the behavior we're seeing)

    So this potentially seems like a pretty bad regression that slipped into the 2.4 release. Is there any chance we could get this fixed in 2.4.1 (there's probably more people out there who will have problems with this bug)?



  • @opalmer:

    I also have Google Fiber I have this exact same problem and I'm running a Netgate SG-4860. On pfsense 2.3 I was getting >800 up and down. I'm less concerned about the download speed and more concerned with the upload speed I'm now getting;
    http://beta.speedtest.net/result/6702311671.png

    In order to use pfsense to with Google Fiber you have to configure the WAN to operate on vlan2 with priority 3, see this guide: https://www.dslreports.com/forum/r30908033-Bypass-Google-Fiber-Box-How-To

    If it's not setup properly then your upload speeds get throttled to <=10mbps which is what I'm now getting even though all I did was upgrade to pfsense 2.4.0. What I've tried to fix the problem:

    Yeah –- I actually created that thread @ DSLREPORTS. Im not sure what the cause was, im sure it has to do with the vlan tagging. I went ahead and reverted back to 2.3.4 and all is well. I will play around with it a little this weekend and see if i can narrow down the cause.

    @opalmer:

    So after looking around a bit on the bug tracker, this appears to be related: https://redmine.pfsense.org/issues/7748

    I don't have a packet capture right now to prove it myself but the behavior appears to be the same:

    • ifconfig shows the proper priority for pfsense 2.3 and 2.4 (which is what our logs in this post show)

    • Packet capture in the bug shows the priority bit not being set. Without the proper priority bit, Google Fiber throttles upload speeds (which is the behavior we're seeing)

    So this potentially seems like a pretty bad regression that slipped into the 2.4 release. Is there any chance we could get this fixed in 2.4.1 (there's probably more people out there who will have problems with this bug)?

    That appears to be the exact issue. Downstream was flaky, upstream was horrid and now that I think about it, it was that way before I worked with GF support on the vlan tags.

    Also - looks like they updated the bug-tracker to advise would be fixed in 2.4.1



  • Yeah I'm pretty sure it's a vlan priority tagging issue too. I replied to the bug and posted some output from tcpdump and looks like nothing appears to be tagged with priority 3. I'll try to downgrade this weekend which will probably fix this issue for me too in the short term.

    By the way, thanks a lot for that thread over on dslreports. It eliminated a lot of guess work and it's certainly something I continue to point people to when they they run into issues with pfsense and GF :).



  • @opalmer:

    By the way, thanks a lot for that thread over on dslreports. It eliminated a lot of guess work and it's certainly something I continue to point people to when they they run into issues with pfsense and GF :).

    You're most welcome. I'm actually in the process of getting them to re-enable my account, my daughter was on there clicking around 1 day and poof my account was gone. Ive tried emailing and posting anyon in the forums, but no response. Hopefully they will reinstate my account. Ill be working on the GF TV stuff next, that's going to be a pain in the you know…


  • Rebel Alliance Developer Netgate

    We'll try to get that fixed up for 2.4.1 which should be out in a week or so. Assuming there isn't something crazy complicated breaking it.



  • @jimp:

    We'll try to get that fixed up for 2.4.1 which should be out in a week or so. Assuming there isn't something crazy complicated breaking it.

    Sweet! Thank you so much!!



  • @opalmer:

    So after looking around a bit on the bug tracker, this appears to be related: https://redmine.pfsense.org/issues/7748

    I don't have a packet capture right now to prove it myself but the behavior appears to be the same:

    • ifconfig shows the proper priority for pfsense 2.3 and 2.4 (which is what our logs in this post show)

    • Packet capture in the bug shows the priority bit not being set. Without the proper priority bit, Google Fiber throttles upload speeds (which is the behavior we're seeing)

    So this potentially seems like a pretty bad regression that slipped into the 2.4 release. Is there any chance we could get this fixed in 2.4.1 (there's probably more people out there who will have problems with this bug)?

    I have Sonic 1gb fiber.  @opalmer  it sounds it will/may affect me as well?

    Thx


  • LAYER 8 Netgate

    Does sonic rely on vlan priority tagging like google apparently does? If so, maybe. If not, maybe not.



  • Does sonic rely on vlan priority tagging like google apparently does? If so, maybe. If not, maybe not.

    This is the important bit. I know for Google Fiber it makes all the difference but I wouldn't know about other ISPs sorry. I'm normally really hesitant to blame speed issues on a router upgrade and the only reason I'm certain the upgrade is the culprit here is because the missing priority tagging has very clearly defined behavior in this case and I did a factory reset/tcpdump to test it too.

    We'll try to get that fixed up for 2.4.1 which should be out in a week or so. Assuming there isn't something crazy complicated breaking it.

    Awesome, thank you!!



  • Similar case here…

    We have two German Telekom DSL, which we run in parallel. After the upgrade from 2.3.4_1 to 2.4 the connection was terribly slow.

    We do VLAN tagging in the DSL modem (as it did not work letting pfSense do the VLAN tagging at the time we set it up).

    Nothing fancy in our setup...

    Reverted back to 2.3.4_1 using VMware snapshot.

    After reverting to the snapshot the connection was still slow. After a reboot all was fine, speed as before.

    I'll wait for 2.4.1 and try again.



  • @CobraGT2000:

    :(

    I was on the 2.3.4_1 version before the 2.4.0 upgrade. Prior to the upgrade my Google Fiber stats were (.8ms ping with a rttd of .2ms) and a download upload of 950/950.

    After the upgrade my pings are 300+ with an rttd of 500+, speeds are horrible at best, 200/10….....

    Ive tried 9 different speed test sites, pings etc, im 100% its not my connection as it was tested prior to the upgrade.

    Any ideas?

    Just wanted to chime in and say that I've been banging my head against the wall for the past few days trying to confirm where this problem was coming from.  I also have Google Fiber, and prior to 2.4.0, I was getting the down/up speeds that you mentioned (950/950).  After 2.4.0, I'm getting like 350/20 (best case scenario).  I'm not sure about pings, but speeds are definitely way lower.  And like opalmer mentioned, the ~20 Mbps upload rate on all speed tests was the red flag that caused me concern.  My family uses WiFi for most everything, so we rarely get to see maximum download speeds in wireless situations, but with Google Fiber, the upload speed has always been nearly matched with whatever download speed we see reported.

    My 2.4.0 upgrade went relatively smoothly - first it mentioned that it failed (nearly instantly after trying), but then I just tried again once more and it upgraded without a hitch.

    I thought maybe Google Fiber was having some intermittent connection/speed issues (there's been some roadwork/utility maintenance being performed on my street by various companies over the last couple weeks), so I figured I'd give it a day or two and see if it cleared up.  No luck.  Then I contacted Google just to see if they would volunteer any info (if they were having any issues, a shortage, etc.)  They said that they were not aware of any present issues.

    This morning (several days post-2.4.0), I finally had a free minute to hook up the Google Fiber Network Box.  Speeds were restored to ~1 Gbps up/down.  I decided to try and reinstall pfSense fresh, so I backed up my 2.4.0 config and completely reinstalled.  Restored from my backup config.  Same speed issues returned.

    Anyway, thanks again for starting this thread - I'm glad to know that we finally have ruled out most other complications.

    If it matters to anyone, I'm running on physical hardware (not a VMWare setup) -

    AMD 6400k APU @ 3.9 ghz
    MSI A78M-35 Military Class 4 motherboard
    4 GB DDR3 PC3 12800 @ 1600 mhz
    2x Intel Gigabit PCIe cards
    1 physical HDD (I forget the specs, but just a standard platter drive)

    Edit(s):  Just clearing up some typos.



  • This seems to also be affecting me on AT&T gigapower. pfSense box I’m getting 20Mbps down/102ish Mbps up. Directly from the AT&T box, I’m getting 950Mbps/940Mbps Down/Up respectively.

    I do not have any VLans setup, so I don't think that is the problem.

    The pfSense box hardware:
    Gigabyte Built-In Intel Celeron J1900 (2.0 GHz) Dual LAN Mini ITX DDR3 1333 NA Motherboard GA-J1900N-D3V
    Crucial 8GB Kit (4GBx2) DDR3 1333


  • Rebel Alliance Developer Netgate

    There is a fix for the VLAN priority issue (when set on Interfaces > Assignments, VLAN tab entries) coming in the next round of snapshots. When you see a snapshot with today's date, it should be OK to test.



  • Question did you install zfs as the file system during install ?? zfs has been a pain for me over the years as the overhead is high to run it.



  • @jimp:

    There is a fix for the VLAN priority issue (when set on Interfaces > Assignments, VLAN tab entries) coming in the next round of snapshots. When you see a snapshot with today's date, it should be OK to test.

    Do you have an ETA when it will be available via 2.4.1 ?


  • Rebel Alliance Developer Netgate

    We have bumped 2.4.1 to 2.4.1-RC and we're running some internal tests on it, it should be up with the regular snapshots in the next few hours.



  • @jimp:

    We have bumped 2.4.1 to 2.4.1-RC and we're running some internal tests on it, it should be up with the regular snapshots in the next few hours.

    Thank you, so I should be able to upgrade 2.3.4-RELEASE-p1 => 2.4.1-RC I assume.


  • Rebel Alliance Developer Netgate

    If you set your update settings to "Next Major Version" from 2.3.4 then it should land you there. Be sure to set it back to "Stable" once you're on 2.4.1.



  • @chudak:

    I have Sonic 1gb fiber.  @opalmer  it sounds it will/may affect me as well?

    Thx

    Answering my own query - I see no speed slowness on Sonic 1gb fiber on 2.4.1 release.


Log in to reply