Updates taking over an hour
-
Updates used to be pretty quick, but since the release of 26.03, especially on hardware like the SG-2100, it's taking over an hour just to "fetch" the packages. I've spent a ridiculous amount of time updating six of them so far over the past 4 nights. Higher end hardware like the NG4200 is only slightly better at around 30+ minutes. This used to take only like 2 minutes to download. Does anyone know what's going on? I don't want to spend three weeks upgrading firewalls when I used to do it in a night or two.
Also, I've removed all but basic packages (Nexus and System_Patches), rebooted prior to attempting the upgrade, and made sure I have over 50% free space on the internal drive, and this still happens. CPU use during fetching seems to spike up to 100%, but ranges down in to the 50% area, while only pulling around 400 kbps. Are the servers restricting traffic to an extreme?
-
@JVComputers I did several a week ago with no issues and upgraded one last night in like 5 steps from 23.05 or some such, in a bit over an hour.
Check if DNS is working before you start…Diagnostics> DNS Lookup. And IPv6 if not disabled.
-
@SteveITS I'll double check DNS is working, but I'm sure it is, as these are being used all day every day without any complaints. Plus, I can remote into workstations on those networks and can browse sites from those stations.
I know IPv6 is absolutely disabled on most, if not all these routers, as the ISPs in this area don't use IPv6 and none of the local subnets use it, either. Why would IPv6 need to be enabled?
-
@JVComputers if ip6 is enabled but broken, or pfSense dealing with dns timeouts, those are common reasons for these symptoms.
-
@SteveITS I see what you meant, now. I did a DNS lookup for netgate.com and, while a bit high, at 22 and 23 msec, it's not timing out or anything.
-
@JVComputers
FWIW, my 4200 updated to 26.03 in about its usual amount of time (5-ish minutes), and mine is also an IPv4-only environment. I concur with @SteveITS that your issue smells like a network configuration problem rather than something wrong with Netgate's servers or your router. Hard to diagnose on this amount of info though. -
@tgl So, what doesn't make sense is that I do this every month on 23 pfSense routers, all official hardware, I haven't changed anything globally that would affect all of them, and this is the only release with which I've ever had this issue, and it's happening on all of them. They're all at different sites and there are at least 3 different ISPs involved across the lot of them. So, it's hard for me to point the finger at anywhere other than Netgate's servers or something they're doing to limit traffic. But, if no one else is having these issues, I'd love to know where to start diagnosing issues, since nothing unusual is poking out of the logs from what I can tell.
If it's at all helpful, they're all being updated from version 25.11.1 to v26.03
-
@JVComputers Huh. As I said, I'm not seeing this, and there's not a flood of other people complaining here either. Maybe it's time for you to reach out to Netgate support.
-
Here’s something I experienced updating over Starlink first an 1100 cold spare, then a production 4200 and finally my second cold spare 1100:
-
The first 1100 done from the GUI took over an hour or so; the downloads were painfully slow.
-
The 4200 update using the GUI took about 45 minutes.
-
The last 1100 I decided to do I signed out of the GUI and used the console option 13 and it took about five minutes.
So, in my case, the GUI seemed to be the common factor in long updates. Packages installed or not seemed to make no significant difference.
Maybe it’s not an epidemic but you are definitely not the only one experiencing long updates that kill productivity. Ironically, those who experience this and have been the best customers buying many installations are punished the most.
I don’t know if you will attract any attention from Netgate. I hope so. You are a customer and customer pay their bills over the long haul. My comments about the slow updates seemed to not get a second thought by anyone as if it’s just how it goes.
I’d sure hate to be an MSP or corporate IT tech who has dozens of these that is going to lose hours and hours of productivity over this. When I did this work I was so overload over my whole career I would have resented unnecessarily losing valuable time to slow updates from a vendor that did not seem to value my time and money enough to pursue fixing it.
-
-
I assume this is during the pkg fetch phase of the update not whilst applying them or rebooting into the new version? If so there shouldn't be any significantly longer disconnection for users behind it.
But, yes, it shouldn't take that long. Are these instances ZFS or UFS?
-
@Mission-Ghost I read about your experience during my searching through, hoping to find others who experienced the same thing as I, and read about your potential solution. However, I'm not physically present during these updates; I do them via remote over OpenVPN, as I have been doing monthly for several years, so I don't have the opportunity to connect a console cable to each one of them. I even tried a different method by remotely accessing one of the workstations on the client's network through a different method, not using a VPN connection, and running the update through a locally attached machine, but with the same result.
-
@stephenw10 Correct, this is during the "Fetching..." phase of the package downloads. After the downloads eventually complete, the rest of the update process goes quickly, as normal, reboots are fine, and subsequent package downloads to re-install things like pfBlocker, Status_Totals, etc., download in a normal amount of time, typically only a few seconds. The only time this is happening, and consistently so, is during the download of the 26.03 update packages, themselves.
-
Hmm, interesting. Yes if you have a partial IPv6 setup that can cause problems if it's still preferred over IPv4 (the default). But that shouldn't have changed since 25.11.1. Unless perhaps your ISP started handing out v6 info in the mean time.
Otherwise I'd check for issues in the route perhaps? Given you're seeing it across several sites I'd look for commonality. All the same ISP for example. -
@stephenw10 Definitely not an IPv6 issue. ISP isn't handing out v6 and even if they were, have all the IPv6 Gateways disabled to avoid issues. I'll try a site with a different ISP tonight. So far, coincidentally, 7 of the upgrades I've done this run have been from the same ISP.
-
@stephenw10 in my cases, ZFS file system and IPv6 disabled. Download only slow..not the connection to each package…just the download part. Each……..dot…….takes……10, 20, 30 or more seconds……..to……go…..by.
-
@Mission-Ghost Yeah, mine will take many minutes. Especially with certain steps in the package downloads. I remember step 8, I think 51 and 52, and several others taking not seconds, but tens of minutes each.
-
Hmm, not seen any general reports of that. Or anything personally from here in the UK.
If you can replicate it check the traffic graphs in another browser tab while it's running. I expect to see very low throughput.
Try a pcap to see if you have massive fragmentation or packet loss.
-
@stephenw10 as mentioned early on in this thread:
CPU use during fetching seems to spike up to 100%, but ranges down in to the 50% area, while only pulling around 400 kbps
So, yes, I'm seeing very low throughput. I'll try a pcap tonight, as well.
-
@JVComputers but is it a constant 400k, or download, long pause, download, repeat? The latter is more of a connection issue, eg DNS fails, 2 second timeout per non working DNS server, and finally one works.
-
@SteveITS for my case, it seems unlikely to be DNS because of my three Netgate devices, and two identical 1100s, same configuration, same internet service, same network, same network jack used to update, one 1100 took an hour +/-, and one 1100 took five minutes.
CPU power and memory seem to be ruled out because my 4200 was the second of the three I updated and it took 45 minutes.
It appears it was the download...that is...the process after the [n/m] appeared while the . . . slowly crawled across the screen for minutes per package, sometimes minutes per dot.
DNS functions appeared to work perfectly on the network and the Netgate boxes before, during and after the update process. IPv6 is disabled on all three of my boxes.
I didn't report it because it's hard to know how fast it should take during any update cycle, until some people began to report it taking just a few (literally) minutes and @JVComputers reported the same experience I had.
Privacy Policy · Cookie Policy