VERY slow System Update
-
@steveits said in VERY slow System Update:
@sergei_shablovsky If DNS isn't working then everything will be slow as every request has to time out.
Not look like this: as i wrote before all other working OK, media streaming, web sites surfing, TVstreams, chats, trackers, even torrents...
@sergei_shablovsky
Do have any packages?Yes, a lot of.
@sergei_shablovsky For instance if I install pfBlockerNG-devel unbound ends up stopped, I just have to start it. (as I understand it, it's a known issue but with pfSense not the package)
Yes, I have pfBlockerNG-devel but Unbound working.
The main question are How to locate the problem ?
-
Any news ?
-
@sergei_shablovsky you said “a lot” of packages. Per https://docs.netgate.com/pfsense/en/latest/releases/21-09.html
“ When upgrading to pfSense Plus 21.09 and later versions, the pfSense-upgrade process will forcefully reinstall all operating system packages and add-on packages to ensure a consistent state and package set. This mayincrease the time the upgrade will take to download and install.”
Presumably 2.6 will be the same?Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
When upgrading, allow 10-15 minutes to reboot, or more depending on packages, and device or disk speed.
Upvote 👍 helpful posts! -
@steveits said in VERY slow System Update:
@sergei_shablovsky you said “a lot” of packages. Per https://docs.netgate.com/pfsense/en/latest/releases/21-09.html
“ When upgrading to pfSense Plus 21.09 and later versions, the pfSense-upgrade process will forcefully reinstall all operating system packages and add-on packages to ensure a consistent state and package set. This mayincrease the time the upgrade will take to download and install.”
Presumably 2.6 will be the same?No, that happened each time when update to snapshot. And to additional to this I see that only a few numbers of packages (4-8) updated.
-
@steveits How to trace to pfSense servers with updates? (if they have own standalone or AWS-based)
Do You know IP's or FQDN ?
P.S. Examples with NetGate config auto-backup servers acb.netgate.com
PING acb.netgate.com (208.123.73.78): 56 data bytes 64 bytes from 208.123.73.78: icmp_seq=0 ttl=49 time=160.389 ms 64 bytes from 208.123.73.78: icmp_seq=1 ttl=49 time=160.319 ms 64 bytes from 208.123.73.78: icmp_seq=2 ttl=49 time=160.464 ms --- acb.netgate.com ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 160.319/160.390/160.464/0.059 msTRACEROUTE acb.netgate.com w/ Reverse Address Lookup
1 isp.net (AAA.BBB.CCC.DDD) 0.507 ms 0.490 ms 0.500 ms (our local ISP) 2 isp.ip.twelve99-cust.net (80.239.128.62) 13.979 ms 13.748 ms 13.718 ms (first hop for ISP in this trace) 3 bpt-b2-link.ip.twelve99.net (80.239.128.61) 21.163 ms 21.094 ms 21.093 ms 4 bpt-b3-link.ip.twelve99.net (62.115.134.132) 21.421 ms 21.571 ms 21.497 ms 5 ae13.edge2.Miami1.Level3.net (4.68.62.149) 20.993 ms 21.562 ms 21.120 ms 6 * * ae-2-3601.ear5.Dallas1.Level3.net (4.69.216.153) 153.938 ms 7 ZAYO-BANDWI.ear5.Dallas1.Level3.net (4.14.49.2) 194.289 ms 159.928 ms 159.913 ms 8 ae0.aus01-mls-dc-core-a.infr.zcolo.com (64.20.229.158) 166.024 ms 158.114 ms 158.131 ms 9 net66-219-34-194.static-customer.corenap.com (66.219.34.194) 158.056 ms 158.058 ms 158.044 ms 10 fw1-zcolo.netgate.com (208.123.73.4) 159.163 ms 159.127 ms 159.106 ms 11 acb.netgate.com (208.123.73.78) 160.306 ms 160.271 ms 160.214 ms -
Use:
https://files00.netgate.com/packages/The master branch there is 2.6.I'm not seeing any slowness with 2.6 snapshots here though.
Steve
-
@stephenw10 said in VERY slow System Update:
Use:
https://files00.netgate.com/packages/The master branch there is 2.6.I'm not seeing any slowness with 2.6 snapshots here though.
Steve
Thank You for answering.
TRACEROUTE - ICMP - files00.netgate.com w/ name resolve
1 isp.net (AAA.BBB.CCC.DDD) 0.512 ms 0.531 ms 0.474 ms 2 isp.ip.twelve99-cust.net (AAA.BBB.CCC.DDD) 13.723 ms 13.682 ms 13.987 ms 3 bpt-b2-link.ip.twelve99.net (80.239.128.61) 22.058 ms 21.952 ms 21.846 ms 4 win-bb3-link.ip.twelve99.net (62.115.119.38) 130.719 ms 130.862 ms 130.449 ms 5 ffm-bb1-link.ip.twelve99.net (62.115.137.202) 130.181 ms 130.281 ms 130.278 ms 6 * prs-bb1-link.ip.twelve99.net (62.115.123.13) 127.522 ms * 7 ldn-bb1-link.ip.twelve99.net (62.115.135.24) 54.606 ms 54.434 ms 54.787 ms 8 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 130.036 ms 130.075 ms 130.027 ms 9 nyk-b3-link.ip.twelve99.net (62.115.139.151) 128.414 ms 128.288 ms 128.258 ms 10 nyi-ic345091-nyk-b3.ip.twelve99-cust.net (62.115.175.141) 124.150 ms 124.082 ms 123.097 ms 11 64.147.115.3 (64.147.115.3) 140.721 ms 140.689 ms 140.703 ms 12 cs80.cs99new.v.ewr.nyinternet.net (96.47.77.214) 140.709 ms 140.643 ms 140.665 ms 13 162.208.119.41 (162.208.119.41) 128.929 ms 128.651 ms 128.609 msTRACEROUTE - UDP - files00.netgate.com w/name resolve
1 * * * 2 ISP.ip.twelve99-cust.net (AAA.BBB.CCC.DDD) 14.551 ms 14.348 ms 14.066 ms 3 bpt-b2-link.ip.twelve99.net (80.239.128.61) 23.828 ms 23.872 ms 23.852 ms 4 win-bb3-link.ip.twelve99.net (62.115.119.38) 129.599 ms 130.403 ms 132.482 ms 5 ffm-bb1-link.ip.twelve99.net (62.115.137.202) 132.099 ms 130.269 ms 129.537 ms 6 * prs-bb1-link.ip.twelve99.net (62.115.123.13) 127.047 ms * 7 ldn-bb1-link.ip.twelve99.net (62.115.135.24) 56.178 ms 53.879 ms 54.535 ms 8 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 132.087 ms 132.434 ms 132.604 ms 9 nyk-b3-link.ip.twelve99.net (62.115.139.151) 130.013 ms 127.570 ms 128.564 ms 10 nyi-ic345091-nyk-b3.ip.twelve99-cust.net (62.115.175.141) 124.704 ms 124.851 ms 124.232 ms 11 64.147.115.3 (64.147.115.3) 139.984 ms 141.058 ms 139.911 ms 12 cs80.cs99new.v.ewr.nyinternet.net (96.47.77.214) 143.101 ms 142.207 ms 142.255 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * *Hmmm ... Please give me suggestion how to locate the problem?
-
What happens if you just try to fetch a file from there?:
[2.6.0-DEVELOPMENT][admin@26dev.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg /dev/null 86 MB 5648 kBps 16sSteve
-
@stephenw10 said in VERY slow System Update:
What happens if you just try to fetch a file from there?:
[2.6.0-DEVELOPMENT][admin@26dev.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg /dev/null 86 MB 5648 kBps 16sSteve
fetch: https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg: Not Found /dev/null 0 B 0 Bps 00s fetch: 86: No such file or directory -
Those files change every time there's a new build so you need to use one that currently exists.
[22.01-DEVELOPMENT][root@2100-2.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg /dev/null 86 MB 6686 kBps 13s -
Do you have IPv6 on that system?
I think I may have just hit this on a system that does. Can you test disabling IPv6 if so?
Steve
-
@stephenw10 said in VERY slow System Update:
Those files change every time there's a new build so you need to use one that currently exists.
[22.01-DEVELOPMENT][root@2100-2.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg /dev/null 86 MB 6686 kBps 13s[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg /dev/null 0% of 86 MB 7368 Bps 03h24m -
@stephenw10 said in VERY slow System Update:
Do you have IPv6 on that system?
I think I may have just hit this on a system that does. Can you test disabling IPv6 if so?
Steve
In System / Advanced / Networking ?
Yes, I try to turn OFF all about IPv6. Even restart, but nothing changed, still SLOWWW
P.S. In the location where this pfSense installed most of ISP unable to routing IPv6, clearly say "99,9% of ISPs in this area IPv4 only"
-
Mmm, not that then. It would fail to get that file with fetch too. I assume that firewall doesn't actually have IPv6 connectivity at all?
What happens if you run
pkg-static -d up-date?
It complete that in a few seconds. Does it stall anywhere?Steve
-
@stephenw10 said in VERY slow System Update:
Mmm, not that then. It would fail to get that file with fetch too. I assume that firewall doesn't actually have IPv6 connectivity at all?
What happens if you run
pkg-static -d up-date?pkg-static -d update
[2.6.0-DEVELOPMENT][gamma-fw-dev.local.lan]/root: pkg-static -d update DBG(1)[52270]> pkg initialized Updating pfSense-core repository catalogue... DBG(1)[52270]> PkgRepo: verifying update for pfSense-core DBG(1)[52270]> Pkgrepo, begin update of '/root/var/db/pkg/repo-pfSense-core.sqlite' DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/meta.conf DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/meta.conf with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.pkg DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/packagesite.pkg with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.txz DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/packagesite.txz with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https pfSense-core repository is up to date. Updating pfSense repository catalogue... DBG(1)[52270]> PkgRepo: verifying update for pfSense DBG(1)[52270]> Pkgrepo, begin update of '/root/var/db/pkg/repo-pfSense.sqlite' DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/meta.conf DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/meta.conf with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.pkg DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.pkg with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.txz DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.txz with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https pfSense repository is up to date. All repositories are up to date.It complete that in a few seconds. Does it stall anywhere?
No stall.
But no few sec
approx 12sec to completeThank You for help.
-
iperf3 from the pfSense itself also good
(it was downloading some updates from office lan, so 67-75 Mbits/sec is ok in this case)[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: iperf3 -c iperf.scottlinux.com Connecting to host iperf.scottlinux.com, port 5201 [ 5] local AAA.BBB.CCC.DDD port 17173 connected to 45.33.39.39 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 348 KBytes 2.85 Mbits/sec 0 122 KBytes [ 5] 1.00-2.00 sec 4.75 MBytes 39.9 Mbits/sec 0 3.00 MBytes [ 5] 2.00-3.00 sec 8.46 MBytes 71.0 Mbits/sec 0 3.00 MBytes [ 5] 3.00-4.00 sec 8.59 MBytes 72.1 Mbits/sec 0 3.00 MBytes [ 5] 4.00-5.00 sec 8.51 MBytes 71.4 Mbits/sec 0 3.00 MBytes [ 5] 5.00-6.00 sec 8.29 MBytes 69.5 Mbits/sec 0 3.00 MBytes [ 5] 6.00-7.00 sec 8.33 MBytes 69.9 Mbits/sec 0 3.00 MBytes [ 5] 7.00-8.00 sec 8.42 MBytes 70.7 Mbits/sec 0 3.00 MBytes [ 5] 8.00-9.00 sec 8.59 MBytes 72.1 Mbits/sec 0 3.00 MBytes [ 5] 9.00-10.00 sec 8.24 MBytes 69.1 Mbits/sec 0 3.00 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 72.5 MBytes 60.8 Mbits/sec 0 sender [ 5] 0.00-10.20 sec 72.5 MBytes 59.7 Mbits/sec receiver iperf Done. -
So, to confirm, you do not have a routable IPv6 IP on that firewall?
Just to be sure try running:
fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkgCompare that with:
fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkgThe issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.Steve
-
@stephenw10 said in VERY slow System Update:
So, to confirm, you do not have a routable IPv6 IP on that firewall?
Just to be sure try running:
fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg /dev/null 0% of 86 MB 6851 Bps 03h35mCompare that with:
fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkgThe issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg fetch: https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg: No route to host -
@stephenw10 said in VERY slow System Update:
The issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.More that sure the repo are OK
-
Hmm, so something specifically to that host from where you are.
MTU problem?
You see the same in files00 and files01?
