VERY slow System Update
-
What happens if you just try to fetch a file from there?:
[2.6.0-DEVELOPMENT][admin@26dev.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg /dev/null 86 MB 5648 kBps 16sSteve
-
@stephenw10 said in VERY slow System Update:
What happens if you just try to fetch a file from there?:
[2.6.0-DEVELOPMENT][admin@26dev.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg /dev/null 86 MB 5648 kBps 16sSteve
fetch: https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg: Not Found /dev/null 0 B 0 Bps 00s fetch: 86: No such file or directory -
Those files change every time there's a new build so you need to use one that currently exists.
[22.01-DEVELOPMENT][root@2100-2.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg /dev/null 86 MB 6686 kBps 13s -
Do you have IPv6 on that system?
I think I may have just hit this on a system that does. Can you test disabling IPv6 if so?
Steve
-
@stephenw10 said in VERY slow System Update:
Those files change every time there's a new build so you need to use one that currently exists.
[22.01-DEVELOPMENT][root@2100-2.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg /dev/null 86 MB 6686 kBps 13s[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg /dev/null 0% of 86 MB 7368 Bps 03h24m -
@stephenw10 said in VERY slow System Update:
Do you have IPv6 on that system?
I think I may have just hit this on a system that does. Can you test disabling IPv6 if so?
Steve
In System / Advanced / Networking ?
Yes, I try to turn OFF all about IPv6. Even restart, but nothing changed, still SLOWWW
P.S. In the location where this pfSense installed most of ISP unable to routing IPv6, clearly say "99,9% of ISPs in this area IPv4 only"
-
Mmm, not that then. It would fail to get that file with fetch too. I assume that firewall doesn't actually have IPv6 connectivity at all?
What happens if you run
pkg-static -d up-date?
It complete that in a few seconds. Does it stall anywhere?Steve
-
@stephenw10 said in VERY slow System Update:
Mmm, not that then. It would fail to get that file with fetch too. I assume that firewall doesn't actually have IPv6 connectivity at all?
What happens if you run
pkg-static -d up-date?pkg-static -d update
[2.6.0-DEVELOPMENT][gamma-fw-dev.local.lan]/root: pkg-static -d update DBG(1)[52270]> pkg initialized Updating pfSense-core repository catalogue... DBG(1)[52270]> PkgRepo: verifying update for pfSense-core DBG(1)[52270]> Pkgrepo, begin update of '/root/var/db/pkg/repo-pfSense-core.sqlite' DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/meta.conf DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/meta.conf with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.pkg DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/packagesite.pkg with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.txz DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/packagesite.txz with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https pfSense-core repository is up to date. Updating pfSense repository catalogue... DBG(1)[52270]> PkgRepo: verifying update for pfSense DBG(1)[52270]> Pkgrepo, begin update of '/root/var/db/pkg/repo-pfSense.sqlite' DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/meta.conf DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/meta.conf with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.pkg DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.pkg with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.txz DBG(1)[52270]> opening libfetch fetcher DBG(1)[52270]> Fetch > libfetch: connecting DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.txz with opts "i" DBG(1)[52270]> Fetch: fetcher chosen: https pfSense repository is up to date. All repositories are up to date.It complete that in a few seconds. Does it stall anywhere?
No stall.
But no few sec
approx 12sec to completeThank You for help.
-
iperf3 from the pfSense itself also good
(it was downloading some updates from office lan, so 67-75 Mbits/sec is ok in this case)[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: iperf3 -c iperf.scottlinux.com Connecting to host iperf.scottlinux.com, port 5201 [ 5] local AAA.BBB.CCC.DDD port 17173 connected to 45.33.39.39 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 348 KBytes 2.85 Mbits/sec 0 122 KBytes [ 5] 1.00-2.00 sec 4.75 MBytes 39.9 Mbits/sec 0 3.00 MBytes [ 5] 2.00-3.00 sec 8.46 MBytes 71.0 Mbits/sec 0 3.00 MBytes [ 5] 3.00-4.00 sec 8.59 MBytes 72.1 Mbits/sec 0 3.00 MBytes [ 5] 4.00-5.00 sec 8.51 MBytes 71.4 Mbits/sec 0 3.00 MBytes [ 5] 5.00-6.00 sec 8.29 MBytes 69.5 Mbits/sec 0 3.00 MBytes [ 5] 6.00-7.00 sec 8.33 MBytes 69.9 Mbits/sec 0 3.00 MBytes [ 5] 7.00-8.00 sec 8.42 MBytes 70.7 Mbits/sec 0 3.00 MBytes [ 5] 8.00-9.00 sec 8.59 MBytes 72.1 Mbits/sec 0 3.00 MBytes [ 5] 9.00-10.00 sec 8.24 MBytes 69.1 Mbits/sec 0 3.00 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 72.5 MBytes 60.8 Mbits/sec 0 sender [ 5] 0.00-10.20 sec 72.5 MBytes 59.7 Mbits/sec receiver iperf Done. -
So, to confirm, you do not have a routable IPv6 IP on that firewall?
Just to be sure try running:
fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkgCompare that with:
fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkgThe issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.Steve
-
@stephenw10 said in VERY slow System Update:
So, to confirm, you do not have a routable IPv6 IP on that firewall?
Just to be sure try running:
fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg /dev/null 0% of 86 MB 6851 Bps 03h35mCompare that with:
fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkgThe issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg fetch: https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg: No route to host -
@stephenw10 said in VERY slow System Update:
The issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.More that sure the repo are OK
-
Hmm, so something specifically to that host from where you are.
MTU problem?
You see the same in files00 and files01?
-
@stephenw10 said in VERY slow System Update:
Hmm, so something specifically to that host from where you are.
MTU problem?
I do not make any modification in NIC settings.
You see the same in files00 and files01?
Yes, the same
-
@sergei_shablovsky said in VERY slow System Update:
I do not make any modification in NIC settings.
That doesn't mean something didn't change in the route though.
What do you see in a packet capture of the traffic? A lot of fragmentation perhaps?
-
@stephenw10 said in VERY slow System Update:
Hmm, so something specifically to that host from where you are.
To this pfSense we have 2 WAN with static IPs from local ISP.
MTU problem?
I was a little play with this: 1500, 1350 - not solving my problem.
-
@stephenw10 said in VERY slow System Update:
@sergei_shablovsky said in VERY slow System Update:
I do not make any modification in NIC settings.
That doesn't mean something didn't change in the route though.
What do you see in a packet capture of the traffic? A lot of fragmentation perhaps?
But if fragmentation or something miss-working on a TCP level, iperf also could be affected. But iperf working as many days before - the same measurement.
Please point me with sample what You mean about “measuring level of fragmentation by packet capturing”.
Thank You for help. This issue slowly make me sad...
-
Indeed iperf would also be affected if it was going over the same route but that's unlikely. We ca say whatever part of the route that is shared does not have a problem.
Run a pcap for traffic to/from the update server. Look for fragmentation.
Steve
-
@stephenw10 said in VERY slow System Update:
Indeed iperf would also be affected if it was going over the same route but that's unlikely. We ca say whatever part of the route that is shared does not have a problem.
Run a pcap for traffic to/from the update server. Look for fragmentation.
Thank You, Steve for suggestions!
No fragmentation, checked at the first step.
BTW, when doing “pkg upgrade” from command line on local console, after downloading first packet the output from pkg looks like stopped. No, system working and no freeze, all other LANs working...
I even trying to “pkg unlock -a”, but this also no impact on situation...
-
So it sends and then never receives a reply? And doesn't just timeout?
That sounds more like something broken in the pkg system locally.
Steve
