VERY slow System Update

Sergei_Shablovsky

Hi pfSense Gurus!

Recently I see that pfSense (pfSense: 2.6.0.a, 2 x WAN) update in manual mode thru the System/Update/System Update are VERY slow (from 15-40sec -> up to 120 min) in fetching .pkg.
And sometime fetching not success

[1/2] Fetching pfSense-base-2.6.0.a.20211001.0100.pkg: ......client_loop: send disconnect: Broken pipe

Time for Updating repositories metadata - the same changes: from "immediately" -> to 4-6 sec.

Ping and traceroute from the firewall itself are looks like before this situation, all internal networks working ok without any problems and issues...

IDS/IPS, packet shaper - nothing of this impact on situation (i try to on/off and restart)
Or may be I lost something...

How to locate the problem ?
Is this problem linked to DNS resolver ?

Any comments or suggestions ?

P.S. Even I change manually ClientAliveInterval to 300 (in /etc/ssh/sshd_config), nothing going better.

SteveITS

@sergei_shablovsky If DNS isn't working then everything will be slow as every request has to time out. Do have any packages? For instance if I install pfBlockerNG-devel unbound ends up stopped, I just have to start it. (as I understand it, it's a known issue but with pfSense not the package)

Sergei_Shablovsky

@steveits said in VERY slow System Update:

@sergei_shablovsky If DNS isn't working then everything will be slow as every request has to time out.

Not look like this: as i wrote before all other working OK, media streaming, web sites surfing, TVstreams, chats, trackers, even torrents...

@sergei_shablovsky
Do have any packages?

Yes, a lot of.

@sergei_shablovsky For instance if I install pfBlockerNG-devel unbound ends up stopped, I just have to start it. (as I understand it, it's a known issue but with pfSense not the package)

Yes, I have pfBlockerNG-devel but Unbound working.

The main question are How to locate the problem ?

Sergei_Shablovsky

Any news ?

SteveITS

@sergei_shablovsky you said “a lot” of packages. Per https://docs.netgate.com/pfsense/en/latest/releases/21-09.html
“ When upgrading to pfSense Plus 21.09 and later versions, the pfSense-upgrade process will forcefully reinstall all operating system packages and add-on packages to ensure a consistent state and package set. This mayincrease the time the upgrade will take to download and install.”
Presumably 2.6 will be the same?

Sergei_Shablovsky

@steveits said in VERY slow System Update:

@sergei_shablovsky you said “a lot” of packages. Per https://docs.netgate.com/pfsense/en/latest/releases/21-09.html
“ When upgrading to pfSense Plus 21.09 and later versions, the pfSense-upgrade process will forcefully reinstall all operating system packages and add-on packages to ensure a consistent state and package set. This mayincrease the time the upgrade will take to download and install.”
Presumably 2.6 will be the same?

No, that happened each time when update to snapshot. And to additional to this I see that only a few numbers of packages (4-8) updated.

Sergei_Shablovsky

@steveits How to trace to pfSense servers with updates? (if they have own standalone or AWS-based)

Do You know IP's or FQDN ?

P.S. Examples with NetGate config auto-backup servers acb.netgate.com

PING acb.netgate.com (208.123.73.78): 56 data bytes
64 bytes from 208.123.73.78: icmp_seq=0 ttl=49 time=160.389 ms
64 bytes from 208.123.73.78: icmp_seq=1 ttl=49 time=160.319 ms
64 bytes from 208.123.73.78: icmp_seq=2 ttl=49 time=160.464 ms

--- acb.netgate.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 160.319/160.390/160.464/0.059 ms

TRACEROUTE acb.netgate.com w/ Reverse Address Lookup

1  isp.net (AAA.BBB.CCC.DDD)  0.507 ms  0.490 ms  0.500 ms (our local ISP)
2  isp.ip.twelve99-cust.net (80.239.128.62)  13.979 ms  13.748 ms  13.718 ms (first hop for ISP in this trace)
3  bpt-b2-link.ip.twelve99.net (80.239.128.61)  21.163 ms  21.094 ms  21.093 ms
4  bpt-b3-link.ip.twelve99.net (62.115.134.132)  21.421 ms  21.571 ms  21.497 ms
5  ae13.edge2.Miami1.Level3.net (4.68.62.149)  20.993 ms  21.562 ms  21.120 ms
6  * * ae-2-3601.ear5.Dallas1.Level3.net (4.69.216.153)  153.938 ms
7  ZAYO-BANDWI.ear5.Dallas1.Level3.net (4.14.49.2)  194.289 ms  159.928 ms  159.913 ms
8  ae0.aus01-mls-dc-core-a.infr.zcolo.com (64.20.229.158)  166.024 ms  158.114 ms  158.131 ms
9  net66-219-34-194.static-customer.corenap.com (66.219.34.194)  158.056 ms  158.058 ms  158.044 ms
10  fw1-zcolo.netgate.com (208.123.73.4)  159.163 ms  159.127 ms  159.106 ms
11  acb.netgate.com (208.123.73.78)  160.306 ms  160.271 ms  160.214 ms

stephenw10

Use: https://files00.netgate.com/packages/ The master branch there is 2.6.

I'm not seeing any slowness with 2.6 snapshots here though.

Steve

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

Use: https://files00.netgate.com/packages/ The master branch there is 2.6.

I'm not seeing any slowness with 2.6 snapshots here though.

Steve

Thank You for answering.

TRACEROUTE - ICMP - files00.netgate.com w/ name resolve

1  isp.net (AAA.BBB.CCC.DDD)  0.512 ms  0.531 ms  0.474 ms
 2  isp.ip.twelve99-cust.net (AAA.BBB.CCC.DDD)  13.723 ms  13.682 ms  13.987 ms
 3  bpt-b2-link.ip.twelve99.net (80.239.128.61)  22.058 ms  21.952 ms  21.846 ms
 4  win-bb3-link.ip.twelve99.net (62.115.119.38)  130.719 ms  130.862 ms  130.449 ms
 5  ffm-bb1-link.ip.twelve99.net (62.115.137.202)  130.181 ms  130.281 ms  130.278 ms
 6  * prs-bb1-link.ip.twelve99.net (62.115.123.13)  127.522 ms *
 7  ldn-bb1-link.ip.twelve99.net (62.115.135.24)  54.606 ms  54.434 ms  54.787 ms
 8  nyk-bb2-link.ip.twelve99.net (62.115.113.20)  130.036 ms  130.075 ms  130.027 ms
 9  nyk-b3-link.ip.twelve99.net (62.115.139.151)  128.414 ms  128.288 ms  128.258 ms
10  nyi-ic345091-nyk-b3.ip.twelve99-cust.net (62.115.175.141)  124.150 ms  124.082 ms  123.097 ms
11  64.147.115.3 (64.147.115.3)  140.721 ms  140.689 ms  140.703 ms
12  cs80.cs99new.v.ewr.nyinternet.net (96.47.77.214)  140.709 ms  140.643 ms  140.665 ms
13  162.208.119.41 (162.208.119.41)  128.929 ms  128.651 ms  128.609 ms

TRACEROUTE - UDP - files00.netgate.com w/name resolve

1  * * *
 2  ISP.ip.twelve99-cust.net (AAA.BBB.CCC.DDD)  14.551 ms  14.348 ms  14.066 ms
 3  bpt-b2-link.ip.twelve99.net (80.239.128.61)  23.828 ms  23.872 ms  23.852 ms
 4  win-bb3-link.ip.twelve99.net (62.115.119.38)  129.599 ms  130.403 ms  132.482 ms
 5  ffm-bb1-link.ip.twelve99.net (62.115.137.202)  132.099 ms  130.269 ms  129.537 ms
 6  * prs-bb1-link.ip.twelve99.net (62.115.123.13)  127.047 ms *
 7  ldn-bb1-link.ip.twelve99.net (62.115.135.24)  56.178 ms  53.879 ms  54.535 ms
 8  nyk-bb2-link.ip.twelve99.net (62.115.113.20)  132.087 ms  132.434 ms  132.604 ms
 9  nyk-b3-link.ip.twelve99.net (62.115.139.151)  130.013 ms  127.570 ms  128.564 ms
10  nyi-ic345091-nyk-b3.ip.twelve99-cust.net (62.115.175.141)  124.704 ms  124.851 ms  124.232 ms
11  64.147.115.3 (64.147.115.3)  139.984 ms  141.058 ms  139.911 ms
12  cs80.cs99new.v.ewr.nyinternet.net (96.47.77.214)  143.101 ms  142.207 ms  142.255 ms
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *

Hmmm ... Please give me suggestion how to locate the problem?

stephenw10

What happens if you just try to fetch a file from there?:

[2.6.0-DEVELOPMENT][admin@26dev.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg
/dev/null                                               86 MB 5648 kBps    16s

Steve

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

What happens if you just try to fetch a file from there?:

[2.6.0-DEVELOPMENT][admin@26dev.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg
/dev/null                                               86 MB 5648 kBps    16s

Steve

fetch: https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg: Not Found
/dev/null                                                0  B    0  Bps    00s
fetch: 86: No such file or directory

stephenw10

Those files change every time there's a new build so you need to use one that currently exists.

[22.01-DEVELOPMENT][root@2100-2.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg
/dev/null                                               86 MB 6686 kBps    13s

stephenw10

Do you have IPv6 on that system?

I think I may have just hit this on a system that does. Can you test disabling IPv6 if so?

Steve

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

Those files change every time there's a new build so you need to use one that currently exists.

[22.01-DEVELOPMENT][root@2100-2.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg
/dev/null                                               86 MB 6686 kBps    13s

[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg
/dev/null                                       0% of   86 MB 7368  Bps 03h24m

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

Do you have IPv6 on that system?

I think I may have just hit this on a system that does. Can you test disabling IPv6 if so?

Steve

In System / Advanced / Networking ?

Yes, I try to turn OFF all about IPv6. Even restart, but nothing changed, still SLOWWW

P.S. In the location where this pfSense installed most of ISP unable to routing IPv6, clearly say "99,9% of ISPs in this area IPv4 only"

stephenw10

Mmm, not that then. It would fail to get that file with fetch too. I assume that firewall doesn't actually have IPv6 connectivity at all?

What happens if you run pkg-static -d up-date ?
It complete that in a few seconds. Does it stall anywhere?

Steve

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

Mmm, not that then. It would fail to get that file with fetch too. I assume that firewall doesn't actually have IPv6 connectivity at all?

What happens if you run pkg-static -d up-date ?

pkg-static -d update

[2.6.0-DEVELOPMENT][gamma-fw-dev.local.lan]/root: pkg-static -d update
DBG(1)[52270]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[52270]> PkgRepo: verifying update for pfSense-core
DBG(1)[52270]> Pkgrepo, begin update of '/root/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/meta.conf
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/meta.conf with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.pkg
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/packagesite.pkg with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.txz
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/packagesite.txz with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
DBG(1)[52270]> PkgRepo: verifying update for pfSense
DBG(1)[52270]> Pkgrepo, begin update of '/root/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/meta.conf
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/meta.conf with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.pkg
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.pkg with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.txz
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.txz with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
pfSense repository is up to date.
All repositories are up to date.

It complete that in a few seconds. Does it stall anywhere?

No stall.
But no few sec
approx 12sec to complete

Thank You for help.

Sergei_Shablovsky

@stephenw10

iperf3 from the pfSense itself also good
(it was downloading some updates from office lan, so 67-75 Mbits/sec is ok in this case)

[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: iperf3 -c iperf.scottlinux.com
Connecting to host iperf.scottlinux.com, port 5201
[  5] local AAA.BBB.CCC.DDD port 17173 connected to 45.33.39.39 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   348 KBytes  2.85 Mbits/sec    0    122 KBytes
[  5]   1.00-2.00   sec  4.75 MBytes  39.9 Mbits/sec    0   3.00 MBytes
[  5]   2.00-3.00   sec  8.46 MBytes  71.0 Mbits/sec    0   3.00 MBytes
[  5]   3.00-4.00   sec  8.59 MBytes  72.1 Mbits/sec    0   3.00 MBytes
[  5]   4.00-5.00   sec  8.51 MBytes  71.4 Mbits/sec    0   3.00 MBytes
[  5]   5.00-6.00   sec  8.29 MBytes  69.5 Mbits/sec    0   3.00 MBytes
[  5]   6.00-7.00   sec  8.33 MBytes  69.9 Mbits/sec    0   3.00 MBytes
[  5]   7.00-8.00   sec  8.42 MBytes  70.7 Mbits/sec    0   3.00 MBytes
[  5]   8.00-9.00   sec  8.59 MBytes  72.1 Mbits/sec    0   3.00 MBytes
[  5]   9.00-10.00  sec  8.24 MBytes  69.1 Mbits/sec    0   3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  72.5 MBytes  60.8 Mbits/sec    0             sender
[  5]   0.00-10.20  sec  72.5 MBytes  59.7 Mbits/sec                  receiver

iperf Done.

stephenw10

So, to confirm, you do not have a routable IPv6 IP on that firewall?

Just to be sure try running:

fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg

Compare that with:

fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg

The issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.

Steve

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

So, to confirm, you do not have a routable IPv6 IP on that firewall?

Just to be sure try running:
fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg

[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg
/dev/null                                       0% of   86 MB 6851  Bps 03h35m

Compare that with:
fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg
The issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.

[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg
fetch: https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg: No route to host