VERY slow System Update

Sergei_Shablovsky

@steveits How to trace to pfSense servers with updates? (if they have own standalone or AWS-based)

Do You know IP's or FQDN ?

P.S. Examples with NetGate config auto-backup servers acb.netgate.com

PING acb.netgate.com (208.123.73.78): 56 data bytes
64 bytes from 208.123.73.78: icmp_seq=0 ttl=49 time=160.389 ms
64 bytes from 208.123.73.78: icmp_seq=1 ttl=49 time=160.319 ms
64 bytes from 208.123.73.78: icmp_seq=2 ttl=49 time=160.464 ms

--- acb.netgate.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 160.319/160.390/160.464/0.059 ms

TRACEROUTE acb.netgate.com w/ Reverse Address Lookup

1  isp.net (AAA.BBB.CCC.DDD)  0.507 ms  0.490 ms  0.500 ms (our local ISP)
2  isp.ip.twelve99-cust.net (80.239.128.62)  13.979 ms  13.748 ms  13.718 ms (first hop for ISP in this trace)
3  bpt-b2-link.ip.twelve99.net (80.239.128.61)  21.163 ms  21.094 ms  21.093 ms
4  bpt-b3-link.ip.twelve99.net (62.115.134.132)  21.421 ms  21.571 ms  21.497 ms
5  ae13.edge2.Miami1.Level3.net (4.68.62.149)  20.993 ms  21.562 ms  21.120 ms
6  * * ae-2-3601.ear5.Dallas1.Level3.net (4.69.216.153)  153.938 ms
7  ZAYO-BANDWI.ear5.Dallas1.Level3.net (4.14.49.2)  194.289 ms  159.928 ms  159.913 ms
8  ae0.aus01-mls-dc-core-a.infr.zcolo.com (64.20.229.158)  166.024 ms  158.114 ms  158.131 ms
9  net66-219-34-194.static-customer.corenap.com (66.219.34.194)  158.056 ms  158.058 ms  158.044 ms
10  fw1-zcolo.netgate.com (208.123.73.4)  159.163 ms  159.127 ms  159.106 ms
11  acb.netgate.com (208.123.73.78)  160.306 ms  160.271 ms  160.214 ms

stephenw10

Use: https://files00.netgate.com/packages/ The master branch there is 2.6.

I'm not seeing any slowness with 2.6 snapshots here though.

Steve

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

Use: https://files00.netgate.com/packages/ The master branch there is 2.6.

I'm not seeing any slowness with 2.6 snapshots here though.

Steve

Thank You for answering.

TRACEROUTE - ICMP - files00.netgate.com w/ name resolve

1  isp.net (AAA.BBB.CCC.DDD)  0.512 ms  0.531 ms  0.474 ms
 2  isp.ip.twelve99-cust.net (AAA.BBB.CCC.DDD)  13.723 ms  13.682 ms  13.987 ms
 3  bpt-b2-link.ip.twelve99.net (80.239.128.61)  22.058 ms  21.952 ms  21.846 ms
 4  win-bb3-link.ip.twelve99.net (62.115.119.38)  130.719 ms  130.862 ms  130.449 ms
 5  ffm-bb1-link.ip.twelve99.net (62.115.137.202)  130.181 ms  130.281 ms  130.278 ms
 6  * prs-bb1-link.ip.twelve99.net (62.115.123.13)  127.522 ms *
 7  ldn-bb1-link.ip.twelve99.net (62.115.135.24)  54.606 ms  54.434 ms  54.787 ms
 8  nyk-bb2-link.ip.twelve99.net (62.115.113.20)  130.036 ms  130.075 ms  130.027 ms
 9  nyk-b3-link.ip.twelve99.net (62.115.139.151)  128.414 ms  128.288 ms  128.258 ms
10  nyi-ic345091-nyk-b3.ip.twelve99-cust.net (62.115.175.141)  124.150 ms  124.082 ms  123.097 ms
11  64.147.115.3 (64.147.115.3)  140.721 ms  140.689 ms  140.703 ms
12  cs80.cs99new.v.ewr.nyinternet.net (96.47.77.214)  140.709 ms  140.643 ms  140.665 ms
13  162.208.119.41 (162.208.119.41)  128.929 ms  128.651 ms  128.609 ms

TRACEROUTE - UDP - files00.netgate.com w/name resolve

1  * * *
 2  ISP.ip.twelve99-cust.net (AAA.BBB.CCC.DDD)  14.551 ms  14.348 ms  14.066 ms
 3  bpt-b2-link.ip.twelve99.net (80.239.128.61)  23.828 ms  23.872 ms  23.852 ms
 4  win-bb3-link.ip.twelve99.net (62.115.119.38)  129.599 ms  130.403 ms  132.482 ms
 5  ffm-bb1-link.ip.twelve99.net (62.115.137.202)  132.099 ms  130.269 ms  129.537 ms
 6  * prs-bb1-link.ip.twelve99.net (62.115.123.13)  127.047 ms *
 7  ldn-bb1-link.ip.twelve99.net (62.115.135.24)  56.178 ms  53.879 ms  54.535 ms
 8  nyk-bb2-link.ip.twelve99.net (62.115.113.20)  132.087 ms  132.434 ms  132.604 ms
 9  nyk-b3-link.ip.twelve99.net (62.115.139.151)  130.013 ms  127.570 ms  128.564 ms
10  nyi-ic345091-nyk-b3.ip.twelve99-cust.net (62.115.175.141)  124.704 ms  124.851 ms  124.232 ms
11  64.147.115.3 (64.147.115.3)  139.984 ms  141.058 ms  139.911 ms
12  cs80.cs99new.v.ewr.nyinternet.net (96.47.77.214)  143.101 ms  142.207 ms  142.255 ms
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *

Hmmm ... Please give me suggestion how to locate the problem?

stephenw10

What happens if you just try to fetch a file from there?:

[2.6.0-DEVELOPMENT][admin@26dev.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg
/dev/null                                               86 MB 5648 kBps    16s

Steve

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

What happens if you just try to fetch a file from there?:

[2.6.0-DEVELOPMENT][admin@26dev.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg
/dev/null                                               86 MB 5648 kBps    16s

Steve

fetch: https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211005.0100.pkg: Not Found
/dev/null                                                0  B    0  Bps    00s
fetch: 86: No such file or directory

stephenw10

Those files change every time there's a new build so you need to use one that currently exists.

[22.01-DEVELOPMENT][root@2100-2.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg
/dev/null                                               86 MB 6686 kBps    13s

stephenw10

Do you have IPv6 on that system?

I think I may have just hit this on a system that does. Can you test disabling IPv6 if so?

Steve

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

Those files change every time there's a new build so you need to use one that currently exists.

[22.01-DEVELOPMENT][root@2100-2.stevew.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg
/dev/null                                               86 MB 6686 kBps    13s

[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211007.1158.pkg
/dev/null                                       0% of   86 MB 7368  Bps 03h24m

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

Do you have IPv6 on that system?

I think I may have just hit this on a system that does. Can you test disabling IPv6 if so?

Steve

In System / Advanced / Networking ?

Yes, I try to turn OFF all about IPv6. Even restart, but nothing changed, still SLOWWW

P.S. In the location where this pfSense installed most of ISP unable to routing IPv6, clearly say "99,9% of ISPs in this area IPv4 only"

stephenw10

Mmm, not that then. It would fail to get that file with fetch too. I assume that firewall doesn't actually have IPv6 connectivity at all?

What happens if you run pkg-static -d up-date ?
It complete that in a few seconds. Does it stall anywhere?

Steve

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

Mmm, not that then. It would fail to get that file with fetch too. I assume that firewall doesn't actually have IPv6 connectivity at all?

What happens if you run pkg-static -d up-date ?

pkg-static -d update

[2.6.0-DEVELOPMENT][gamma-fw-dev.local.lan]/root: pkg-static -d update
DBG(1)[52270]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[52270]> PkgRepo: verifying update for pfSense-core
DBG(1)[52270]> Pkgrepo, begin update of '/root/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/meta.conf
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/meta.conf with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.pkg
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/packagesite.pkg with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.txz
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files00.netgate.com/packages/pfSense_master_amd64-core/packagesite.txz with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
DBG(1)[52270]> PkgRepo: verifying update for pfSense
DBG(1)[52270]> Pkgrepo, begin update of '/root/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/meta.conf
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/meta.conf with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.pkg
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.pkg with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
DBG(1)[52270]> Request to fetch pkg+https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.txz
DBG(1)[52270]> opening libfetch fetcher
DBG(1)[52270]> Fetch > libfetch: connecting
DBG(1)[52270]> Fetch: fetching from: https://files01.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.txz with opts "i"
DBG(1)[52270]> Fetch: fetcher chosen: https
pfSense repository is up to date.
All repositories are up to date.

It complete that in a few seconds. Does it stall anywhere?

No stall.
But no few sec
approx 12sec to complete

Thank You for help.

Sergei_Shablovsky

@stephenw10

iperf3 from the pfSense itself also good
(it was downloading some updates from office lan, so 67-75 Mbits/sec is ok in this case)

[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: iperf3 -c iperf.scottlinux.com
Connecting to host iperf.scottlinux.com, port 5201
[  5] local AAA.BBB.CCC.DDD port 17173 connected to 45.33.39.39 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   348 KBytes  2.85 Mbits/sec    0    122 KBytes
[  5]   1.00-2.00   sec  4.75 MBytes  39.9 Mbits/sec    0   3.00 MBytes
[  5]   2.00-3.00   sec  8.46 MBytes  71.0 Mbits/sec    0   3.00 MBytes
[  5]   3.00-4.00   sec  8.59 MBytes  72.1 Mbits/sec    0   3.00 MBytes
[  5]   4.00-5.00   sec  8.51 MBytes  71.4 Mbits/sec    0   3.00 MBytes
[  5]   5.00-6.00   sec  8.29 MBytes  69.5 Mbits/sec    0   3.00 MBytes
[  5]   6.00-7.00   sec  8.33 MBytes  69.9 Mbits/sec    0   3.00 MBytes
[  5]   7.00-8.00   sec  8.42 MBytes  70.7 Mbits/sec    0   3.00 MBytes
[  5]   8.00-9.00   sec  8.59 MBytes  72.1 Mbits/sec    0   3.00 MBytes
[  5]   9.00-10.00  sec  8.24 MBytes  69.1 Mbits/sec    0   3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  72.5 MBytes  60.8 Mbits/sec    0             sender
[  5]   0.00-10.20  sec  72.5 MBytes  59.7 Mbits/sec                  receiver

iperf Done.

stephenw10

So, to confirm, you do not have a routable IPv6 IP on that firewall?

Just to be sure try running:

fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg

Compare that with:

fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg

The issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.

Steve

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

So, to confirm, you do not have a routable IPv6 IP on that firewall?

Just to be sure try running:
fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg

[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg
/dev/null                                       0% of   86 MB 6851  Bps 03h35m

Compare that with:
fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg
The issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.

[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg
fetch: https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg: No route to host

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

The issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.

More that sure the repo are OK

stephenw10

Hmm, so something specifically to that host from where you are.

MTU problem?

You see the same in files00 and files01?

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

Hmm, so something specifically to that host from where you are.

MTU problem?

I do not make any modification in NIC settings.

You see the same in files00 and files01?

Yes, the same

stephenw10

@sergei_shablovsky said in VERY slow System Update:

I do not make any modification in NIC settings.

That doesn't mean something didn't change in the route though.

What do you see in a packet capture of the traffic? A lot of fragmentation perhaps?

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

Hmm, so something specifically to that host from where you are.

To this pfSense we have 2 WAN with static IPs from local ISP.

MTU problem?

I was a little play with this: 1500, 1350 - not solving my problem.

Sergei_Shablovsky

@stephenw10 said in VERY slow System Update:

@sergei_shablovsky said in VERY slow System Update:

I do not make any modification in NIC settings.

That doesn't mean something didn't change in the route though.

What do you see in a packet capture of the traffic? A lot of fragmentation perhaps?

But if fragmentation or something miss-working on a TCP level, iperf also could be affected. But iperf working as many days before - the same measurement.

Please point me with sample what You mean about “measuring level of fragmentation by packet capturing”.

Thank You for help. This issue slowly make me sad...