VERY slow System Update
-
@stephenw10 said in VERY slow System Update:
So, to confirm, you do not have a routable IPv6 IP on that firewall?
Just to be sure try running:
fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -4 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg /dev/null 0% of 86 MB 6851 Bps 03h35mCompare that with:
fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkgThe issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.[2.6.0-DEVELOPMENT][root@gamma-fw-dev.local.lan]/root: fetch -6 -o /dev/null https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg fetch: https://files00.netgate.com/packages/pfSense_master_amd64-core/All/pfSense-kernel-debug-pfSense-2.6.0.a.20211008.0500.pkg: No route to host -
@stephenw10 said in VERY slow System Update:
The issue I hit yesterday was a local setting I had forgotten about.
As far as I know there are no issues with the repo.More that sure the repo are OK
-
Hmm, so something specifically to that host from where you are.
MTU problem?
You see the same in files00 and files01?
-
@stephenw10 said in VERY slow System Update:
Hmm, so something specifically to that host from where you are.
MTU problem?
I do not make any modification in NIC settings.
You see the same in files00 and files01?
Yes, the same
-
@sergei_shablovsky said in VERY slow System Update:
I do not make any modification in NIC settings.
That doesn't mean something didn't change in the route though.
What do you see in a packet capture of the traffic? A lot of fragmentation perhaps?
-
@stephenw10 said in VERY slow System Update:
Hmm, so something specifically to that host from where you are.
To this pfSense we have 2 WAN with static IPs from local ISP.
MTU problem?
I was a little play with this: 1500, 1350 - not solving my problem.
-
@stephenw10 said in VERY slow System Update:
@sergei_shablovsky said in VERY slow System Update:
I do not make any modification in NIC settings.
That doesn't mean something didn't change in the route though.
What do you see in a packet capture of the traffic? A lot of fragmentation perhaps?
But if fragmentation or something miss-working on a TCP level, iperf also could be affected. But iperf working as many days before - the same measurement.
Please point me with sample what You mean about “measuring level of fragmentation by packet capturing”.
Thank You for help. This issue slowly make me sad...
-
Indeed iperf would also be affected if it was going over the same route but that's unlikely. We ca say whatever part of the route that is shared does not have a problem.
Run a pcap for traffic to/from the update server. Look for fragmentation.
Steve
-
@stephenw10 said in VERY slow System Update:
Indeed iperf would also be affected if it was going over the same route but that's unlikely. We ca say whatever part of the route that is shared does not have a problem.
Run a pcap for traffic to/from the update server. Look for fragmentation.
Thank You, Steve for suggestions!
No fragmentation, checked at the first step.
BTW, when doing “pkg upgrade” from command line on local console, after downloading first packet the output from pkg looks like stopped. No, system working and no freeze, all other LANs working...
I even trying to “pkg unlock -a”, but this also no impact on situation...
-
So it sends and then never receives a reply? And doesn't just timeout?
That sounds more like something broken in the pkg system locally.
Steve
-
@stephenw10 said in VERY slow System Update:
So it sends and then never receives a reply? And doesn't just timeout?
Thank You for patience with this case, Steve.
Not looks like this.
When come into local shell, trying to make manually
pkg upgradeafter depositories checks, the packet downloading process starting, even first .pkg in Queue downloaded fast (at 70-90Mb/s), but after this first .pkg (or just from start of Queue), the download speed on screen indicated as
16,4kB/s 01:43:17ETA(this is for pfSense-kernel-pfSense-2.6.0.a.20211029.0500.pkg fetching for example)
and on a big .pkg size connection closed, may be from the NetGate updates server sideLet me remind, any other downloads or third party .pkg downloaded on full available bandwidth, I test this more than triple-twice.
That sounds more like something broken in the pkg system locally.
I also thinking about this, so try to pkg unlock -a, purging pkg cache....
But still unsuccessfully...[UPDATE]
Today I have a little bit more time to make some experiments. And found that the SLOW speed only for 2nd, 3rd, and down the list .pkg files, but FIRST FILE ALWAYS DOWNLOADING AT FULL AVAILABLE BANDWITH. (i.e. from 24 to 80MB/s).q—
CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
Help Ukraine to resist, save civilians people’s lives !
(Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.) -
Hmm, that really looks like a routing issue or an IPv6 issue. But I believe we already ruled that out?
Do you have a VPN or some other tunnel you could try to route that traffic across as a test?
Steve
-
@stephenw10
Today I have a little bit more time to make some experiments. And found that the SLOW speed only for 2nd, 3rd, and down the list .pkg files, but FIRST FILE ALWAYS DOWNLOADING AT FULL AVAILABLE BANDWITH. (i.e. from 24 to 80MB/s). -
@stephenw10 said in VERY slow System Update:
Hmm, that really looks like a routing issue or an IPv6 issue. But I believe we already ruled that out?
The routing - is the first that I thinking on. But if something wrong with routing **why this not impact on any other traffic from/to any other destination/source.
Do you have a VPN or some other tunnel you could try to route that traffic across as a test?
I'm a little bit stupid today, could You be so please to write in details what exactly you mean and how doing that? Thx a lot!!!
-
If there is something specifically in the route that traffic is taking that throttling it then routing if over a VPN will likely change that route and might allow full speed repeatedly. It would at least prove the routing issue theory.
I have an OpenVPN tunnel setup to an instance in GCP for that purpose. But almost anywhere would change the route to test it.
Steve
-
@stephenw10 said in VERY slow System Update:
If there is something specifically in the route that traffic is taking that throttling it then routing if over a VPN will likely change that route and might allow full speed repeatedly. It would at least prove the routing issue theory.
I have an OpenVPN tunnel setup to an instance in GCP for that purpose. But almost anywhere would change the route to test it.
Thank You for all help here, Stephen!
Task SOLVED, due the source of this sow pfSense update was in speed/control settings for NIC's interface: need to be changed from 100TX full-duplex to 100TX (mean 100TX half-duplex).
After this speed of update come from fixed 16,4k to 1,5-2,5M.I do not know why *the exactly pfSense updates are so slow and no other netflow affected”, but anyway, this certain issue closed and finding the source of problem come to another level, please see Errors In, but no Errors Out and No Collisions on both WAN.
-
@sergei_shablovsky said in [SOLVED] - VERY slow System Update:
[UPDATE]
Today I have a little bit more time to make some experiments. And found that the SLOW speed only for 2nd, 3rd, and down the list .pkg files, but FIRST FILE ALWAYS DOWNLOADING AT FULL AVAILABLE BANDWITH. (i.e. from 24 to 80MB/s).qNeed to note about this “full speed” I make wrong conclusion: the speed on screen are just measured from already readed cache, so extremely hi-speed to view possible, like 1,200Mb on 100M physical connection & port speed. ;)
But the real speed are with the same limitation 16,4kBs.I make some extra experiments on this server:
trying to download the 420Mb full pfSense install by fetchfetch -v -o /dev/null https://snapshots.netgate.com/amd64/pfSense_master/installer/pfSense-CE-memstick-2.6.0-DEVELOPMENT-amd64-latest.img.gzThe speed and ETA on screen was:
6770 Bps 18h08mI conclude this speed are max per one connect from Netgate hosting provider.
-
@sergei_shablovsky said in [SOLVED] - VERY slow System Update:
I conclude this speed are max per one connect from Netgate hosting provider.
Not true at all.. Maybe from your ISP and peering issue to there.. But its clearly not a "netgate" connection or server issue.
-
@johnpoz said in [SOLVED] - VERY slow System Update:
@sergei_shablovsky said in [SOLVED] - VERY slow System Update:
I conclude this speed are max per one connect from Netgate hosting provider.
Not true at all.. Maybe from your ISP and peering issue to there.. But its clearly not a "netgate" connection or server issue.
I also love Netgate ;)
And really thankful for their support the pfSense's User Community.But as I wrote before in this tread (and other see Errors In, but no Errors Out and No Collisions on both WAN), the iperf3 to different geographic servers (USA, UK, France, HonKong, Latvia, ...) in different regions give me 70-88Mb/s on 100M physical connection.
The SAME ISP and the SAME physical connection. -
@sergei_shablovsky said in [SOLVED] - VERY slow System Update:
the iperf3 to different geographic servers (USA, UK, France, HonKong, Latvia, ...)
How does that have anything to do with downloading from the netgate servers?
