Netgate 3100 URL unknown

rcoleman-netgate

@michael_samer Try toggling your System>Update branches to see if it will re-load the repo details.

SteveITS

@michael_samer that, but also https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html#packages-netgate-com-has-no-a-aaaa-record

michael_samer

@rcoleman-netgate Did that (and waiting for about 5min, than switching back and waiting again) and it still looks for this non existing URL.

michael_samer

@SteveITS Hi Steve; by searching I found exactly that, but the searched name is not a DNS listed name so the proxy answers with http_503 service invalid. hard to deal with that.
Our parent proxy is an SOCKS5 compliant server (logon, filtering, ...).
I observed the newer "plus... URLs" since V23.01 when I upgraded this box. All older installations still calls only for the "usual" firmware and ews URLs and succeed.

michael_samer

Here's another funny constellation on this box:
Some Info (V23.05.1 is available) he could gather without using the "pfsense-plus*" site, while being stuck on 23.05 for the "main dish"

michael_samer

@SteveITS Hi Steve, today I asked the proxy team if they could see any further but no reply yet.
afaik the SRV call should be a srv://URL or srv+https://URL and not the https/tcp call it is doing. Maybe our local (squid on the box) is malforming the call?
Why is only the V23.x trapped there? The V22.x are still able to use this proxy and the "usual calls" to fetch their pkg repo.

SteveITS

@michael_samer 23.x uses a new dynamic package repo, per what I've read. Ryan or @stephenw10 may have more details on that. pfSense has to get to 23.01 to upgrade to later versions. Per the doc page I linked the SRV lookup returns the hostnames to connect to via HTTPS.

Did you go through https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html#upgrade-not-offered-library-errors already?

The brute force method would be to reinstall.

michael_samer

@SteveITS Hi, no I did not go thru the usual TS site yet, as the error seems more housed in the external.

The installation is two month old and was done with the recovery method V23.05. I'vre chosen this way because non of my test V23.01 could update since it could not get data thru the proxy since V23.x changes its procedure comapred to V22.x und before. With that findings I stopped all updates of our (about 70 boxes) so far until we found a solution.

As the Updates did never work because my proxy (squid then Bluecoat uplink proxy) could not answer the web call started by the V23.x series.
What puzzels me is the different "could not update info" (Latest base=23.05) info and the Branch 23.05.1 Info. A part is going through our proxy, but not the whole info.
While I could change the local (pfsense) squid to adjust, but cannot change parent proxy (=Bluecoat) it seems there's little

@stephenw10: is there a way around the SRV call; for whatever reasons the SRV is called as normal https call instead of and srv+https one (see above) and my bluecoat parent answers it with an htttp_503 error and leaving me in this loop.

stephenw10

Just seeing this but we are already looking at it via your open TAC ticket.

stephenw10

Are you in fact using two proxies here? Both Squid on the box itself and an upstream proxy?

What do you have set in the Proxy settings in System > Advanced > Misc? Localhost?

michael_samer

@stephenw10: yes, due to local restrictions I need to bundle the calls to one IP and pfsense GUI proxy couldn't handle our login procedure, while the very detailed squid could. The local box is asking 127.0.0.1 to check (which worked until the update from 22.05)

michael_samer

P.S.: and DNS Requests must be sent to the proxy as well, as our local DNS is pure intranet; this might be the whole point as pkg seems (by getting an NXDOMAIN answer) to tunnel directly to pfsense-plus instead of asking the proxy to resolve the SRV record. In my TAC call there's still the tcp dump log to be completely analyzed.

michael_samer

@stephenw10 : Hi Steve
my TAC ticket vanished (none is shown about that case) somehow and I never received further information about the case after I included the asked questions,

What happened to the case? TicketID was (afair) 1776782444

stephenw10

Huh, I'm sorry about that. Let me check....

But to summarize the issue appears to be that pkg is using the local Squid proxy correctly but that is sending the srv request incorrectly as a normal https call to the Bluecoat proxy which rejects it.

And this didn't happen before 23.0X.

SRV calls have been used by pkg for years so it seems like this is a regression in Squid. However the entire pkg system was changed in 23.01.

Is there any test you can do to rule out one of those components? Perhaps manually login to the upstream proxy and try to pull pkgs without Squid?

Steve

michael_samer

@stephenw10 said in Netgate 3100 URL unknown:

Huh, I'm sorry about that. Let me check....

But to summarize the issue appears to be that pkg is using the local Squid proxy correctly but that is sending the srv request incorrectly as a normal https call to the Bluecoat proxy which rejects it.

And this didn't happen before 23.0X.

SRV calls have been used by pkg for years so it seems like this is a regression in Squid. However the entire pkg system was changed in 23.01.

Is there any test you can do to rule out one of those components? Perhaps manually login to the upstream proxy and try to pull pkgs without Squid?

Steve

Hi Steve
that's exactly what I supposed as well. It might be the newer squid... but I rarely had any update issues with squid (which I use in many instances since ~2000 in a few very strange ways/configs)

If you tell me any curl/wget check I should check I can fill the rest to ask the proxy directly.

Good to know that SRV is not an issue as our proxy team did not answer to my questions.

As the V22.x and V21.x work (still) on the same proxy (=same machine), but no V22.05.1 and above it's surely a small thing which is done differently. Same happened in V21 to V22. when the pkg asked for the two google DNS servers at the script start, else it stopped working (so we had to build a workaround for that DNS search)=>works plain in INet connected situations with open DNS but not in restricted areas.

Cheers
Michael

stephenw10

Do you actually mean 22.05.1? That was never released as an upgrade only as a new install for 8200 and some 2100s.
However that was the first release that had the dynamic repo systems and still used the same packages (including Squid) as 22.05. So that seems like a big clue if so.

The most likely thing here is that the new dynamic repo pkg system is not respecting the system proxy configuration. However I'm unsure what you would see if that's the case. Would the upstream proxy see those requests at all?

When we were investigating this we did find a bug that might do that but it looked like in your setup that would result in the upstream proxy simply not seeing anything.

michael_samer

@stephenw10 yes I mean since 22.05.1 as our new 2100 modells have the same error with this config, while the older SG3100 with 22.05 and below work as expected.

So I guessed it's the same problem. Updated (V23.x) SG3100 when downgraded (reinstalled from stick) to V22.05 (the latest that really works) works with the same config.

We have about 60 SG3100, 10 SG2220 and now 20 SG2100, where the SG2220 works until V22.05; same goes for the 3100 as long as under V23.01 and all the SG2100 (only four used/tested and "reshelved until solved") experience the same "no packages/plugins, no update anymore".
The local squid sees the questions/fetches for the files but the steps are different. I included the squid logs from both (successful and unsuccessful) tasks when triggered in the TAC ticket. Someone even analyzed the tcpdump log from all traffic, but apart from DNS lookup failures there was nothing mentioned.

The last entry I provided was the constellation:
Inet <->Bluecoat/Proxy LAN<-> pfsense WAN without INet DNS (just local addresses resolvable) <->LAN1-4 = our testing enviroment

so basically the pfsense is used as a LAN - 2- LAN Router+NAT with only a proxy to fetch "usual" (only port 80+443 is allowed) http+https stuff.

If you use two stacked pfsenses while the stacked one has no DNS and only proxy uplink it should be reproducable.

I've a very limited possibility (Enterprise IT...) into what the bluecoat/Plink Proxy is doing/seeing. My best guess are the two different squid fetches.

Have you found out what happend to the ticket?

Cheers
Michael

michael_samer

@stephenw10 said in Netgate 3100 URL unknown:

Is there any test you can do to rule out one of those components? Perhaps manually login to the upstream proxy and try to pull pkgs without Squid?

If you sent me any URL to call I'm glad to test and post the squid and or (uplink) proxy reply.

michael_samer

@stephenw10 : Hi Steve; did you find the original call?

stephenw10

Hey, yes I was reviewing the ticket and there was definitely some confusion as to what the setup is. It's unusual!

So as I said we did find a bug that affects the configured proxy for pg fetches. However it looked like it couldn't be applying here because Squid is still seeing the requests.

Do you know what the expected route would be for traffic that isn't set explicitly via the proxy? Would it just fail?

I've attached here an experimental patch for the issue we did find. If you're able to test that it will at least rule that out as the cause. I've tested it here in 23.05.1 on a 3100.

local_proxy.patch

Steve