Unable to upgrade due to HTTPS scanning (NOT USING A PROXY)

Gertjan

@brianjg said in Unable to upgrade due to HTTPS scanning.:

Redeploying and restoring is also not an option since the deployment also fails.

edit : after stepenw10 posted above : see the below as a alternative method with a guaranteed outcome :

Make a copy ('export') of you current pfSense config.
Re installing (a 10 minutes job ?) a clean 2.7.2 will always work.

Now, import your config, and I give you a 99% chance that "your done".

stephenw10

The Net Installer doesn't, yet, support fetching via a proxy.

brianjg

@stephenw10 We aren't using a proxy. Its the upstream gateway that does the HTTPs scanning. Already tried to rehash the certs, no change.

stephenw10

That is proxying the traffic. You will need to enter it manually.

brianjg

@stephenw10 We don't use a proxy server for traffic.

stephenw10

If you have something scanning inside https then that is proxying the traffic to do so. The pkg command used to update will only work if you tell it to use that proxy directly.

However on 2.7.0 you might also hit this: https://redmine.pfsense.org/issues/14609

brianjg

@stephenw10 OK lets me explain this better. The upstream GW inspects HTTPs traffic, it does not act as a proxy server. Entering the IP in the proxy section just breaks traffic. Every other device (including the current pfsense appliance) is routing traffic just fine. The only bit that is broken is the update option.

And since I have imported the root CA cert why is pfSense still rejecting the cert as invalid as per the original error message.

stephenw10

It may not use the word proxy but the only way to inspect https traffic is to proxy it. It's a transparent proxy but it probably still has a proxy port it can listen on directly.

Can you use fetch at the pfSense CLI directly?

[2.7.2-RELEASE][admin@t70.stevew.lan]/root: fetch https://www.pfsense.org
www.pfsense.org                                         25 kB   31 MBps    00s

Or curl?:

[2.7.2-RELEASE][admin@t70.stevew.lan]/root: curl https://www.pfsense.org
<!DOCTYPE html> <html lang="en"> <head> <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
        new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
        j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
        'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
....

brianjg

@stephenw10 Both commands fail.

curl https://www.pfsense.org

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

fetch https://www.pfsense.org

Certificate verification failed for /C=US/ST=Texas/L=Round Rock/O=Dell Technologies/OU=Cybersecurity/CN=Dell Technologies Root Certificate Authority 2018
34370682880:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/crypto/openssl/ssl/statem/statem_clnt.c:1916:
fetch: https://www.pfsense.org: Authentication error
[2.6.0-RELEASE][admin@vxrail-pfsense-gateway.crk.lab.emc.com]/root:

stephenw10

You may need to restart php or reboot if you have not done so since uploading the CA cert.

What version is your other pfSense device that is working via the proxy?

brianjg

@stephenw10 I've rebooted multiple times, same result.

This is the only appliance I have.

brianjg

@stephenw10 Also openssl is connecting fine,

openssl s_client pkg01-atx.netgate.com:443
CONNECTED(00000003)

stephenw10

@brianjg said in Unable to upgrade due to HTTPS scanning (NOT USING A PROXY):

Every other device (including the current pfsense appliance) is routing traffic just fine.

So by that you mean just routing? Not that the current pfSense appliance is able to connect?

Is this something new that has just stopped? You recently added the proxy/filter?

You can just install 2.7.2 clean and restore the config. The proxy setting is fixed there.

brianjg

@stephenw10 I've edited the /usr/local/share/pfSense/pkg/repos/pfSense-repo.conf from https to http only which allowed me to update to the latest 2.6 release.

I then changed the branch to 2.7 and exited the file again to try to upgrade but after running the command: # pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade It failed with,

pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
pkg-static: Repository pfSense-core missing. 'pkg update' required
pkg-static: No package database installed.  Nothing to do!
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating pfSense-core repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    2 KiB   1.8kB/s    00:01
Processing entries: 100%
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%  156 KiB 159.4kB/s    00:01
Processing entries:   0%
pkg-static: Newer FreeBSD version for package unzoo:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1400085
- running kernel: 1203500

pkg-static: repository pfSense contains packages for wrong OS version: FreeBSD:14:amd64
Processing entries: 100%
Unable to update repository pfSense
Error updating repositories!

Is there a way to get it to proceed?

brianjg

@stephenw10 said in Unable to upgrade due to HTTPS scanning (NOT USING A PROXY):

@brianjg said in Unable to upgrade due to HTTPS scanning (NOT USING A PROXY):

Every other device (including the current pfsense appliance) is routing traffic just fine.

So by that you mean just routing? Not that the current pfSense appliance is able to connect?

Is this something new that has just stopped? You recently added the proxy/filter?

You can just install 2.7.2 clean and restore the config. The proxy setting is fixed there.

Yes, just routing is working. I've inherited this setup from the previous admin. It must have worked at some stage in the past since packages are installed and I can't install them now.

Yes I can deploy a new CE release but the issue persists, so since I can't install packages this isn't a solution.

stephenw10

What should work here is to install 2.7.2 clean then set the proxy rather than trying to use it transparently. The proxy setting works correctly in 2.7.2.

Can you test that in a VM?

The pkg repos are https only from 2.7.0.

I would have expected the imported CA cert to work for command line connections.

brianjg

@stephenw10
For any other VM I have no need to define any proxy settings.

I know it won't be popular but I deployed OPNsense, imported my CA cert and was able to grab updates/packages without an issue. No proxy settings just defined the upstream gateway.

Since I have no visibility beyond my initial upstream GW I've no idea about which device is doing the HTTPs inspection so setting that IP isn't really an option or solution for me.

In my mind once I have imported the cert the HTTPS should just work. As a test I deployed the 2.7.2 release and also had the same issue once the cert was imported.

stephenw10

Ok how exactly did you import the CA cert?

Ah, you don't control the upstream proxy? Some other admin just gives you a CA cert to use and you're forced to trust they do the right thing with it?

stephenw10

Testing here an imported CA cert is used from the OS trust store by curl from the CLI in 2.7.2.

What error exactly do you see when you try that?

stephenw10

Or try running: pkg -d update

What error do you see there?

Part of that output should include:

* Couldn't find host pkg01-atx.netgate.com in the .netrc file; using defaults
* Hostname pkg01-atx.netgate.com was found in DNS cache
*   Trying 208.123.73.209:443...
* Connected to pkg01-atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: none
*  CApath: /etc/ssl/certs/
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

And the imported CA cert should be in /etc/ssl/certs if the trust store box was checked.