Unable to upgrade due to HTTPS scanning (NOT USING A PROXY)

brianjg · Jun 25, 2024, 12:18 PM

When I try to upgrade via the console I receive the following error message.

Enter an option: 13

pfSense-repoc-static: no package 'name'
pfSense-repoc-static: no pfSense packages installed
Certificate verification failed for /C=US/ST=Texas/L=Round Rock/O=Dell Technologies/OU=Cybersecurity/CN=Dell Technologies Root Certificate Authority 2018
34368225280:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-RELENG_2_7_0/crypto/openssl/ssl/statem/statem_clnt.c:1921:
pfSense-repoc-static: failed to fetch the repo data
failed to read the repo data.
failed to update the repository settings!!!
failed to update the repository settings!!!

I've imported our root ca and checked the box to import it to the OS. I've also tried to rehash the certs.

Redeploying and restoring is also not an option since the deployment also fails.

stephenw10 · Jun 25, 2024, 12:15 PM

You have to enter the proxy IP and port directly in System > Advanced > Misc to use that for updates.

Additionally coming from 2.7.0 you will probably need to run certctl rehash manually at the CLI once pfSense can see the update repo.

Steve

Gertjan · Jun 25, 2024, 12:16 PM

@brianjg said in Unable to upgrade due to HTTPS scanning.:

Redeploying and restoring is also not an option since the deployment also fails.

edit : after stepenw10 posted above : see the below as a alternative method with a guaranteed outcome :

Make a copy ('export') of you current pfSense config.
Re installing (a 10 minutes job ?) a clean 2.7.2 will always work.

Now, import your config, and I give you a 99% chance that "your done".

stephenw10 · Jun 25, 2024, 12:16 PM

The Net Installer doesn't, yet, support fetching via a proxy.

brianjg · Jun 25, 2024, 12:17 PM

@stephenw10 We aren't using a proxy. Its the upstream gateway that does the HTTPs scanning. Already tried to rehash the certs, no change.

stephenw10 · Jun 25, 2024, 12:19 PM

That is proxying the traffic. You will need to enter it manually.

brianjg · Jun 25, 2024, 12:22 PM

@stephenw10 We don't use a proxy server for traffic.

stephenw10 · Jun 25, 2024, 12:26 PM

If you have something scanning inside https then that is proxying the traffic to do so. The pkg command used to update will only work if you tell it to use that proxy directly.

However on 2.7.0 you might also hit this: https://redmine.pfsense.org/issues/14609

brianjg · Jun 25, 2024, 2:57 PM

@stephenw10 OK lets me explain this better. The upstream GW inspects HTTPs traffic, it does not act as a proxy server. Entering the IP in the proxy section just breaks traffic. Every other device (including the current pfsense appliance) is routing traffic just fine. The only bit that is broken is the update option.

And since I have imported the root CA cert why is pfSense still rejecting the cert as invalid as per the original error message.

stephenw10 · Jun 25, 2024, 1:08 PM

It may not use the word proxy but the only way to inspect https traffic is to proxy it. It's a transparent proxy but it probably still has a proxy port it can listen on directly.

Can you use fetch at the pfSense CLI directly?

[2.7.2-RELEASE][admin@t70.stevew.lan]/root: fetch https://www.pfsense.org
www.pfsense.org                                         25 kB   31 MBps    00s

Or curl?:

[2.7.2-RELEASE][admin@t70.stevew.lan]/root: curl https://www.pfsense.org
<!DOCTYPE html> <html lang="en"> <head> <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
        new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
        j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
        'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
....

brianjg · Jun 25, 2024, 1:08 PM

@stephenw10 Both commands fail.

curl https://www.pfsense.org

curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

fetch https://www.pfsense.org

Certificate verification failed for /C=US/ST=Texas/L=Round Rock/O=Dell Technologies/OU=Cybersecurity/CN=Dell Technologies Root Certificate Authority 2018
34370682880:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-img-build/BUILD_NODE/amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/sources/FreeBSD-src-RELENG_2_6_0/crypto/openssl/ssl/statem/statem_clnt.c:1916:
fetch: https://www.pfsense.org: Authentication error
[2.6.0-RELEASE][admin@vxrail-pfsense-gateway.crk.lab.emc.com]/root:

stephenw10 · Jun 25, 2024, 2:04 PM

You may need to restart php or reboot if you have not done so since uploading the CA cert.

What version is your other pfSense device that is working via the proxy?

brianjg · Jun 25, 2024, 2:19 PM

@stephenw10 I've rebooted multiple times, same result.

This is the only appliance I have.

brianjg · Jun 25, 2024, 2:33 PM

@stephenw10 Also openssl is connecting fine,

openssl s_client pkg01-atx.netgate.com:443
CONNECTED(00000003)

stephenw10 · Jun 25, 2024, 2:57 PM

@brianjg said in Unable to upgrade due to HTTPS scanning (NOT USING A PROXY):

Every other device (including the current pfsense appliance) is routing traffic just fine.

So by that you mean just routing? Not that the current pfSense appliance is able to connect?

Is this something new that has just stopped? You recently added the proxy/filter?

You can just install 2.7.2 clean and restore the config. The proxy setting is fixed there.

brianjg · Jun 25, 2024, 2:58 PM

@stephenw10 I've edited the /usr/local/share/pfSense/pkg/repos/pfSense-repo.conf from https to http only which allowed me to update to the latest 2.6 release.

I then changed the branch to 2.7 and exited the file again to try to upgrade but after running the command: # pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade It failed with,

pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
pkg-static: Repository pfSense-core missing. 'pkg update' required
pkg-static: No package database installed.  Nothing to do!
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating pfSense-core repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    2 KiB   1.8kB/s    00:01
Processing entries: 100%
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%  156 KiB 159.4kB/s    00:01
Processing entries:   0%
pkg-static: Newer FreeBSD version for package unzoo:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1400085
- running kernel: 1203500

pkg-static: repository pfSense contains packages for wrong OS version: FreeBSD:14:amd64
Processing entries: 100%
Unable to update repository pfSense
Error updating repositories!

Is there a way to get it to proceed?

brianjg · Jun 25, 2024, 3:00 PM

@stephenw10 said in Unable to upgrade due to HTTPS scanning (NOT USING A PROXY):

@brianjg said in Unable to upgrade due to HTTPS scanning (NOT USING A PROXY):

Every other device (including the current pfsense appliance) is routing traffic just fine.

So by that you mean just routing? Not that the current pfSense appliance is able to connect?

Is this something new that has just stopped? You recently added the proxy/filter?

You can just install 2.7.2 clean and restore the config. The proxy setting is fixed there.

Yes, just routing is working. I've inherited this setup from the previous admin. It must have worked at some stage in the past since packages are installed and I can't install them now.

Yes I can deploy a new CE release but the issue persists, so since I can't install packages this isn't a solution.

stephenw10 · Jun 25, 2024, 3:40 PM

What should work here is to install 2.7.2 clean then set the proxy rather than trying to use it transparently. The proxy setting works correctly in 2.7.2.

Can you test that in a VM?

The pkg repos are https only from 2.7.0.

I would have expected the imported CA cert to work for command line connections.

brianjg · Jun 25, 2024, 4:11 PM

@stephenw10
For any other VM I have no need to define any proxy settings.

I know it won't be popular but I deployed OPNsense, imported my CA cert and was able to grab updates/packages without an issue. No proxy settings just defined the upstream gateway.

Since I have no visibility beyond my initial upstream GW I've no idea about which device is doing the HTTPs inspection so setting that IP isn't really an option or solution for me.

In my mind once I have imported the cert the HTTPS should just work. As a test I deployed the 2.7.2 release and also had the same issue once the cert was imported.

stephenw10 · Jun 25, 2024, 4:26 PM

Ok how exactly did you import the CA cert?

Ah, you don't control the upstream proxy? Some other admin just gives you a CA cert to use and you're forced to trust they do the right thing with it?