error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed

kevdog · Dec 8, 2024, 2:35 PM

I'm using pfsense 2.7.2-RELEASE (amd64)
I'm getting the following error coming up repeatedly in my unbound logs -- almost every minute:

Dec 8 07:27:56	unbound	99937	[99937:0] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Dec 8 07:27:56	unbound	99937	[99937:0] notice: ssl handshake failed 10.0.1.1 port 853

The weird thing about this error is its complaining against an ssl handshake that is failing against the router itself -- The ip address of my pfsense installation is 10.0.1.1 and this is what seems to be generating the error -- or at least this is how I'm interpreting this.

The complete sequence of events where the unbound service stops and starts is here:

Dec 8 07:22:14	unbound	99937	[99937:0] info: start of service (unbound 1.18.0).
Dec 8 07:22:14	unbound	99937	[99937:0] notice: init module 0: iterator
Dec 8 07:22:14	unbound	99937	[99937:0] notice: Restart of unbound 1.18.0.
Dec 8 07:22:14	unbound	99937	[99937:0] info: 8.000000 16.000000 5
Dec 8 07:22:14	unbound	99937	[99937:0] info: 4.000000 8.000000 13
Dec 8 07:22:14	unbound	99937	[99937:0] info: 2.000000 4.000000 4
Dec 8 07:22:14	unbound	99937	[99937:0] info: 1.000000 2.000000 5
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.262144 0.524288 14
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.131072 0.262144 20
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.065536 0.131072 36
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.032768 0.065536 36
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.016384 0.032768 24
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.008192 0.016384 11
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.004096 0.008192 8
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.002048 0.004096 1
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.000000 0.000001 4
Dec 8 07:22:14	unbound	99937	[99937:0] info: lower(secs) upper(secs) recursions
Dec 8 07:22:14	unbound	99937	[99937:0] info: [25%]=0.0308907 median[50%]=0.0773689 [75%]=0.234291
Dec 8 07:22:14	unbound	99937	[99937:0] info: histogram of recursion processing times
Dec 8 07:22:14	unbound	99937	[99937:0] info: average recursion processing time 0.844864 sec
Dec 8 07:22:14	unbound	99937	[99937:0] info: server stats for thread 1: requestlist max 6 avg 0.80663 exceeded 0 jostled 0
Dec 8 07:22:14	unbound	99937	[99937:0] info: server stats for thread 1: 327 queries, 146 answers from cache, 181 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Dec 8 07:22:14	unbound	99937	[99937:0] info: 8.000000 16.000000 12
Dec 8 07:22:14	unbound	99937	[99937:0] info: 4.000000 8.000000 22
Dec 8 07:22:14	unbound	99937	[99937:0] info: 2.000000 4.000000 6
Dec 8 07:22:14	unbound	99937	[99937:0] info: 1.000000 2.000000 3
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.524288 1.000000 2
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.262144 0.524288 8
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.131072 0.262144 14
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.065536 0.131072 41
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.032768 0.065536 34
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.016384 0.032768 27
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.008192 0.016384 14
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.004096 0.008192 17
Dec 8 07:22:14	unbound	99937	[99937:0] info: 0.000000 0.000001 9
Dec 8 07:22:14	unbound	99937	[99937:0] info: lower(secs) upper(secs) recursions
Dec 8 07:22:14	unbound	99937	[99937:0] info: [25%]=0.0238175 median[50%]=0.0711305 [75%]=0.28672
Dec 8 07:22:14	unbound	99937	[99937:0] info: histogram of recursion processing times
Dec 8 07:22:14	unbound	99937	[99937:0] info: average recursion processing time 1.334821 sec
Dec 8 07:22:14	unbound	99937	[99937:0] info: server stats for thread 0: requestlist max 10 avg 0.733032 exceeded 0 jostled 0
Dec 8 07:22:14	unbound	99937	[99937:0] info: server stats for thread 0: 424 queries, 203 answers from cache, 221 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Dec 8 07:22:14	unbound	99937	[99937:0] info: service stopped (unbound 1.18.0).
Dec 8 07:20:21	unbound	99937	[99937:1] notice: ssl handshake failed 10.0.1.1 port 853
Dec 8 07:20:21	unbound	99937	[99937:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Dec 8 07:20:21	unbound	99937	[99937:1] notice: ssl handshake failed 10.0.1.1 port 853
Dec 8 07:20:21	unbound	99937	[99937:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Dec 8 07:19:38	unbound	99937	[99937:1] notice: ssl handshake failed 10.0.1.1 port 853
Dec 8 07:19:38	unbound	99937	[99937:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Dec 8 07:19:38	unbound	99937	[99937:1] notice: ssl handshake failed 10.0.1.1 port 853
Dec 8 07:19:38	unbound	99937	[99937:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Dec 8 07:15:06	unbound	99937	[99937:0] notice: ssl handshake failed 10.0.1.1 port 853
Dec 8 07:15:06	unbound	99937	[99937:0] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Dec 8 07:15:06	unbound	99937	[99937:0] notice: ssl handshake failed 10.0.1.1 port 853
Dec 8 07:15:06	unbound	99937	[99937:0] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed

I'm not exactly sure how to debug this error of the cause so I'm just going to post some screenshots of my various settings and certificates. I'm hoping someone can actually help with this:

login-to-view

dmorda · Jan 1, 2025, 1:11 PM

I'm seeing something similar, but not quite the same.

Jan 1 08:10:06	unbound	56909	[56909:0] error: remote control failed ssl crypto error:0A000415:SSL routines::sslv3 alert certificate expired
Jan 1 08:10:06	unbound	56909	[56909:0] notice: failed connection from 127.0.0.1 port 1236
Jan 1 08:10:05	unbound	56909	[56909:0] error: remote control failed ssl crypto error:0A000415:SSL routines::sslv3 alert certificate expired
Jan 1 08:10:05	unbound	56909	[56909:0] notice: failed connection from 127.0.0.1 port 25279
Jan 1 08:10:04	unbound	56909	[56909:0] error: remote control failed ssl crypto error:0A000415:SSL routines::sslv3 alert certificate expired
Jan 1 08:10:04	unbound	56909	[56909:0] notice: failed connection from 127.0.0.1 port 54171
Jan 1 08:10:03	unbound	56909	[56909:0] error: remote control failed ssl crypto error:0A000415:SSL routines::sslv3 alert certificate expired
Jan 1 08:10:03	unbound	56909	[56909:0] notice: failed connection from 127.0.0.1 port 15107

I've been troubleshooting, but can't seem to track it down yet. Have you had any luck? I'm running the pfSense 24.11-RELEASE.

Gertjan · Jan 1, 2025, 1:27 PM

@dmorda said in error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed:

alert certificate expired

Not really complicated. You have / are using a certificate that is expired.
I propose that you renew it

edit :

This one - here you have the name :

login-to-view

used by the Resolver (unbound).

dmorda · Jan 1, 2025, 1:26 PM

@Gertjan Yep, that's the first thing I checked. I had unbound configured to use an active, non-expired, certificate. I then generated a new one specifically for it, deleted some older ones that were expired, but not in use. I then rebooted pfSense and the issue seems to have resolved itself. Hoping it stays resolved, thanks for chiming in!

Gertjan · Jan 1, 2025, 1:36 PM

@dmorda said in error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed:

Hoping it stays resolved

Certificates have an end-of-life date build in.

I "solved" this by using a cert for unbound that gets auto regenerated (by Encrypted) - as I use the same cert for the GUI, captive portal, my printers etc etc..

johnpoz · Jan 1, 2025, 3:31 PM

@Gertjan That cert to use is meaningless unless you have the enabled ssl/tls service enabled.

Now @kevdog does have it enabled - but why would be my question. Do you really have clients talking to unbound on pfsense via dot? Clients don't normally use dot, they would use doh.. And while you can get unbound to listen for doh, you would have to call it out in options.

If you don't have clients talking to unbound via dot, just uncheck that box..

Gertjan · Jan 1, 2025, 3:42 PM

@johnpoz said in error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed:

Now @kevdog does have it enabled - but why would be my question. Do you really have clients talking to unbound on pfsense via dot?

Has has DOT (DNS over TLS on port 853, TCP (only !), right?) enabled, because he has set up his LAN devices to use it - I guess ?

With, for example, Windows 10, external software was needed to make use of DOT, as some don't trust there own Wifi and don't trust their own network cables.
Recently, browsers ca be set up to make use of DOT, and Windows 11 (pro, at least) can also use it if needed.

The bottom line : this must exist : it's pfSense, so why not, let's select and check every option in the GUI ^^

Btw : If "Enable SSL/TLS Service" is not selected, does unbound actually need a cert ?

johnpoz · Jan 1, 2025, 4:44 PM

@Gertjan said in error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed:

Btw : If "Enable SSL/TLS Service" is not selected, does unbound actually need a cert ?

no - that is my point.. If you don't actively have devices wanting to talk to unbound via dot there is little point in having that enabled. I find it odd that windows would default to using dot vs doh if wanting to use dns over tls.. Dot is almost exclusively for NS to use talking to other NSers.. I am not aware of clients that would default to dot vs doh. But sure its not out of the realm of possibility - MS does there own thing - I think they have servers using doh and clients using dot from some of the stuff I just read - which makes zero sense.. Clients would use doh and servers, that normally are dns servers etc and talk to other NSers which would normally be dot.

kevdog · Jan 1, 2025, 7:19 PM

Thanks for this thread finally getting some traction since I've seemingly received this error for months.

So just to give some more information
I've checked all my certificates on pfSense.

My unbound resolver is using a certificate known as pfsense.<domain>.com
and the actual certificate is issued through E5: Here is some of the actual certificate:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:67:40:6c:0a:52:f4:4f:95:6b:cf:4b:6c:2d:ca:42:a4:40
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C = US, O = Let's Encrypt, CN = E5
        Validity
            Not Before: Nov  7 08:18:06 2024 GMT
            Not After : Feb  5 08:18:05 2025 GMT
        Subject: CN = pfsense.<domain>.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub:
                  ....
                ASN1 OID: secp384r1
                NIST CURVE: P-384
...

Certificate isn't expired so I don't think that's it.

So my latest log looks like the following - 10.0.1.1 is the actual address of the pfSense installation. In terms of what's causing the error I'm not sure:

Jan 1 12:37:26	unbound	1633	[1633:1] notice: ssl handshake failed 10.0.1.1 port 853
Jan 1 12:37:26	unbound	1633	[1633:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Jan 1 12:37:26	unbound	1633	[1633:1] notice: ssl handshake failed 10.0.1.1 port 853
Jan 1 12:37:26	unbound	1633	[1633:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Jan 1 12:35:33	unbound	1633	[1633:1] notice: ssl handshake failed 10.0.1.1 port 853
Jan 1 12:35:33	unbound	1633	[1633:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Jan 1 12:35:33	unbound	1633	[1633:1] notice: ssl handshake failed 10.0.1.1 port 853
Jan 1 12:35:33	unbound	1633	[1633:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Jan 1 12:32:27	unbound	1633	[1633:1] notice: ssl handshake failed 10.0.1.1 port 853
Jan 1 12:32:27	unbound	1633	[1633:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Jan 1 12:32:27	unbound	1633	[1633:1] notice: ssl handshake failed 10.0.1.1 port 853
Jan 1 12:32:27	unbound	1633	[1633:1] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
Jan 1 12:32:10	unbound	1633	[1633:0] info: start of service (unbound 1.18.0).

In terms of the question in regards to DOT/DOH.
I believe I'm using DOT. Although TLS DNS resolver queries aren't exactly necessary, I'm trying to observe best practices. In terms of who is looking up TLS encrypted queries -- my setup involves a local LAN with pfsense access. The pfsense is connected to another pfSense router through a site-to-site Wireguard VPN. I believe it's this site-to-site configuration that's causing the problem (more below). Why I don't believe it's the LAN clients causing this error is the following:

My VM LAN clients using TLS Unbound resolution are VM's utilizing systemd-networkd and systemd-resolved. My resolve configuration file on the VM's look like:

[Resolve]
DNSSEC=no
Domains=~. <domain>.com
DNS=10.0.1.1#pfsense.<domain>.com
DNSOverTLS=opportunistic
FallbackDNS=127.0.0.1 10.0.1.1#pfsense.<domain>.com 9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 1.1.1.2#cloudflare-dns.com 1.1.1.3#cloudflare-dns.com 1.1.1.1#cloudflare-dns.com
LLMNR=no

Although this is the default configuration, you can see that since all these VM's get receive a reserved IP address through pfSense, the default configuration really isn't used since I believe Dynamically assigned IP addesses fall back on a "per-link" interface configuration that is sent through pfsense:

# resolvectl status
Global
           Protocols: -LLMNR -mDNS DNSOverTLS=opportunistic DNSSEC=no/unsupported
    resolv.conf mode: stub
  Current DNS Server: 10.0.1.1#pfsense.<domain>.com
         DNS Servers: 10.0.1.1#pfsense.<domain>.com
Fallback DNS Servers: 127.0.0.1 10.0.1.1 9.9.9.9 1.1.1.2 1.1.1.3
          DNS Domain: <domain>.com ~.

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS DNSOverTLS=opportunistic DNSSEC=no/unsupported
Current DNS Server: 10.0.1.1
       DNS Servers: 10.0.1.1
        DNS Domain: <domain>.com

Back to the site-to-site Wireguard VPN.

VPN Tunnel Network 10.99.210.0/30
Peer #1 - VPN - LAN IP address 10.0.1.1 with Tunnel address of 10.99.210.1
Peer #2 - VPN - LAN IP address of 10.1.0.1 with Tunnel address of 10.99.210.2

Here is the actual Interfaces for the Wireguard VPN:
Peer #1 -
login-to-view

Peer #2 -
login-to-view

In terms of the error Log on Peer #2 - I'm getting this:

Dec 31 23:35:00	unbound	64101	[64101:2] notice: ssl handshake failed 10.99.210.1 port 38077
Dec 31 23:35:00	unbound	64101	[64101:2] error: ssl handshake failed crypto error:00000000:lib(0)::reason(0)
Dec 30 23:16:52	unbound	64101	[64101:2] notice: ssl handshake failed 10.99.210.1 port 45160
Dec 30 23:16:52	unbound	64101	[64101:2] error: ssl handshake failed crypto error:00000000:lib(0)::reason(0)
Dec 30 11:03:43	unbound	64101	[64101:2] notice: ssl handshake failed 10.99.210.1 port 19464
Dec 30 11:03:43	unbound	64101	[64101:2] error: ssl handshake failed crypto error:00000000:lib(0)::reason(0)
Dec 30 05:27:21	unbound	64101	[64101:2] notice: ssl handshake failed 10.99.210.1 port 48125
Dec 30 05:27:21	unbound	64101	[64101:2] error: ssl handshake failed crypto error:00000000:lib(0)::reason(0)
Dec 29 13:22:22	unbound	64101	[64101:2] notice: ssl handshake failed 10.99.210.1 port 25532
Dec 29 13:22:22	unbound	64101	[64101:2] error: ssl handshake failed crypto error:00000000:lib(0)::reason(0)
Dec 29 09:41:12	unbound	64101	[64101:3] notice: ssl handshake failed 10.99.210.1 port 18575

I have no idea if the two are related. In logs above, I don't know what the port numbers 48125, 25532, 18575, etc represent.

Hopefully that's more representative of the problem.

johnpoz · Jan 1, 2025, 11:33 PM

@kevdog said in error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed:

notice: ssl handshake failed 10.99.210.1 port 38077

why would you gateway be talking to unbound for dot, or why would you be talking to your gateway= for dot?

If you uncheck that box - do these errors go away.. Those ports are not dot port or doh ports - are those the source ports that 10.99.210.1 is talking to your 10.99.210.2 IP?

kevdog · Jan 2, 2025, 12:08 AM

@johnpoz

Ok hopefully I can explain my setup

I'm running two pfsense installations -- each running their own unbound server.
Each installation has domain overrides for servers running on their respective LAN.

Although it's an unconventional setup since I don't have a bind setup, I want clients on say Network #1 when doing dot lookup or dns lookups to consult the local unbound server first, then if name resolution not found then use the pfsense unbound server on Network #2. The gateways represent the tunnel addresses of the pfsense installations at either end of the tunnel.

"If you uncheck that box - do the errors go away?" -- What box are you referring to?

In terms of ports for the wireguard for this site to site, the only wireguard port being employed is 51822 in the wireguard setup. I have no idea why those random ports are being used.

johnpoz · Jan 2, 2025, 12:26 AM

@kevdog no clue to how/why you would think such a setup would work - dns is not done in any sort of order - you cant say check this dns server first, etc.

You can not redirect dot or doh - not from a sane client, because the client should validate that its talking to the device it was told to talk to that validates via the cert be it the fqdn or the Ip in the cert.

If your already inside a tunnel, what is the point of dot or doh in the first place?

Have your clients ask unbound for whatever they are looking for - if you want unbound to then forward this to some dot server - then do that on unbound, which has zero to do with what unbound listening on dot or doh.

the ability to redirect dot or doh - invalidates one of its prime functions - validation that your actually talking the NS you told the client to talk too.

kevdog · Jan 2, 2025, 12:49 AM

@johnpoz I admit the setup isn't ideal, however somehow despite the error messages the system seems to work -- clearly I don't really understand all the underpinnings of how things work.

How should I be constructing things??

Two pfsense installations running unbound with same domain. Each pfsense installation has domain overrides for subdomains running on their installation. Additionally each pfsense answering DNS over DOT on port 853.

domain.com------->>>Pfsense  #1 (domain=domain.com) -> Overrides--->test.domain.com
                                                                --->test2.domain.com
                                                                --->test3.domain.com
               
           ------>>>Pfsense #2 (domain=domain.com) -> Overrides --->test4.domain.com
                                                                --->test5.domain.com
                                                                --->test6.domain.com

Each installation can resolve locally, however if pfsense installations connected by vpn, I need name resolution for devices on Pfsense #1 network accessible to devices on Pfsense#2 network -- and vice versa. If VPN is broken or down, local domain overrides will still work.

I'm just making use right now of the unbound domain overrides section similar to this:
login-to-view

In terms of DOT -- no I don't need it between the pfsense nodes on either end of the tunnel -- however how do I have it only active for LAN clients but not for the tunnel?

I'm not looking actually to forward DNS requests, rather have unbound "resolve" them and then pass the answer back to the clients. In terms of resolving (not forwarding), how does each unbound server know what DNS server is definitive for a specific local domain that's split? I thought I was accomplishing this by listing the servers within the domain overrrides section.