Certificate Expiring Soon | ACME log Could not get nonce, let's try again.

posix · Nov 25, 2021, 2:25 PM

Hello,
I have Lets Encrypt SSL wild card certificate setup on

pfsense 21.05.1-RELEASE on SG-5100
acme 0.6.10

When logged into pfsense today I saw the following error:
The following CA/Certificate entries are expiring:
Certificate: WildCardCert.name.com (6148ef1dd2fd4): Expiring soon, in 24 days @ 2021-11-25 03:01:00

more acme_issuecert.log

Could not get nonce, let's try again.
[Thu Nov 25 00:47:51 EST 2021] _request_retry_times='18'
[Thu Nov 25 00:47:51 EST 2021] Get nonce with GET. ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
[Thu Nov 25 00:47:51 EST 2021] GET
[Thu Nov 25 00:47:51 EST 2021] url='https://acme-v02.api.letsencrypt.org/directory'
[Thu Nov 25 00:47:51 EST 2021] timeout=
[Thu Nov 25 00:47:51 EST 2021] curl exists=0
[Thu Nov 25 00:47:51 EST 2021] wget exists=127
[Thu Nov 25 00:47:51 EST 2021] _CURL='curl -L --silent --dump-header /tmp/acme/WildCardCert.hamies.world//http.header '
[Thu Nov 25 00:47:51 EST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 35
[Thu Nov 25 00:47:51 EST 2021] ret='35'
[Thu Nov 25 00:47:51 EST 2021] _headers
[Thu Nov 25 00:47:51 EST 2021] _CACHED_NONCE
[Thu Nov 25 00:47:51 EST 2021] nonce
[Thu Nov 25 00:47:51 EST 2021] Could not get nonce, let's try again.
[Thu Nov 25 00:47:53 EST 2021] _request_retry_times='19'
[Thu Nov 25 00:47:53 EST 2021] Get nonce with GET. ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
[Thu Nov 25 00:47:53 EST 2021] GET
[Thu Nov 25 00:47:53 EST 2021] url='https://acme-v02.api.letsencrypt.org/directory'
[Thu Nov 25 00:47:53 EST 2021] timeout=
[Thu Nov 25 00:47:53 EST 2021] curl exists=0
[Thu Nov 25 00:47:53 EST 2021] wget exists=127
[Thu Nov 25 00:47:53 EST 2021] _CURL='curl -L --silent --dump-header /tmp/acme/WildCardCert.hamies.world//http.header '
[Thu Nov 25 00:47:53 EST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 35
[Thu Nov 25 00:47:53 EST 2021] ret='35'
[Thu Nov 25 00:47:53 EST 2021] _headers
[Thu Nov 25 00:47:53 EST 2021] _CACHED_NONCE
[Thu Nov 25 00:47:53 EST 2021] nonce
[Thu Nov 25 00:47:53 EST 2021] Could not get nonce, let's try again.
[Thu Nov 25 00:47:55 EST 2021] _request_retry_times='20'
[Thu Nov 25 00:47:55 EST 2021] Get nonce with GET. ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
[Thu Nov 25 00:47:55 EST 2021] GET
[Thu Nov 25 00:47:55 EST 2021] url='https://acme-v02.api.letsencrypt.org/directory'
[Thu Nov 25 00:47:55 EST 2021] timeout=
[Thu Nov 25 00:47:55 EST 2021] curl exists=0
[Thu Nov 25 00:47:55 EST 2021] wget exists=127
[Thu Nov 25 00:47:55 EST 2021] _CURL='curl -L --silent --dump-header /tmp/acme/WildCardCert.hamies.world//http.header '
[Thu Nov 25 00:47:55 EST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 35
[Thu Nov 25 00:47:55 EST 2021] ret='35'
[Thu Nov 25 00:47:55 EST 2021] _headers
[Thu Nov 25 00:47:55 EST 2021] _CACHED_NONCE
[Thu Nov 25 00:47:55 EST 2021] nonce
[Thu Nov 25 00:47:55 EST 2021] Could not get nonce, let's try again.
[Thu Nov 25 00:47:57 EST 2021] Giving up sending to CA server after 20 retries.
[Thu Nov 25 00:47:57 EST 2021] Register account Error:
[Thu Nov 25 00:47:57 EST 2021] _on_issue_err
[Thu Nov 25 00:47:57 EST 2021] Please check log file for more details: /tmp/acme/WildCardCert.hamies.world/acme_issuecert.log
[Thu Nov 25 00:47:57 EST 2021] _chk_vlist

I am using cloudflare for DNS to host the domain name. The LetsEncrypt and cloudflare account were working before and nothing on the FW has changed. It just started up recently. Going back to 11/22.

jimp · Nov 29, 2021, 3:53 PM

The cURL error suggests it's having trouble negotiating SSL with that server for some reason. Your clock doesn't look that far off, but you might check it to be certain.

If you were on an older version of pfSense I might think it was the root certs being out of date, but it should be OK on 21.05.1.

posix · Nov 29, 2021, 7:29 PM

@jimp
Thanks for the response and pointer.

It looks like the system clock is unsynchronized.

Here is snippet of the logs:

Nov 14 20:28:39	ntpd	44115	Listening on routing socket on fd #40 for interface updates
Nov 14 20:28:39	ntpd	44115	kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Nov 14 20:28:39	ntpd	44115	kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Nov 14 20:28:42	ntpd	44115	Soliciting pool server 2001:4860:4806:4::
Nov 14 20:29:44	ntpd	44115	Soliciting pool server 2001:4860:4806:c::
Nov 14 20:30:48	ntpd	44115	Soliciting pool server 2001:4860:4806::
Nov 14 20:31:52	ntpd	44115	Soliciting pool server 2001:4860:4806:8::
Nov 14 20:32:56	ntpd	44115	Soliciting pool server 216.239.35.12
Nov 14 20:32:57	ntpd	44115	Soliciting pool server 216.239.35.4
Nov 14 20:32:58	ntpd	44115	Soliciting pool server 216.239.35.8
Nov 14 20:32:59	ntpd	44115	Soliciting pool server 216.239.35.0
Nov 14 20:33:00	ntpd	44115	Soliciting pool server 2001:4860:4806:8::
Nov 29 14:24:45	ntpd	44115	ntpd exiting on signal 15 (Terminated)
Nov 29 14:24:45	ntpd	44115	216.239.35.12 local addr PUBLIC_IP -> <null>
Nov 29 14:24:45	ntpd	44115	216.239.35.4 local addr PUBLIC_IP -> <null>
Nov 29 14:24:45	ntpd	44115	216.239.35.8 local addr PUBLIC_IP -> <null>
Nov 29 14:24:45	ntpd	44115	216.239.35.0 local addr PUBLIC_IP -> <null>
Nov 29 14:24:46	ntpd	88480	ntpd 4.2.8p15@1.3728-o Tue Jul 27 00:09:40 UTC 2021 (1): Starting
Nov 29 14:24:46	ntpd	88480	Command line: /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
Nov 29 14:24:46	ntpd	88480	----------------------------------------------------
Nov 29 14:24:46	ntpd	88480	ntp-4 is maintained by Network Time Foundation,
Nov 29 14:24:46	ntpd	88480	Inc. (NTF), a non-profit 501(c)(3) public-benefit
Nov 29 14:24:46	ntpd	88480	corporation. Support and training for ntp-4 are
Nov 29 14:24:46	ntpd	88480	available at https://www.nwtime.org/support
Nov 29 14:24:46	ntpd	88480	----------------------------------------------------
Nov 29 14:24:46	ntpd	88570	proto: precision = 0.138 usec (-23)
Nov 29 14:24:46	ntpd	88570	basedate set to 2021-07-15
Nov 29 14:24:46	ntpd	88570	gps base set to 2021-07-18 (week 2167)
Nov 29 14:24:46	ntpd	88570	Listen and drop on 0 v6wildcard [::]:123
Nov 29 14:24:46	ntpd	88570	Listen and drop on 1 v4wildcard 0.0.0.0:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 2 igb0 [fe80::290:bff:fea2:a829%1]:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 3 igb0 PUBLIC_IP:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 4 igb1 [fe80::290:bff:fea2:a82a%2]:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 5 lo0 [::1]:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 6 lo0 [fe80::1%8]:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 7 lo0 127.0.0.1:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 8 igb0.4090 [fe80::290:bff:fea2:a829%11]:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 9 igb1.30 [fe80::290:bff:fea2:a82a%12]:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 10 igb1.30 192.168.30.1:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 11 igb1.40 [fe80::290:bff:fea2:a82a%13]:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 12 igb1.40 192.168.40.1:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 13 igb1.10 [fe80::290:bff:fea2:a82a%14]:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 14 igb1.10 192.168.10.1:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 15 igb1.100 [fe80::290:bff:fea2:a82a%15]:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 16 igb1.100 192.168.1.1:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 17 igb1.100 10.10.10.1:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 18 ovpns1 [fe80::290:bff:fea2:a829%16]:123
Nov 29 14:24:46	ntpd	88570	Listen normally on 19 ovpns1 192.168.60.1:123
Nov 29 14:24:46	ntpd	88570	Listening on routing socket on fd #40 for interface updates
Nov 29 14:24:46	ntpd	88570	kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized
Nov 29 14:24:46	ntpd	88570	kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized

Not sure which NTP server to use so I configured based on https://www.pool.ntp.org/zone/north-america

Before that I was using googles public NTP just added a few more servers to the list.

posix · Dec 19, 2021, 6:12 PM

@jimp

Just tried again after a fresh install using USB 21.05.2-RELEASE PFSENSE+

[Sun Dec 19 13:10:37 EST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 35
[Sun Dec 19 13:10:39 EST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 35
[Sun Dec 19 13:10:41 EST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 35
[Sun Dec 19 13:10:43 EST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 35
[Sun Dec 19 13:10:45 EST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 35
[Sun Dec 19 13:10:47 EST 2021] Register account Error: 
[Sun Dec 19 13:10:47 EST 2021] Please check log file for more details: /tmp/acme/WildCardCert.hamies.world/acme_issuecert.log

Any other suggestions?

sundaydiver · Jan 1, 2022, 9:01 PM

Do you have pfBlocker running? I had similar issue due to pfblocker blocking letsencrypt url.

posix · Jan 4, 2022, 11:20 PM

@sundaydiver that's exactly what I did.

Excuse the delay everyone but holiday and other stuff.
I performed some further troubleshooting on this in the background and confirmed @sundaydiver mentioned pfblocker is the culprit. One of the DNSBL list is blocking .letsencrypt.org

Performing a curl -v

curl -v -Ii https://acme-v02.api.letsencrypt.org/directory
*   Trying 10.10.10.1:443...
* Connected to acme-v02.api.letsencrypt.org (10.10.10.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /usr/local/share/certs/ca-root-nss.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

Seeing the 10.10.10.1 is the VIP for DNSBL black hole. After adding

.letsencrypt.org #ACME SSL-CERT

to the DNSBL Whitelist I was able renew the SSL cert.

curl -v -Ii https://acme-v02.api.letsencrypt.org/directory
*   Trying 172.65.32.248:443...
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
<OUTPUT OMITTED>

< 
* Connection #0 to host acme-v02.api.letsencrypt.org left intact

Gertjan · Jan 5, 2022, 11:02 AM

@posix said in Certificate Expiring Soon | ACME log Could not get nonce, let's try again.:

Trying 10.10.10.1:443...

Yeah, no need to guess who that is.

@posix said in Certificate Expiring Soon | ACME log Could not get nonce, let's try again.:

letsencrypt.org #ACME SSL-CERT
to the DNSBL Whitelist I was able renew the SSL cert.

That's the easy patch.
The problem is bigger : you should also reveiw the method you use when choosing pfBlockerNG feeds.

And, most important, what feed blacklisted Letsencrypt IP addresses ? I tend to think that feed is actually "mal ware" ...

posix · Jan 6, 2022, 10:32 PM

@gertjan

Correct

So checking Firewall -> pfBlockerNG -> Alerts:
Reports: Alerts:

DNSBL Block

acme-v02.api.letsencrypt.org [ TLD ]  
DNSBL-HTTPS	Abuse_urlhaus
DNSBL_Phishing

This Feed/group is the culprit.