Possible ACME bug in 23.09.1

[ak4]

I upgraded from pfSense+ 22.01 -> 23.09.1 last night. On 22.01 I had Let's Encrypt certificates issued successfully for a few of my domains where port 80 is open on NAT redirecting to port 8080. Ever since going to 23.09.1 ACME complains that there is "likely a firewall problem".

The exact error is Verify error:During secondary validation: 72.42.172.136: Fetching http://mydomain.cloud/.well-known/acme-challenge/63GdkExPYbPZKPOgM45HHGVBEQoxN5jfFB0j37__DLE: Timeout during connect (likely firewall problem) [Thu Apr 4 18:59:56 AKDT 2024] Please check log file for more details: /tmp/acme/pfSense-Cert/acme_issuecert.log

The logs don't really show anything helpful related to a "timeout".

In Services -> Acme Certificates I re-created a fresh Account key to try and rule everything out. I have attached a few screenshots to show how the firewall NAT + rule is configured. I am at a complete loss as to why this configuration no longer works with the latest version of pfSense I am on. Of course if I set the NAT rule to "ANY" then I can see port 80 and other ports are open. Unfortunately Acme still fails but not with a firewall error, but an error that it can't handle/understand -- I think because it tries to use https instead of http. My ISP does not block port 80.

ACME version: 0.7.5

Hoping someone can shed some light on what is going on.

--UPDATE--
For anyone else who might run into this snag, the problem ended up being related to pfBlocker. Specifically version 3.20_7. I don't know enough about pfBlocker's latest code that would cause this problem with Acme certs when port 80 from NAT side (redirecting to 8080) is allowed/open. If anyone can help advise how pfBlocker can allow Acme issued certs to work without needing to disable pfBlocker that would be incredible.

[Gertjan]

@ak4 said in Possible ACME bug in 23.09.1:

ended up being related to pfBlocker

pfBlocker, by itself, when you install it, does 'nothing'.
I'll advise you strongly to fact check this and not just believe me ^^

So, yes, it's actually easy : you install 'some DNSBL' or IP list and suddenly you can't, from your LANs, reach some destinations on the internet.
And also the other way around : suddenly, you block sources that needs to contact your pfSense, like the Letenscrypt verification service, as you've picked a list that contains these sources (IP, etc).
I presume you manged to do just that.

Remember : you use these DNSBL lists and IP lists 'as is'. But shouldn't you check them before using them ?
It has happened : a list conatins all the Amazon WS IPs. Right after you use this list, suddenly, pfBlockerng can't update any list anymore, as most lists are hosted on Amazon WS.
Or : also pure fun : some IP list managed to include all RFC1918 and suddenly pfBlocker start to block all your LAN devices and it's "Internet & pfSense is broken again" time, or its actually the "the admin didn't do it's job" time.

What I normally do if I use a new list : I use this package :

and I make notes, like : installed on 2024-04-04 IP list Xyz, and then I see what happens. if somethings strange happens, I undo what I've done last, and often the issue is solved. Then I go through the 'why' phase.

But before testing : don't you want de latest acme version (with the latest correction etc) ?

I'm using : 23.09.1 (actually 24.03-BETA since a week as it is rock solid) and :

which came out .... weeks ago.
Why wait ?