Ah ha! I found the issue! jimp, was HAProxy updated with the upgrade to 2.4.2?
Using the standalone method, I created a backend in the HAProxy UI with a single server bound to localhost on port 8082, with no health checks, or timeout/retry settings.
(I didn't use health checks was because this backend is "down" whenever the acme.sh script isn't running. So I just ignored the overhead of doing a health check.)
The problem here is there is an IPv4 and IPv6 address for localhost, so in the newest version of HAProxy, it actually created two servers even though the UI only had one specified:
backend 0_HTTP_ACME_Standalone_http_ipvANY
mode http
log global
timeout connect 30000
timeout server 30000
retries 3
server pfsense_0 127.0.0.1:8082
server pfsense_1 ::1:8082
The ACME client only binds to the IPv4 interface using socat:
[2.4.2-RELEASE][root@pfsense]/tmp/acme: sockstat -l46 | grep 8082
root socat 96563 5 tcp4 *:8082 *:*
(I ran this sockstat command during the execution of certificate creation/renewal and it only ever listens on 127.0.0.1:8082.)
And the nail in the coffin is that with the None option specified in the load balancing section of the backend, it defaults to Round Robin:
@HAProxy:
The load balancing algorithm of a backend is set to roundrobin when no other
algorithm, mode nor option have been set. The algorithm may only be set once
for each backend.
So LE was successfully reaching in to my infrastructure (as I noted with the packet capture) on the 127.0.0.1:8082 server, but when it attempted to reach in again, it would be Round Robin'd to the ::1:8082 server, to which the ACME client wasn't bound. This would then timeout and cause the validation process to fail.
–-
I've adjusted the backend to only listen on 127.0.0.1, not localhost, but I would be more satisfied if I knew the proper knobs to turn such that the HAProxy backend would timeout quickly, and a new request would be issued to the next server in the list. Alas, I will have to figure that out another day.
Thanks for your help, jimp!
–-
PS - I actually tried to pull the HEAD of acme.sh's repo to execute on the pfSense box, but as soon as I saw the output I knew there had to be adjustments made to the source code that made it compatible with pfSense. I gave that up quickly.
I also only ran it from the command line because the ACME UI actually prints out the full command of what it's executing under the covers. I thought it was safe to copy and paste it so I could have control over execution during debugging. :)
Thanks again