HAProxy/acme script bug?

pfdog

PfSense Community Editions – 2.4.5-RELEASE (amd64)
Packages: acme 0.6.8, haproxy 0.60_4

Suspect files: *.crt_list, *.pem, and luascript_acme-http01-webroot.lua in /var/etc/haproxy_test/ and /var/etc/haproxy/

HAProxy's integration of acme appears to insert an erroneous '^M' character in the above files.
(These do not appear using the cat command in web gui's command prompt; Instead, verify in the console with the vi editor).

I believe this to cause an error such as:
[ALERT] 139/152851 (21304) : parsing [/var/etc/haproxy_test/haproxy.cfg:27] : 'bind … :443' : 'crt-list' : error processing line 2 in file '

Manually correcting these files is futile, as they re-written when changes are applied in the web gui at Services / HAProxy / Frontend

Advise please.

PiBa

@pfdog
If the 'error' comes back when applying haproxy's configuration that means the certs are probably already 'wrong' in the pfSense certificate manager as well.?

So then indeed the question how was that filled, i guess your using haproxy's acme script to validate the webroot request.. But that does not retrieve the certificates from LE.. So the issue would have to be in the acme package itself.. but that to isn't actually that likely as lots of people are using it without issue..

-Can you check the cert that acme.sh downloads in the /tmp/acme// subfolders is it 'valid'?
-Can you check that cert's that are in the pfSense-certificate manager for extra ^M chars?

Have you ever patched/re-saved the .php / .inc / .sh files on pfSense? And possibly changed the line endings in there in the process?

pfdog

@PiBa-NL
The certificates in both /tmp/acme/ and in the pfSense cert manager look good (no ^M char).

I patched acme_accountkeys_edit.php a couple years ago on an older version of pfSense. In this current version of pfSense, that same file is date-stamped Apr 29th 2020. I found no '^M' char in subdir file of /usr/local/www/

By the way, ^M shows up in http.header ...

HTTP/2 200 ^M
server: nginx^M
date: Tue, 19 May 2020 17:48:02 GMT^M
content-type: application/pem-certificate-chain^M
content-length: 3579^M
cache-control: public, max-age=0, no-cache^M
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"^M
replay-nonce: 0102On8yDgwQnxkw_iYcAcZWRXoovhcYnxGR9C7OUietI6U^M
x-frame-options: DENY^M
strict-transport-security: max-age=604800^M
^M

PiBa

@pfdog
http headers 'must' be separated with 'CR LF' characters.. The ^M is vi's way of showing the LF part, so that http headers contain these characters is good..

If the certs are good in cert-manager, then acme is 'done' and nolonger a suspect. So then i guess it must be the last step where haproxy saves the cert from pfSense cert-manager to its own configuration directory.?. Can you check both the /usr/local/pkg/haproxy and /usr/local/www/haproxy folders for unexpected LF's ?

pfdog

@PiBa

The /usr/local/pkg/haproxy and /usr/local/www/haproxy directories look fine.

In the pfSense web gui, at Services / HAProxy / Frontend, under SSL Offloading, I see that my configs include an invalid "additional certificate." Deleting that reference and then re-applying brings only the following message...

WARNING] 141/092843 (12781) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
[info] 141/092843 (12781) : [acme] http-01 plugin v0.1.1

So, it appears that a defunct additional certificate was the culprit.

PiBa

@pfdog
Do you have an idea how this 'defunct' certificate came in place? Did the certificate-id not exist in pfSense certificate configuration at all? Or was the reference valid but the certificate itself somehow not 'valid'.?

I guess the whole ^M was a unrelated goose chase.. I could reproduce a similar error that actually showed a bit more complete error message though:

[ALERT] 116/054134 (69835) : parsing [/var/etc/haproxy_test/haproxy.cfg:53] : 'bind 192.168.0.19:443' : 'crt-list' : error processing line 2 in file '/var/etc/haproxy_test/test.crt_list' : unable to load SSL private key from PEM file '/var/etc/haproxy_test/test/test_5ea4e604a1ab6.pem'.

This for example tells the problem is with the test_5ea4e604a1ab6.pem file, looking in that file its empty at least in my reproduction where i threw away a cert from the pfSense config by unset-ting it with a php command..

unset($config['cert'][16]);
write_config();

Though i guess thats not what you did to get the problem.?.

pfdog

@PiBa
I believe that the reference to the cert was valid. If memory serves me correctly, there were validation problems with the acme-challenge or missing txn parameters for acme's Domain SAN list / DNS-[provider] method. Perhaps that caused an empty cert file.

The ^M remains as before, and now Let's Encrypt successfully verifies the domain.