Possible bug + fix for HAproxy issue during upgrade to 2.8.1

[ndemou]

Hi all,

I hit an HAProxy startup failure after upgrading to pfSense CE 2.8.1-RELEASE that seems like it's a package code issue:

If someone hits:

• HAProxy not starting after upgrade/repair
• These alerts:

[ALERT] : config : parsing [/var/etc/haproxy/haproxy.cfg:29] :
'bind ... ssl crt-list /var/etc/haproxy/Shared_HTTPS_fronted.crt_list'
: unable to load certificate from file
'/var/etc/haproxy/Shared_HTTPS_fronted.pem': no start line.

then it may be this exact issue and I’m posting it in case it solves their issue too.

Note that troubleshooting was performed 100% by ChatGPT so I'm not confident about the details. But this is certain: HAProxy was not starting and after my custom troubleshooting GPT patched function haproxy_lookup_cert() it was starting fine. Here's a report by ChatGPT:

---

Environment

System:

• pfSense CE 2.8.1-RELEASE
• haproxy-2.8.3
• pfSense-pkg-haproxy-0.63_2

The system had previously gone through repair work after an incomplete/misaligned upgrade state, but by the time I investigated HAProxy, the firewall itself was otherwise healthy and other repaired services were working.

Symptom

HAProxy would not start.

The key failure was:

[ALERT] : config : parsing [/var/etc/haproxy/haproxy.cfg:29] :
'bind ... ssl crt-list /var/etc/haproxy/Shared_HTTPS_fronted.crt_list'
: unable to load certificate from file
'/var/etc/haproxy/Shared_HTTPS_fronted.pem': no start line.

What made this interesting is that the referenced PEM file existed but was 0 bytes:

-rw-r--r--  1 root wheel   47 ... /var/etc/haproxy/Shared_HTTPS_fronted.crt_list
-rw-r--r--  1 root wheel    0 ... /var/etc/haproxy/Shared_HTTPS_fronted.pem

The crt_list pointed to that empty PEM:

/var/etc/haproxy/Shared_HTTPS_fronted.pem []

Initial suspicion

At first I assumed either:

• the selected pfSense certificate object was missing,
• the private key was missing,
• or HAProxy config had a stale cert reference.

That turned out not to be the case.

The configured ssloffloadcert existed in pfSense config and the certificate/private key pair was valid.

What I verified

The configured cert reference in the HAProxy package section was present and pointed to a valid certificate.
The raw cert object in pfSense config contained both crt and prv, and both parsed correctly with OpenSSL.

In other words:

• cert existed
• key existed
• cert/key matched
• cert dates were fine

So the failure was not “bad certificate content”.

Where it went wrong

The HAProxy package code writes the PEM through this path:

function haproxy_write_certificate_fullchain($filename, $certid, $append = false, $skiproot = true) {
    $cert = haproxy_lookup_cert($certid);

    $certcontent = base64_decode($cert['crt']);
    if (isset($cert['prv'])) {
        $certcontent .= "\r\n".base64_decode($cert['prv']);
    }
    ...
    file_put_contents($filename, $certcontent, $flags);
}

And haproxy_lookup_cert() was:

function haproxy_lookup_cert($certid) {
    $res = lookup_ca($certid);
    if (!$res) {
        $res = lookup_cert($certid);
    }
    return $res;
}

On this pfSense 2.8.1 system, lookup_cert() returns a wrapper structure like:

Array
(
    [idx] => 17
    [item] => Array
        (
            [refid] => ...
            [descr] => ...
            [crt] => ...
            [prv] => ...
        )
)

The important part is that crt and prv are under item, not at top level.

What I observed was:

• lookup_cert($refid) returned ['idx' => ..., 'item' => [...]]
• haproxy_lookup_cert($refid) ended up returning an empty wrapper-like result
• the code later tried to read $cert['crt'] and $cert['prv'] at the top level
• result: empty content got written to the PEM
• HAProxy then failed with no start line

So the practical effect is: HAProxy writes a zero-byte PEM even though the selected pfSense certificate object is valid.

Probable cause

This looks like a mismatch between the HAProxy package code and the pfSense certificate lookup return format.

More specifically, the package appears to expect a direct cert array, while current lookup helpers return wrapper arrays with idx and item.

There is also a strong possibility that lookup_ca() returning a non-false wrapper-like value prevents the fallback to lookup_cert(), which makes the issue worse.

I did not fully trace every helper beyond this point, but the observed behavior was consistent and the workaround below fixed the problem immediately.

Temporary workaround that restored service

I patched haproxy_lookup_cert() locally so it unwraps the item result when needed:

function haproxy_lookup_cert($certid) {
    $res = lookup_ca($certid);
    if (is_array($res) && isset($res["item"]) && is_array($res["item"]) && !empty($res["item"])) {
        return $res["item"];
    }
    if (is_array($res) && isset($res["crt"])) {
        return $res;
    }
    $res = lookup_cert($certid);
    if (is_array($res) && isset($res["item"]) && is_array($res["item"]) && !empty($res["item"])) {
        return $res["item"];
    }
    return $res;
}

After that I regenerated the PEM using the package helper and got a normal non-empty file:

/var/etc/haproxy/Shared_HTTPS_fronted.pem
Size: 6966

Then:

haproxy -c -V -f /var/etc/haproxy/haproxy.cfg
Configuration file is valid

And HAProxy started successfully, with listeners bound on the expected addresses.

The website behind HAProxy also worked again immediately after restart.