HAProxy 1.5 and OCSP

nikolaii

Hi,

I'd like to add the OCSP functionality to the HAProxy configuration. I know that since version 1.5 this is somewhat possible.

Does anyone knows how to enable it in Pfsense?

Thank you.

PiBa

Its not currently possible to configure this using the build in webgui options.. Ive got it on my 'to investigate' list.. But don't think ill be implementing it any time soon..

To add it a cron job will need to download and reconfigure the new oscp response every X time interval.. It will need to know where to get the oscp responses from and download them. Then put configure them over the unix socket. Its not that haproxy takes care of this by making a few configuration settings…

If you can send a pullrequest on github for this feature that would be great :D.

nikolaii

Yes, I've read some topic on this and this is indeed not that easy I guess.

For the pull request on github I'm no developer at all (I'm a sysadmin guy), so on this one you can't count on me, sorry :)

Thanks for your reply anyway.

michaelschefczyk

Dear PiBa, dear All,

It would be really good to have OCSP stapling. This should avoid informing another party about encrypted communications and speed up access.

Please let us pool ideas about practical workarounds that might get us OCSP stapling without waiting for the best implementation possible.

On a linux server, the script published here http://www.jinnko.org/2015/03/ocsp-stapling-with-haproxy.html, does the job (thank you, Jinn!):

#!/bin/sh -e

# Get an OSCP response from the certificates OCSP issuer for use
# with HAProxy, then reload HAProxy if there have been updates.

# Path to certificates
PEMSDIR=/etc/pki/pems

# Path to log output to
LOGDIR=/var/log/haproxy

# Create the log path if it doesn't already exist
[ -d ${LOGDIR} ] || mkdir ${LOGDIR}
UPDATED=0

cd ${PEMSDIR}
for pem in *.pem; do
    echo "= $(date)" >> ${LOGDIR}/${pem}.log

    # Get the OCSP URL from the certificate
    ocsp_url=$(openssl x509 -noout -ocsp_uri -in $pem)

    # Extract the hostname from the OCSP URL
    ocsp_host=$(echo $ocsp_url | cut -d/ -f3)

    # Only process the certificate if we have a .issuer file
    if [ -r ${pem}.issuer ]; then

        # Request the OCSP response from the issuer and store it
        openssl ocsp \
            -issuer ${pem}.issuer \
            -cert ${pem} \
            -url ${ocsp_url} \
            -header Host ${ocsp_host} \
            -respout ${pem}.ocsp >> ${LOGDIR}/${pem}.log 2>&1
    fi
    UPDATED=$(( $UPDATED + 1 ))
done

if [ $UPDATED -gt 0 ]; then
    echo "= $(date) - Updated $UPDATED OCSP responses" >> ${LOGDIR}/${pem}.log
    service haproxy reload > ${LOGDIR}/service-reload.log 2>&1
else
    echo "= $(date) - No updates" >> ${LOGDIR}/${pem}.log
fi

Everything above the last six lines does generate the .ocsp-file. While I do get this to work on linux, I do see logs but no output on freebsd. This may be a permissions issue, but I did not find out. My next try was to run this on a linux server and send the output to pfsense via scp. This does work and as I run two HA pfsense servers, this would at least lead to the .ocsp-file just being generated once for both servers. The only change required is to set the PEMSDIR variable correctly (i. e. /var/etc/haproxy in pfsense, I think). With a startssl-certificate, I had to put the intermediate certificate in one file with the server certificate (which is good anyway) and the intermediate certificate as well as the CA certificate in the issuer file.

The next challenge is to restart haproxy after placing the .ocsp-file. What does seem to work is "/usr/pbi/rc.d/haproxy onerestart", albeit not overly well and not as quickly as a reload. A reload might avoid interrupting service which would be better but it does not seem to exist. A major problem is then, that the restart clears files from the /var/etc/haproxy-directory. The same happens after restarting haproxy from the console. That renders placing the .ocsp-file there useless, because the file cannot be used without a restart and it does not exist after a restart.

So even with the two modifications (PEMSDIR and reload/restart command) and splitting the script across two servers, it is close to working but not fully there, yet. Can anyone point us to the right direction, please?

Regards,

Michael

PiBa

Just tried that a little and actually seems simpler than i imagined. At least the first part of getting a actual ocsp response. Assuming the file i got as output will actually works. That i might check in the weekend :)

Next part is what happens when the next OCSP response is generated by the CA? Would it be available before the current one expires? Would a 'once a day' cron job be enough to keep it up to date? (Also i dont want a complete restart but want to update it over the unix socket,, but i think that part shouldnt be to difficult..)

Question, i would like to minimize settings required to enable this option.. Would only a OCSP checkbox in the gui near the frontend certificate selection be enough? And perhaps a update frequency on the global settings tab(ive heard of ocsp responses being valid for only a few minutes..).?

michaelschefczyk

Dear PiBa,

Thank you very much!! Even a solution which does not reach the one-click level of simplicity would be a good step as long as one can demonstrate that pfSense HAProxy is practically able to dish out OCSP stapled responses, I think.

In my case (startcom), the OCSP response stays valid for 48 hours. That should be pretty much the norm. Thus, a once a day update would be good. As long as the update does not terminate all https sessions, time of day would not matter. Once shorter durations will become practically relevant, one would not have to react.

As more comfortable template solution using the unix socket for a seamless update is also available: https://github.com/pierky/haproxy-ocsp-stapling-updater That, however, has more prerequisites such as socat. With the script quoted above, one has to produce the .issuer CA-certificate file manually. In my case, that was somehow good because it did allow to handle the "issue" that the intermediate certificate needed to be included there as well. The script available on github by pierky would automate plainly everything. However, I did not find it as easy to really get it to work even on linux.

Regards,

Michael

PiBa

Hello Michael,

Socat is not actually required for using the unix socket. If you look at 'haproxy_socketinfo.inc' the 'function haproxy_socket_command($command)' does it already by using 'native' php.
Still making it all work 'nicely' together is a bit of a challenge but i think i can make that happen :). Though i wont object if you send a github pullrequest before me :).. I'm also just a pfSense 'user' with some programming/scripting skills. (Which i put to use for the haproxy package..)

Greets
PiBa-NL

PiBa

Pull-request send.. https://github.com/pfsense/pfsense-packages/pull/881
Will probably get pulled in a day or perhaps two..

Haproxy-devel package version 0.24
Feedback will be welcome.

michaelschefczyk

Dear PiBa,

wow, superb - thank you very much!! I will wait for version 0.24 showing up and then swith from the regular package to devel. Thereafter, I will report how OCSP works from a user perspective.

Regards,

Michael

PiBa

Ok 0.24 just became available ;D . Anyway OCSP updates every hour (if you want a different time edit the cron job.), and a single checkbox per frontend to enable it.. The pfsense systemlog will show that updates are done if successful.. (php-fpm[56391]: HAProxy Retrieving OCSP for frontend FEvhost1_default..result: Next Update: Jun 1 14:16:30 2015 GMT)

Now a question, do we even need that checkbox per frontend, or would a single setting on the global settings tab suffice, and ocsp be activated for all certificates that have a ocsp url embeded..? Is once a hour to fast? (there is a 5 minute minimum overlap time compiled into haproxy itself iirc..)

And maybe the most important, does the actual ocsp downloading/updating work properly for all kinds of certs you might have available.?

Depending on input from you guys i also intend to commit these changes, to the haproxy-1_5 package in a few weeks, if they work alright..

michaelschefczyk

Dear PiBa,

Thank you for making that big step so quickly! What did work well in my case was moving to the devel-version of HA-Proxy and syncing it among two firewall servers.

The interface is great, also because of its simplicity. I use multiple secondary frontends and I would use OCSP stapling on all of them. So for me, a single checkbox would be enough. As I use the same certificate on multiple primary/secondary frontends, there might be an efficiency gain in not getting the OCSP response multiple times. On the other hand, I do not understand the interaction of selecting OCSP stapling and (a) multiple certificates based on SNI as well as (b) client certificate veryfication enough to be able to judge this.

In my case, I initially get errors, unfotunately. The user interface shows something like "[WARNING] 148/221657 (27989) : Loading '/var/etc/haproxy/WAN1.pem.ocsp': Unable to parse OCSP response. Content will be ignored." upon confirming the changed configuration. In the folder /var/etc/haproxy, the issuer file does stay empty as well as the ocsp file itself. I assume that that should have been filled automatically? Consequently, ssllabs' ssl test does not recognize OCSP stapling being active. The log contains the following lines:

May 29 22:28:18 php-fpm[8545]: HAProxy Retrieving OCSP for frontend WAN1..
May 29 22:28:18 php-fpm[8545]: HAProxy Retrieving OCSP for frontend WAN1..result: 34379330920:error:0906D06C:PEM routines:PEM_read_bio:no start line:/usr/pfSensesrc/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/pem/pem_lib.c:703:Expecting: TRUSTED CERTIFICATE

How can I help to debug this?

Regards,

Michael

PiBa

Hi Michael,

The .issuer file should have been generated and filled automatically. You do have the issuer's root certificate in the pfsense certificate ca's list and its certificate count is 1 or more?

The .ocsp file is downloaded with help of the issuer file, so if the first step result is empty it will not work..

Greets
PiBa-NL

michaelschefczyk

Dear PiBa,

my answer to your question regarding the certificates is yes and no. I use StartCom which has a bit of a collection of root certificates (http://www.startssl.com/certs/).

My Certificate Authority Manager under Certificates contains the certificate selected in HAProxy. That is a class2 SHA2 StartCom certificate. Bundled with that is the sub.class2.server.ca.crt intermediary certificate because that should be submitted to the client (otherwise ssltest will complain). However, the finding below does not change when not bundling the intermediary certificate with the final certificate.

Initially, I had no root certificate in the certificate manager because for use cases prior to OCSP stapling, that was not required (I think). So the certificate was marked external.

I have tried out pretty much all combination of StartCom root certificates in the Certificate Authority Manager under CAs. That includes importing the traditional ca.pem, ca-sha2.pem, ca-g2.pem (not applicable, I think), ca-bundle.pem (excluding the CRLs) and the CA certificates listed previously followed by the intermediary certificate.

In all cases, the final certificate in use stays marked as "external" under issuer. That does also happen when uploading the certificate after the CA data is present. Correspondingly, the number of certificates per CA stays at 0 (unlike with my internal CAs). That could, however, also be a consequence of not having the public CAs private key of course.

What should I try next?

Regards,

Michael

PiBa

Hello Michael,

You could try after creating a config backup if the 'Recalculate certificate chain' on the haproxy settings tab can perhaps 'fix' it.. I havent used that button in a long while though, and besides some obscure test with self signed certs i did several years ago it never seemed necessary anymore with official certs..

p.s. If you visit https://www.ssllabs.com/ssltest/ to test your website, it does not show any warnings about missing intermediate/root certificates as well? Did you append the intermediate pem info to your 'website cert', that might cause the trouble.. not sure though..

Maybe later ill try and get a free startssl cert and check if i can get it to work..

Greets PiBa-NL

michaelschefczyk

Dear PiBa,

in terms of the certificates, I do bundle the intermediary certificate with the final certificate by just appending it after the final certificate when importing it. If I do that, I do get an A+ in ssltest. Not including the intermediary certificate leads to chain issues and therefore an ssltest of B only. I did not find any way to effectively bundle the intermediary certificate with the root certificate. In fact, my final certificate is marked as "external" despite loading the root certificate also. The recalculate certificate chain button does not change this in any way.

If I do use the plain final certificate without the intermediary certificate, I get the following GUI errors after enabling OCSP stapling and trying to activate the frontend:

Errors found while starting haproxy
[ALERT] 149/233340 (91202) : parsing [/var/etc/haproxy_test/haproxy.cfg:67] : 'bind 192.168.0.2:443' : '/var/etc/haproxy_test/WAN2.pem.ocsp' is present and activates OCSP but it is impossible to compute the OCSP certificate ID (maybe the issuer could not be found)'.
[ALERT] 149/233340 (91202) : Error(s) found in configuration file : /var/etc/haproxy_test/haproxy.cfg
[ALERT] 149/233340 (91202) : Proxy 'WAN2-merged': no SSL certificate specified for bind '192.168.0.2:443' at [/var/etc/haproxy_test/haproxy.cfg:67] (use 'crt').
[ALERT] 149/233340 (91202) : Fatal errors found in configuration.

To me that looks like OCSP stapling does not work without the intermediary certificate. Also under those circumstances, the recalculate certificate chain button does not help.

Regards,

Michael

PiBa

Hi Michael,

If you add a file '/usr/local/www/show_cert.php' to pfSense with the following content:

include_once('certs.inc');
include_once('config.lib.inc');
echo '```
';
foreach($config['ca'] as $cert) {
	echo "\n\nCA          : ".$cert['descr'];
	echo "\nCA   Subject: ".cert_get_subject($cert['crt']);
	echo "\nCA   Issuer : " . cert_get_issuer($cert['crt']);
}
foreach($config['cert'] as $cert) {
	echo "\n\nCert          : ".$cert['descr'];
	echo "\nCert Subject: ".cert_get_subject($cert['crt']);
	echo "\nCert Issuer : " . cert_get_issuer($cert['crt']);
}
?>

And then open that page with a webbrowser, does it show a matching issuer & subject text for 'your' and the 'intermediate' certificate ? So something like this ? (yes all other cas and certs will get output as well and the order is not important..).

But as you can see for each cert the issuer in another CA there is the the exact same text for the subject:

Cert          : *.mydomain.tld
Cert Subject: OU=PositiveSSL Wildcard, OU=Domain Control Validated, CN=*.mydomain.tld
Cert Issuer : ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Domain Validation Secure Server CA, C=GB

CA          : COMODO RSA Domain Validation Secure Server CA
CA   Subject: ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Domain Validation Secure Server CA, C=GB
CA   Issuer : ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Certification Authority, C=GB

CA          : COMODO RSA Certification Authority
CA   Subject: ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Certification Authority, C=GB
CA   Issuer : OU=AddTrust External TTP Network, O=AddTrust AB, CN=AddTrust External CA Root, C=SE

CA          : AddTrust External CA Root
CA   Subject: OU=AddTrust External TTP Network, O=AddTrust AB, CN=AddTrust External CA Root, C=SE
CA   Issuer : OU=AddTrust External TTP Network, O=AddTrust AB, CN=AddTrust External CA Root, C=SE

michaelschefczyk

Dear PiBa,

thank you very much - the script was very helpful in focusing my attention to the remaining parameters to vary! With this help, I did get OSCP stapling to work!

The first important step is to not have the StartCom root certificate loaded under CA certificates. That may lead to an extra certificate in the .issuer file (once it does get created) and that will be reported negatively by ssltest. I never used to load the CA certificate, but did so in trying to resolve this.

The second important step is to load the StartCom intermediate certificate under CA certificates. Then, the final certificates are not listed as external but rather as dependents of the intermediary certificate. Previously, I did not detect the need to load intermediate certificates separately - now I do.

The third important step is to load the final certificate under certificates, but not bundled with the intermediate certificate.

Regards,

Michael

PiBa

Hi Michael,

Ok, good we got that part resolved, ill remove the root-ca from the .pem written for use by haproxy that should resolve the ssllab 'chain issue' if root is present in the CA list, will be in 0.25..

Now for the actual OCSP functionality.. Do you think it needs to be changed/improved in any way?

p.s. Check that you can see in pfSense system log that the ocsp responses are resolved every hour.. If not reinstall the whole package a 2nd time.

Any other questions/improvements regarding ocsp or haproxy package? Let me know :D.

Regards,
PiBa-NL

michaelschefczyk

Dear PiBa,

My log shows that every full hour, OSCP data gets collected for all HAProxy front ends configured.

From my point of view, the OCSP stapling implementation is completely fine :).

Also the pfSense HAProxy package is very strong at large. One item that should be on the agenda sometime is to compile HAProxy with ALPN-support, I think, because NPN will be replaced by ALPN next year.

Regards,

Michael

PiBa

Hi Michael,
There doesn't appear to be any build-flag to enable ALPN.. It seems like it should be 'auto-detected' by haproxy when the library supports it. Not sure what i can do there..
I think the openssl library used by FreeBSD ports does not yet contain the define 'TLSEXT_TYPE_application_layer_protocol_negotiation'.. http://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=6f017a8f9db3a79f3a3406cf8d493ccd346db691 Even though that commit is 2 years back..

If you have any idea's i'm open to suggestions.
Greets PiBa-NL