HAProxy 1.5 and OCSP



  • Hi,

    I'd like to add the OCSP functionality to the HAProxy configuration. I know that since version 1.5 this is somewhat possible.

    Does anyone knows how to enable it in Pfsense?

    Thank you.



  • Its not currently possible to configure this using the build in webgui options.. Ive got it on my 'to investigate' list.. But don't think ill be implementing it any time soon..

    To add it a cron job will need to download and reconfigure the new oscp response every X time interval.. It will need to know where to get the oscp responses from and download them. Then put configure them over the unix socket. Its not that haproxy takes care of this by making a few configuration settings…

    If you can send a pullrequest on github for this feature that would be great :D.



  • Yes, I've read some topic on this and this is indeed not that easy I guess.

    For the pull request on github I'm no developer at all (I'm a sysadmin guy), so on this one you can't count on me, sorry :)

    Thanks for your reply anyway.



  • Dear PiBa, dear All,

    It would be really good to have OCSP stapling. This should avoid informing another party about encrypted communications and speed up access.

    Please let us pool ideas about practical workarounds that might get us OCSP stapling without waiting for the best implementation possible.

    On a linux server, the script published here http://www.jinnko.org/2015/03/ocsp-stapling-with-haproxy.html, does the job (thank you, Jinn!):

    #!/bin/sh -e
    
    # Get an OSCP response from the certificates OCSP issuer for use
    # with HAProxy, then reload HAProxy if there have been updates.
    
    # Path to certificates
    PEMSDIR=/etc/pki/pems
    
    # Path to log output to
    LOGDIR=/var/log/haproxy
    
    # Create the log path if it doesn't already exist
    [ -d ${LOGDIR} ] || mkdir ${LOGDIR}
    UPDATED=0
    
    cd ${PEMSDIR}
    for pem in *.pem; do
        echo "= $(date)" >> ${LOGDIR}/${pem}.log
    
        # Get the OCSP URL from the certificate
        ocsp_url=$(openssl x509 -noout -ocsp_uri -in $pem)
    
        # Extract the hostname from the OCSP URL
        ocsp_host=$(echo $ocsp_url | cut -d/ -f3)
    
        # Only process the certificate if we have a .issuer file
        if [ -r ${pem}.issuer ]; then
    
            # Request the OCSP response from the issuer and store it
            openssl ocsp \
                -issuer ${pem}.issuer \
                -cert ${pem} \
                -url ${ocsp_url} \
                -header Host ${ocsp_host} \
                -respout ${pem}.ocsp >> ${LOGDIR}/${pem}.log 2>&1
        fi
        UPDATED=$(( $UPDATED + 1 ))
    done
    
    if [ $UPDATED -gt 0 ]; then
        echo "= $(date) - Updated $UPDATED OCSP responses" >> ${LOGDIR}/${pem}.log
        service haproxy reload > ${LOGDIR}/service-reload.log 2>&1
    else
        echo "= $(date) - No updates" >> ${LOGDIR}/${pem}.log
    fi 
    

    Everything above the last six lines does generate the .ocsp-file. While I do get this to work on linux, I do see logs but no output on freebsd. This may be a permissions issue, but I did not find out. My next try was to run this on a linux server and send the output to pfsense via scp. This does work and as I run two HA pfsense servers, this would at least lead to the .ocsp-file just being generated once for both servers. The only change required is to set the PEMSDIR variable correctly (i. e. /var/etc/haproxy in pfsense, I think). With a startssl-certificate, I had to put the intermediate certificate in one file with the server certificate (which is good anyway) and the intermediate certificate as well as the CA certificate in the issuer file.

    The next challenge is to restart haproxy after placing the .ocsp-file. What does seem to work is "/usr/pbi/rc.d/haproxy onerestart", albeit not overly well and not as quickly as a reload. A reload might avoid interrupting service which would be better but it does not seem to exist. A major problem is then, that the restart clears files from the /var/etc/haproxy-directory. The same happens after restarting haproxy from the console. That renders placing the .ocsp-file there useless, because the file cannot be used without a restart and it does not exist after a restart.

    So even with the two modifications (PEMSDIR and reload/restart command) and splitting the script across two servers, it is close to working but not fully there, yet. Can anyone point us to the right direction, please?

    Regards,

    Michael



  • Just tried that a little and actually seems simpler than i imagined. At least the first part of getting a actual ocsp response. Assuming the file i got as output will actually works. That i might check in the weekend :)

    Next part is what happens when the next OCSP response is generated by the CA? Would it be available before the current one expires? Would a 'once a day' cron job be enough to keep it up to date? (Also i dont want a complete restart but want to update it over the unix socket,, but i think that part shouldnt be to difficult..)

    Question, i would like to minimize settings required to enable this option.. Would only a OCSP checkbox in the gui near the frontend certificate selection be enough? And perhaps a update frequency on the global settings tab(ive heard of ocsp responses being valid for only a few minutes..).?



  • Dear PiBa,

    Thank you very much!! Even a solution which does not reach the one-click level of simplicity would be a good step as long as one can demonstrate that pfSense HAProxy is practically able to dish out OCSP stapled responses, I think.

    In my case (startcom), the OCSP response stays valid for 48 hours. That should be pretty much the norm. Thus, a once a day update would be good. As long as the update does not terminate all https sessions, time of day would not matter. Once shorter durations will become practically relevant, one would not have to react.

    As more comfortable template solution using the unix socket for a seamless update is also available: https://github.com/pierky/haproxy-ocsp-stapling-updater  That, however, has more prerequisites such as socat. With the script quoted above, one has to produce the .issuer CA-certificate file manually. In my case, that was somehow good because it did allow to handle the "issue" that the intermediate certificate needed to be included there as well. The script available on github by pierky would automate plainly everything. However, I did not find it as easy to really get it to work even on linux.

    Regards,

    Michael



  • Hello Michael,

    Socat is not actually required for using the unix socket. If you look at 'haproxy_socketinfo.inc' the 'function haproxy_socket_command($command)' does it already by using 'native' php.
    Still making it all work 'nicely' together is a bit of a challenge but i think i can make that happen :). Though i wont object if you send a github pullrequest before me :).. I'm also just a pfSense 'user' with some programming/scripting skills. (Which i put to use for the haproxy package..)

    Greets
    PiBa-NL



  • Pull-request send.. https://github.com/pfsense/pfsense-packages/pull/881
    Will probably get pulled in a day or perhaps two..

    Haproxy-devel package version 0.24
    Feedback will be welcome.



  • Dear PiBa,

    wow, superb - thank you very much!! I will wait for version 0.24 showing up and then swith from the regular package to devel. Thereafter, I will report how OCSP works from a user perspective.

    Regards,

    Michael



  • Ok 0.24 just became available ;D . Anyway OCSP updates every hour (if you want a different time edit the cron job.), and a single checkbox per frontend to enable it.. The pfsense systemlog will show that updates are done if successful.. (php-fpm[56391]: HAProxy Retrieving OCSP for frontend FEvhost1_default..result: Next Update: Jun 1 14:16:30 2015 GMT)

    Now a question, do we even need that checkbox per frontend, or would a single setting on the global settings tab suffice, and ocsp be activated for all certificates that have a ocsp url embeded..? Is once a hour to fast? (there is a 5 minute minimum overlap time compiled into haproxy itself iirc..)

    And maybe the most important, does the actual ocsp downloading/updating work properly for all kinds of certs you might have available.?

    Depending on input from you guys i also intend to commit these changes, to the haproxy-1_5 package in a few weeks, if they work alright..



  • Dear PiBa,

    Thank you for making that big step so quickly! What did work well in my case was moving to the devel-version of HA-Proxy and syncing it among two firewall servers.

    The interface is great, also because of its simplicity. I use multiple secondary frontends and I would use OCSP stapling on all of them. So for me, a single checkbox would be enough. As I use the same certificate on multiple primary/secondary frontends, there might be an efficiency gain in not getting the OCSP response multiple times. On the other hand, I do not understand the interaction of selecting OCSP stapling and (a) multiple certificates based on SNI as well as (b) client certificate veryfication enough to be able to judge this.

    In my case, I initially get errors, unfotunately. The user interface shows something like "[WARNING] 148/221657 (27989) : Loading '/var/etc/haproxy/WAN1.pem.ocsp': Unable to parse OCSP response. Content will be ignored." upon confirming the changed configuration. In the folder /var/etc/haproxy, the issuer file does stay empty as well as the ocsp file itself. I assume that that should have been filled automatically? Consequently, ssllabs' ssl test does not recognize OCSP stapling being active. The log contains the following lines:

    May 29 22:28:18 php-fpm[8545]: HAProxy Retrieving OCSP for frontend WAN1..
    May 29 22:28:18 php-fpm[8545]: HAProxy Retrieving OCSP for frontend WAN1..result: 34379330920:error:0906D06C:PEM routines:PEM_read_bio:no start line:/usr/pfSensesrc/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/pem/pem_lib.c:703:Expecting: TRUSTED CERTIFICATE

    How can I help to debug this?

    Regards,

    Michael



  • Hi Michael,

    The .issuer file should have been generated and filled automatically. You do have the issuer's root certificate in the pfsense certificate ca's list and its certificate count is 1 or more?

    The .ocsp file is downloaded with help of the issuer file, so if the first step result is empty it will not work..

    Greets
    PiBa-NL



  • Dear PiBa,

    my answer to your question regarding the certificates is yes and no. I use StartCom which has a bit of a collection of root certificates (http://www.startssl.com/certs/).

    My Certificate Authority Manager under Certificates contains the certificate selected in HAProxy. That is a class2 SHA2 StartCom certificate. Bundled with that is the sub.class2.server.ca.crt intermediary certificate because that should be submitted to the client (otherwise ssltest will complain). However, the finding below does not change when not bundling the intermediary certificate with the final certificate.

    Initially, I had no root certificate in the certificate manager because for use cases prior to OCSP stapling, that was not required (I think). So the certificate was marked external.

    I have tried out pretty much all combination of StartCom root certificates in the Certificate Authority Manager under CAs. That includes importing the traditional ca.pem, ca-sha2.pem, ca-g2.pem (not applicable, I think), ca-bundle.pem (excluding the CRLs) and the CA certificates listed previously followed by the intermediary certificate.

    In all cases, the final certificate in use stays marked as "external" under issuer. That does also happen when uploading the certificate after the CA data is present. Correspondingly, the number of certificates per CA stays at 0 (unlike with my internal CAs). That could, however, also be a consequence of not having the public CAs private key of course.

    What should I try next?

    Regards,

    Michael



  • Hello Michael,

    You could try after creating a config backup if the 'Recalculate certificate chain' on the haproxy settings tab can perhaps 'fix' it.. I havent used that button in a long while though, and besides some obscure test with self signed certs i did several years ago it never seemed necessary anymore with official certs..

    p.s. If you visit https://www.ssllabs.com/ssltest/ to test your website, it does not show any warnings about missing intermediate/root certificates as well? Did you append the intermediate pem info to your 'website cert', that might cause the trouble.. not sure though..

    Maybe later ill try and get a free startssl cert and check if i can get it to work..

    Greets PiBa-NL



  • Dear PiBa,

    in terms of the certificates, I do bundle the intermediary certificate with the final certificate by just appending it after the final certificate when importing it. If I do that, I do get an A+ in ssltest. Not including the intermediary certificate leads to chain issues and therefore an ssltest of B only. I did not find any way to effectively bundle the intermediary certificate with the root certificate. In fact, my final certificate is marked as "external" despite loading the root certificate also. The recalculate certificate chain button does not change this in any way.

    If I do use the plain final certificate without the intermediary certificate, I get the following GUI errors after enabling OCSP stapling and trying to activate the frontend:

    Errors found while starting haproxy
    [ALERT] 149/233340 (91202) : parsing [/var/etc/haproxy_test/haproxy.cfg:67] : 'bind 192.168.0.2:443' : '/var/etc/haproxy_test/WAN2.pem.ocsp' is present and activates OCSP but it is impossible to compute the OCSP certificate ID (maybe the issuer could not be found)'.
    [ALERT] 149/233340 (91202) : Error(s) found in configuration file : /var/etc/haproxy_test/haproxy.cfg
    [ALERT] 149/233340 (91202) : Proxy 'WAN2-merged': no SSL certificate specified for bind '192.168.0.2:443' at [/var/etc/haproxy_test/haproxy.cfg:67] (use 'crt').
    [ALERT] 149/233340 (91202) : Fatal errors found in configuration.

    To me that looks like OCSP stapling does not work without the intermediary certificate. Also under those circumstances, the recalculate certificate chain button does not help.

    Regards,

    Michael



  • Hi Michael,

    If you add a file '/usr/local/www/show_cert.php' to pfSense with the following content:

    include_once('certs.inc');
    include_once('config.lib.inc');
    echo '```
    ';
    foreach($config['ca'] as $cert) {
    	echo "\n\nCA          : ".$cert['descr'];
    	echo "\nCA   Subject: ".cert_get_subject($cert['crt']);
    	echo "\nCA   Issuer : " . cert_get_issuer($cert['crt']);
    }
    foreach($config['cert'] as $cert) {
    	echo "\n\nCert          : ".$cert['descr'];
    	echo "\nCert Subject: ".cert_get_subject($cert['crt']);
    	echo "\nCert Issuer : " . cert_get_issuer($cert['crt']);
    }
    ?>
    

    And then open that page with a webbrowser, does it show a matching issuer & subject text for 'your' and the 'intermediate' certificate ? So something like this ? (yes all other cas and certs will get output as well and the order is not important..).

    But as you can see for each cert the issuer in another CA there is the the exact same text for the subject:

    Cert          : *.mydomain.tld
    Cert Subject: OU=PositiveSSL Wildcard, OU=Domain Control Validated, CN=*.mydomain.tld
    Cert Issuer : ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Domain Validation Secure Server CA, C=GB
    
    CA          : COMODO RSA Domain Validation Secure Server CA
    CA   Subject: ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Domain Validation Secure Server CA, C=GB
    CA   Issuer : ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Certification Authority, C=GB
    
    CA          : COMODO RSA Certification Authority
    CA   Subject: ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Certification Authority, C=GB
    CA   Issuer : OU=AddTrust External TTP Network, O=AddTrust AB, CN=AddTrust External CA Root, C=SE
    
    CA          : AddTrust External CA Root
    CA   Subject: OU=AddTrust External TTP Network, O=AddTrust AB, CN=AddTrust External CA Root, C=SE
    CA   Issuer : OU=AddTrust External TTP Network, O=AddTrust AB, CN=AddTrust External CA Root, C=SE
    
    


  • Dear PiBa,

    thank you very much - the script was very helpful in focusing my attention to the remaining parameters to vary! With this help, I did get OSCP stapling to work!

    The first important step is to not have the StartCom root certificate loaded under CA certificates. That may lead to an extra certificate in the .issuer file (once it does get created) and that will be reported negatively by ssltest. I never used to load the CA certificate, but did so in trying to resolve this.

    The second important step is to load the StartCom intermediate certificate under CA certificates. Then, the final certificates are not listed as external but rather as dependents of the intermediary certificate. Previously, I did not detect the need to load intermediate certificates separately - now I do.

    The third important step is to load the final certificate under certificates, but not bundled with the intermediate certificate.

    Regards,

    Michael



  • Hi Michael,

    Ok, good we got that part resolved, ill remove the root-ca from the .pem written for use by haproxy that should resolve the ssllab 'chain issue' if root is present in the CA list, will be in 0.25..

    Now for the actual OCSP functionality.. Do you think it needs to be changed/improved in any way?

    p.s. Check that you can see in pfSense system log that the ocsp responses are resolved every hour.. If not reinstall the whole package a 2nd time.

    Any other questions/improvements regarding ocsp or haproxy package? Let me know :D.

    Regards,
    PiBa-NL



  • Dear PiBa,

    My log shows that every full hour, OSCP data gets collected for all HAProxy front ends configured.

    From my point of view, the OCSP stapling implementation is completely fine  :).

    Also the pfSense HAProxy package is very strong at large. One item that should be on the agenda sometime is to compile HAProxy with ALPN-support, I think, because NPN will be replaced by ALPN next year.

    Regards,

    Michael



  • Hi Michael,
    There doesn't appear to be any build-flag to enable ALPN.. It seems like it should be 'auto-detected' by haproxy when the library supports it. Not sure what i can do there..
    I think the openssl library used by FreeBSD ports does not yet contain the define 'TLSEXT_TYPE_application_layer_protocol_negotiation'.. http://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=6f017a8f9db3a79f3a3406cf8d493ccd346db691 Even though that commit is 2 years back..

    If you have any idea's i'm open to suggestions.
    Greets PiBa-NL



  • Thanks Michael for blazing the path for me. I also have an SSL cert from StartSSL and just appended it to my domain cert. When I decided to try OCSP today I got the same error. I'll work through what you did tonight hopefully

    And thanks as always PiBa for your continued work on this awesome plugin!



  • Dear PiBa, dear all,

    While I did get OCSP to work some months ago, this is no longer so, unfortunately. I am uncertain, if issues in the haproxy-1_5 stable package towards 0.30 are causing this or if it is issues at the certificate provider.

    In the qualys ssl test, I see two certificate paths, one leading to the SHA1 CA cert and the other leading the the SHA256 CA cert. While OSCP fails at different levels of the certificate chain, problems are always identical for both paths.

    Yesterday, qualys ssl test did yield (for the SHA1 path as a example):

    1 Sent by server analyticum.com
    Fingerprint: ad2b82ac78767c062980f20fc403facdff0a8ddf
    RSA 4096 bits (e 65537) / SHA256withRSA
    OCSP ERROR: Request failed with HTTP status: 500 [http://ocsp.startssl.com/sub/class2/server/ca]

    2 Sent by server StartCom Class 2 Primary Intermediate Server CA
    Fingerprint: 064969b7f4d6a74fd098be59d379fae429a906fb
    RSA 2048 bits (e 65537) / SHA256withRSA

    3 In trust store StartCom Certification Authority  Self-signed
    Fingerprint: 3e2bf7f2031b96f38ce6c4d8a85d3e2d58476a0f
    RSA 4096 bits (e 65537) / SHA1withRSA
    Weak or insecure signature, but no impact on root certificate

    Today (with no changes on my side), OCSP fails for the intermediary certificate instead of the server certificate:

    1 Sent by server analyticum.com
    Fingerprint: ad2b82ac78767c062980f20fc403facdff0a8ddf
    RSA 4096 bits (e 65537) / SHA256withRSA

    2 Sent by server StartCom Class 2 Primary Intermediate Server CA
    Fingerprint: 064969b7f4d6a74fd098be59d379fae429a906fb
    RSA 2048 bits (e 65537) / SHA256withRSA
    OCSP ERROR: Request failed with HTTP status: 500 [http://ocsp.startssl.com/ca]

    3 In trust store StartCom Certification Authority  Self-signed
    Fingerprint: 3e2bf7f2031b96f38ce6c4d8a85d3e2d58476a0f
    RSA 4096 bits (e 65537) / SHA1withRSA
    Weak or insecure signature, but no impact on root certificate

    Can someone please point me at how to avoid this?

    Regarding the future issue of making ALPN available, I did read that openssl 1.0.2 would be required (https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation - I could not find the release notes quoted under (6) in the wikipedia article or anything like this in the openssl release notes which I did find, though.) while pfSense does provide OpenSSL 1.0.1l-freebsd 15 Jan 2015 now. Does anyone know whom to notify to change that well before the end of 2015?

    Regards,

    Michael



  • Hi Michael

    Looks to me like the issue is on the startssl side?: https://forum.startcom.org/viewtopic.php?f=15&p=22411&sid=fe8d39fe277387117a769acfe59cf534

    With haproxy 0.30 my ocsp still works properly for me with certificate from another CA..

    As for openssl 1.0.2, its a bit tricky.. haproxy is build agains openssl from ports.. If you currently check haproxy -vv output youl see that newest dev4 package is using showing the folowing..:

    Built with OpenSSL version : OpenSSL 1.0.2d 9 Jul 2015
    Running on OpenSSL version : OpenSSL 1.0.1m 19 Mar 2015 (VERSIONS DIFFER!)
    
    

    Ive not noticed problems caused by this yet.. Also i have discussed with pfSense devs, and its deemed difficult to fix until the new pkgng package system will be used..

    Best regards
    PiBa-NL



  • Dear PiBa,

    Thank you very much. The OCSP issue was indeed caused by a problem in StartCom's infrastructure, as StartCom's friendly certmaster did confirm. It is gone at present and I hope that it will remain OK in the future.

    Thanks for following up with the OpenSSL issue. It would be good to move from npn to alpn before major providers and their browsers (Chrome) will make the switch, probably by the end of 2015, if they stick to their announcements. Nevertheless all of us should practically take it as it is.

    Regards,

    Michael