HAProxy 1.5 and OCSP

michaelschefczyk

Dear PiBa,

Thank you very much!! Even a solution which does not reach the one-click level of simplicity would be a good step as long as one can demonstrate that pfSense HAProxy is practically able to dish out OCSP stapled responses, I think.

In my case (startcom), the OCSP response stays valid for 48 hours. That should be pretty much the norm. Thus, a once a day update would be good. As long as the update does not terminate all https sessions, time of day would not matter. Once shorter durations will become practically relevant, one would not have to react.

As more comfortable template solution using the unix socket for a seamless update is also available: https://github.com/pierky/haproxy-ocsp-stapling-updater That, however, has more prerequisites such as socat. With the script quoted above, one has to produce the .issuer CA-certificate file manually. In my case, that was somehow good because it did allow to handle the "issue" that the intermediate certificate needed to be included there as well. The script available on github by pierky would automate plainly everything. However, I did not find it as easy to really get it to work even on linux.

Regards,

Michael

PiBa

Hello Michael,

Socat is not actually required for using the unix socket. If you look at 'haproxy_socketinfo.inc' the 'function haproxy_socket_command($command)' does it already by using 'native' php.
Still making it all work 'nicely' together is a bit of a challenge but i think i can make that happen :). Though i wont object if you send a github pullrequest before me :).. I'm also just a pfSense 'user' with some programming/scripting skills. (Which i put to use for the haproxy package..)

Greets
PiBa-NL

PiBa

Pull-request send.. https://github.com/pfsense/pfsense-packages/pull/881
Will probably get pulled in a day or perhaps two..

Haproxy-devel package version 0.24
Feedback will be welcome.

michaelschefczyk

Dear PiBa,

wow, superb - thank you very much!! I will wait for version 0.24 showing up and then swith from the regular package to devel. Thereafter, I will report how OCSP works from a user perspective.

Regards,

Michael

PiBa

Ok 0.24 just became available ;D . Anyway OCSP updates every hour (if you want a different time edit the cron job.), and a single checkbox per frontend to enable it.. The pfsense systemlog will show that updates are done if successful.. (php-fpm[56391]: HAProxy Retrieving OCSP for frontend FEvhost1_default..result: Next Update: Jun 1 14:16:30 2015 GMT)

Now a question, do we even need that checkbox per frontend, or would a single setting on the global settings tab suffice, and ocsp be activated for all certificates that have a ocsp url embeded..? Is once a hour to fast? (there is a 5 minute minimum overlap time compiled into haproxy itself iirc..)

And maybe the most important, does the actual ocsp downloading/updating work properly for all kinds of certs you might have available.?

Depending on input from you guys i also intend to commit these changes, to the haproxy-1_5 package in a few weeks, if they work alright..

michaelschefczyk

Dear PiBa,

Thank you for making that big step so quickly! What did work well in my case was moving to the devel-version of HA-Proxy and syncing it among two firewall servers.

The interface is great, also because of its simplicity. I use multiple secondary frontends and I would use OCSP stapling on all of them. So for me, a single checkbox would be enough. As I use the same certificate on multiple primary/secondary frontends, there might be an efficiency gain in not getting the OCSP response multiple times. On the other hand, I do not understand the interaction of selecting OCSP stapling and (a) multiple certificates based on SNI as well as (b) client certificate veryfication enough to be able to judge this.

In my case, I initially get errors, unfotunately. The user interface shows something like "[WARNING] 148/221657 (27989) : Loading '/var/etc/haproxy/WAN1.pem.ocsp': Unable to parse OCSP response. Content will be ignored." upon confirming the changed configuration. In the folder /var/etc/haproxy, the issuer file does stay empty as well as the ocsp file itself. I assume that that should have been filled automatically? Consequently, ssllabs' ssl test does not recognize OCSP stapling being active. The log contains the following lines:

May 29 22:28:18 php-fpm[8545]: HAProxy Retrieving OCSP for frontend WAN1..
May 29 22:28:18 php-fpm[8545]: HAProxy Retrieving OCSP for frontend WAN1..result: 34379330920:error:0906D06C:PEM routines:PEM_read_bio:no start line:/usr/pfSensesrc/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/pem/pem_lib.c:703:Expecting: TRUSTED CERTIFICATE

How can I help to debug this?

Regards,

Michael

PiBa

Hi Michael,

The .issuer file should have been generated and filled automatically. You do have the issuer's root certificate in the pfsense certificate ca's list and its certificate count is 1 or more?

The .ocsp file is downloaded with help of the issuer file, so if the first step result is empty it will not work..

Greets
PiBa-NL

michaelschefczyk

Dear PiBa,

my answer to your question regarding the certificates is yes and no. I use StartCom which has a bit of a collection of root certificates (http://www.startssl.com/certs/).

My Certificate Authority Manager under Certificates contains the certificate selected in HAProxy. That is a class2 SHA2 StartCom certificate. Bundled with that is the sub.class2.server.ca.crt intermediary certificate because that should be submitted to the client (otherwise ssltest will complain). However, the finding below does not change when not bundling the intermediary certificate with the final certificate.

Initially, I had no root certificate in the certificate manager because for use cases prior to OCSP stapling, that was not required (I think). So the certificate was marked external.

I have tried out pretty much all combination of StartCom root certificates in the Certificate Authority Manager under CAs. That includes importing the traditional ca.pem, ca-sha2.pem, ca-g2.pem (not applicable, I think), ca-bundle.pem (excluding the CRLs) and the CA certificates listed previously followed by the intermediary certificate.

In all cases, the final certificate in use stays marked as "external" under issuer. That does also happen when uploading the certificate after the CA data is present. Correspondingly, the number of certificates per CA stays at 0 (unlike with my internal CAs). That could, however, also be a consequence of not having the public CAs private key of course.

What should I try next?

Regards,

Michael

PiBa

Hello Michael,

You could try after creating a config backup if the 'Recalculate certificate chain' on the haproxy settings tab can perhaps 'fix' it.. I havent used that button in a long while though, and besides some obscure test with self signed certs i did several years ago it never seemed necessary anymore with official certs..

p.s. If you visit https://www.ssllabs.com/ssltest/ to test your website, it does not show any warnings about missing intermediate/root certificates as well? Did you append the intermediate pem info to your 'website cert', that might cause the trouble.. not sure though..

Maybe later ill try and get a free startssl cert and check if i can get it to work..

Greets PiBa-NL

michaelschefczyk

Dear PiBa,

in terms of the certificates, I do bundle the intermediary certificate with the final certificate by just appending it after the final certificate when importing it. If I do that, I do get an A+ in ssltest. Not including the intermediary certificate leads to chain issues and therefore an ssltest of B only. I did not find any way to effectively bundle the intermediary certificate with the root certificate. In fact, my final certificate is marked as "external" despite loading the root certificate also. The recalculate certificate chain button does not change this in any way.

If I do use the plain final certificate without the intermediary certificate, I get the following GUI errors after enabling OCSP stapling and trying to activate the frontend:

Errors found while starting haproxy
[ALERT] 149/233340 (91202) : parsing [/var/etc/haproxy_test/haproxy.cfg:67] : 'bind 192.168.0.2:443' : '/var/etc/haproxy_test/WAN2.pem.ocsp' is present and activates OCSP but it is impossible to compute the OCSP certificate ID (maybe the issuer could not be found)'.
[ALERT] 149/233340 (91202) : Error(s) found in configuration file : /var/etc/haproxy_test/haproxy.cfg
[ALERT] 149/233340 (91202) : Proxy 'WAN2-merged': no SSL certificate specified for bind '192.168.0.2:443' at [/var/etc/haproxy_test/haproxy.cfg:67] (use 'crt').
[ALERT] 149/233340 (91202) : Fatal errors found in configuration.

To me that looks like OCSP stapling does not work without the intermediary certificate. Also under those circumstances, the recalculate certificate chain button does not help.

Regards,

Michael

PiBa

Hi Michael,

If you add a file '/usr/local/www/show_cert.php' to pfSense with the following content:

include_once('certs.inc');
include_once('config.lib.inc');
echo '```
';
foreach($config['ca'] as $cert) {
	echo "\n\nCA          : ".$cert['descr'];
	echo "\nCA   Subject: ".cert_get_subject($cert['crt']);
	echo "\nCA   Issuer : " . cert_get_issuer($cert['crt']);
}
foreach($config['cert'] as $cert) {
	echo "\n\nCert          : ".$cert['descr'];
	echo "\nCert Subject: ".cert_get_subject($cert['crt']);
	echo "\nCert Issuer : " . cert_get_issuer($cert['crt']);
}
?>

And then open that page with a webbrowser, does it show a matching issuer & subject text for 'your' and the 'intermediate' certificate ? So something like this ? (yes all other cas and certs will get output as well and the order is not important..).

But as you can see for each cert the issuer in another CA there is the the exact same text for the subject:

Cert          : *.mydomain.tld
Cert Subject: OU=PositiveSSL Wildcard, OU=Domain Control Validated, CN=*.mydomain.tld
Cert Issuer : ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Domain Validation Secure Server CA, C=GB

CA          : COMODO RSA Domain Validation Secure Server CA
CA   Subject: ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Domain Validation Secure Server CA, C=GB
CA   Issuer : ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Certification Authority, C=GB

CA          : COMODO RSA Certification Authority
CA   Subject: ST=Greater Manchester, O=COMODO CA Limited, L=Salford, CN=COMODO RSA Certification Authority, C=GB
CA   Issuer : OU=AddTrust External TTP Network, O=AddTrust AB, CN=AddTrust External CA Root, C=SE

CA          : AddTrust External CA Root
CA   Subject: OU=AddTrust External TTP Network, O=AddTrust AB, CN=AddTrust External CA Root, C=SE
CA   Issuer : OU=AddTrust External TTP Network, O=AddTrust AB, CN=AddTrust External CA Root, C=SE

michaelschefczyk

Dear PiBa,

thank you very much - the script was very helpful in focusing my attention to the remaining parameters to vary! With this help, I did get OSCP stapling to work!

The first important step is to not have the StartCom root certificate loaded under CA certificates. That may lead to an extra certificate in the .issuer file (once it does get created) and that will be reported negatively by ssltest. I never used to load the CA certificate, but did so in trying to resolve this.

The second important step is to load the StartCom intermediate certificate under CA certificates. Then, the final certificates are not listed as external but rather as dependents of the intermediary certificate. Previously, I did not detect the need to load intermediate certificates separately - now I do.

The third important step is to load the final certificate under certificates, but not bundled with the intermediate certificate.

Regards,

Michael

PiBa

Hi Michael,

Ok, good we got that part resolved, ill remove the root-ca from the .pem written for use by haproxy that should resolve the ssllab 'chain issue' if root is present in the CA list, will be in 0.25..

Now for the actual OCSP functionality.. Do you think it needs to be changed/improved in any way?

p.s. Check that you can see in pfSense system log that the ocsp responses are resolved every hour.. If not reinstall the whole package a 2nd time.

Any other questions/improvements regarding ocsp or haproxy package? Let me know :D.

Regards,
PiBa-NL

michaelschefczyk

Dear PiBa,

My log shows that every full hour, OSCP data gets collected for all HAProxy front ends configured.

From my point of view, the OCSP stapling implementation is completely fine :).

Also the pfSense HAProxy package is very strong at large. One item that should be on the agenda sometime is to compile HAProxy with ALPN-support, I think, because NPN will be replaced by ALPN next year.

Regards,

Michael

PiBa

Hi Michael,
There doesn't appear to be any build-flag to enable ALPN.. It seems like it should be 'auto-detected' by haproxy when the library supports it. Not sure what i can do there..
I think the openssl library used by FreeBSD ports does not yet contain the define 'TLSEXT_TYPE_application_layer_protocol_negotiation'.. http://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=6f017a8f9db3a79f3a3406cf8d493ccd346db691 Even though that commit is 2 years back..

If you have any idea's i'm open to suggestions.
Greets PiBa-NL

josh4trunks

Thanks Michael for blazing the path for me. I also have an SSL cert from StartSSL and just appended it to my domain cert. When I decided to try OCSP today I got the same error. I'll work through what you did tonight hopefully

And thanks as always PiBa for your continued work on this awesome plugin!

michaelschefczyk

Dear PiBa, dear all,

While I did get OCSP to work some months ago, this is no longer so, unfortunately. I am uncertain, if issues in the haproxy-1_5 stable package towards 0.30 are causing this or if it is issues at the certificate provider.

In the qualys ssl test, I see two certificate paths, one leading to the SHA1 CA cert and the other leading the the SHA256 CA cert. While OSCP fails at different levels of the certificate chain, problems are always identical for both paths.

Yesterday, qualys ssl test did yield (for the SHA1 path as a example):

1 Sent by server analyticum.com
Fingerprint: ad2b82ac78767c062980f20fc403facdff0a8ddf
RSA 4096 bits (e 65537) / SHA256withRSA
OCSP ERROR: Request failed with HTTP status: 500 [http://ocsp.startssl.com/sub/class2/server/ca]

2 Sent by server StartCom Class 2 Primary Intermediate Server CA
Fingerprint: 064969b7f4d6a74fd098be59d379fae429a906fb
RSA 2048 bits (e 65537) / SHA256withRSA

3 In trust store StartCom Certification Authority Self-signed
Fingerprint: 3e2bf7f2031b96f38ce6c4d8a85d3e2d58476a0f
RSA 4096 bits (e 65537) / SHA1withRSA
Weak or insecure signature, but no impact on root certificate

Today (with no changes on my side), OCSP fails for the intermediary certificate instead of the server certificate:

1 Sent by server analyticum.com
Fingerprint: ad2b82ac78767c062980f20fc403facdff0a8ddf
RSA 4096 bits (e 65537) / SHA256withRSA

2 Sent by server StartCom Class 2 Primary Intermediate Server CA
Fingerprint: 064969b7f4d6a74fd098be59d379fae429a906fb
RSA 2048 bits (e 65537) / SHA256withRSA
OCSP ERROR: Request failed with HTTP status: 500 [http://ocsp.startssl.com/ca]

3 In trust store StartCom Certification Authority Self-signed
Fingerprint: 3e2bf7f2031b96f38ce6c4d8a85d3e2d58476a0f
RSA 4096 bits (e 65537) / SHA1withRSA
Weak or insecure signature, but no impact on root certificate

Can someone please point me at how to avoid this?

Regarding the future issue of making ALPN available, I did read that openssl 1.0.2 would be required (https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation - I could not find the release notes quoted under (6) in the wikipedia article or anything like this in the openssl release notes which I did find, though.) while pfSense does provide OpenSSL 1.0.1l-freebsd 15 Jan 2015 now. Does anyone know whom to notify to change that well before the end of 2015?

Regards,

Michael

PiBa

Hi Michael

Looks to me like the issue is on the startssl side?: https://forum.startcom.org/viewtopic.php?f=15&p=22411&sid=fe8d39fe277387117a769acfe59cf534

With haproxy 0.30 my ocsp still works properly for me with certificate from another CA..

As for openssl 1.0.2, its a bit tricky.. haproxy is build agains openssl from ports.. If you currently check haproxy -vv output youl see that newest dev4 package is using showing the folowing..:

Built with OpenSSL version : OpenSSL 1.0.2d 9 Jul 2015
Running on OpenSSL version : OpenSSL 1.0.1m 19 Mar 2015 (VERSIONS DIFFER!)

Ive not noticed problems caused by this yet.. Also i have discussed with pfSense devs, and its deemed difficult to fix until the new pkgng package system will be used..

Best regards
PiBa-NL

michaelschefczyk

Dear PiBa,

Thank you very much. The OCSP issue was indeed caused by a problem in StartCom's infrastructure, as StartCom's friendly certmaster did confirm. It is gone at present and I hope that it will remain OK in the future.

Thanks for following up with the OpenSSL issue. It would be good to move from npn to alpn before major providers and their browsers (Chrome) will make the switch, probably by the end of 2015, if they stick to their announcements. Nevertheless all of us should practically take it as it is.

Regards,

Michael