HAProxy and openssl - the 'unrescue'

justme2

All,

Now that the dust has settled... To say that the upgrade to 2.7.1 had some issues would be a vast understatement. Specifically poking at HAProxy where the issue when a litany of sites are front-ended via HAProxy acting as SSL Offload and it's also doing SSL Bridging with 'low grade' certs (on the backend) to keep an application/service/site "operating as expected" on TCP/443.... Needless to say, it was a calamity of outages. Some applications/services just simply do not like to be operated on 443 without a cert and when you try to shift them to 80 become equally problematic. Especially as things continue to appear to shift towards "an administrator's expertise and decision is insufficient" mantra - its a fight between spaces.

Notably missing was anything in the release notes speaking to HAPorxy and potential impacts. Throughout years of dealing with pfSense, while I may have some tough criticisms - in general, its been extremely reliable and upgrades have gone with little to no issue. This is the first time where there's been a "sting" (and it was a vicious one to say the least) as a result of an upgrade. Spending an entire weekend trying to "sort out" issues was not on the docket. Raising some significant critiques:

In some environments, no matter how much you test or try to simulate something - there may be some measure of inevitable "won't know until it's done in production". Which means that if there's a problem - being able to rollback or re-install the prior version and restore the configuration (along with ALL packages) needs to be viable. Just "because" there's a new version shouldn't exclude the ability (in some way) to revert to return to prior state. This is not just highly problematic, but troubling. Even if one excluded some issues where things didn't properly transition during upgrade - when you have a litany of apps/sites that all break, sometimes a rollback is the right move. Making a rollback impossible is irrational (tried to see if this could be done and there appears to be no saving grace there...). This provides necessary time to determine root cause(s) and addressing in a non-emergency cadence in order to then attempt the upgrade again. This simply cannot be understated. Suggesting that it's highly irritating that an upgrade cannot be reverted, which forces a significant amount of change, including demands of other groups/personnel to "fix" other applications that weren't "broken" prior to upgrade - you get the point. It used to be the case that you could specify the prior release (often marked as "deprecated" in the list) and that certainly wasn't the case here. It raises the question of how to download all packages for a given version so that at least for pleasure of chewing up some disk space - there might be a means to get back to a previously working state.
The premise of offloading SSL on the front end with high grade certs and using low grade certs on the SSL bridging - makes sense. Notion is that HAProxy is handling the crypto, helping to ensure that what traverses passes reasonable muster and then one (or more) WAFs being employed for further scrutiny. As nothing in the release notes suggested (or even hinted at) the premise that HAProxy wouldn't have been able to function "as prior" for SSL Bridging, really created some havoc. Could one make the argument that HAProxy depends on openssl - sure. Could one make the argument that the point of HAProxy is to also handle SSL Bridging to maintain 'expected operational architectures' - absolutely. Simply put, an application that is "front-ended" (publicly) via HAProxy should not be forced to conform to 'high grade' certs on the backend. Makes absolutely no sense and that's the point. One could make the argument of transitioning to TCP/80 - sure, however that means customization(s) that veer from 'standard' or expected vs. keeping things as TCP/443 with low grade certs to avoid such customization(s). Again, the "backend" cert(s) have ZERO value on the public side and high grade certs only serve to increase processing overhead - for nothing, absolute ZERO value and only serving to increase loads all around. The entire point of SSL bridging is to keep things "as expected" (or necessary if there's not another option) and as such, use low grade certs to minimize overhead as much as possible.
Given openssl.org's information on deprecated items - it appears that various things (including SHA1 for example) while being strongly discouraged (which they should be) are still available if compiled to include the legacy providers. Will absolutely agree that a 'push' to move forward is good but that needs to come with greater clarity and recognition that there are some use cases where the "legacy" artifacts have legitimate reason to exist, even if included as "unsupported". Would rather have seen a push to 'check/force usage of a modern variant under the following mantra, examples: net new configuration (such as a VPN, etc.) or when attempting to 'change' (checked as part of save) to an existing configuration. Thus enabling existing configurations to prevail so that nothing breaks - at least providing sufficient opportunity to remediate. It's a much more pleasant experience to be denied the ability to 'change' something that is currently working or create new vs. breaking things as though to prove a point. Being able to support albeit "legacy" certs for the backend of HAProxy should not be a problem. This issue should not be an issue - again, for things that are behind something like HAProxy.
What should have happened (my $0.02, could be wrong) is that in the system configuration section - an option is present for "public facing" certificates to be allowed/denied as a policy configuration. That, one can easily stand behind 110% as it enables administrators to transition (or not) without having to risk various production applications. The heavy handedness involved with the SSL presents a very daft tone towards legitimate use cases -and- that shouldn't be the case. Suggest it was an "A" for effort to try to get users to transition to better security practices, but an "F-" on execution as it completely ignored legitimate, reasonable and expected use cases.

While some other outages were experienced, including some cases of having to perform entire rebuilds (kernel "disappeared" and couldn't be booted), they paled in comparison to the HAProxy issue.

Thanks!