Patching/Upgrading OpenSSL



  • A whacking great hole in OpenSSL 1.0.1 was announced today.  Can a fixed version be installed on pfSense 2.1 and/or 2.1.1?  How is this done?



  • You may be able to fix it by hand, but a better approach would be to roll out 2.1.2 with the fix right now, and encourage updates.



  • pfSense 2.1.1 ships with OpenSSL 0.9.8g which is not vulnerable and neither are earlier versions of pfSense, either.

    Edit: pfSense 2.1 and 2.1.1 also ship with OpenSSL 1.0.1 which is vulnerable.



  • @drees:

    pfSense 2.1.1 ships with OpenSSL 0.9.8g which is not vulnerable and neither are earlier versions of pfSense, either.

    You have half the story right, but you need the whole story … freebsd ships with 0.9.8g, but pfsense adds 1.0.1e in /usr/local/bin and uses that version for most pfsense needs.

    So, yes, I'd say pfsense is vulnerable.



  • My apologies, you are correct. I have confirmed that both pfSense 2.1 and 2.1.1 shipped with a vulnerable version of openssl.



  • pfsense 2.1.1 is vulnerable (webserver for sure, openvpn, ipsec and other crypto might be using libssl also).



  • So 2.0.3 is not vulnerable.  The new features that were advertised for that version show openssl 0.9.8y and searches for libssl turn up 'OpenSSL 0.9.8y 5 Feb 2013'.  Any chance some packages put their own version of openssl in?  I don't use many packages so I don't know.

    https://doc.pfsense.org/index.php/2.0.3_New_Features_and_Changes



  • 2.1 and 2.1.1 ARE vulnerable.



  • Large number of reputable projects already updated and fixed the issue.

    Longer pfSense is without an update, lower reputation it will be… I think, it is unacceptable for such security-centric project not to fix the problem ASAP.


  • Banned

    @dgcom:

    Large number of reputable projects already updated and fixed the issue.

    Longer pfSense is without an update, lower reputation it will be… I think, it is unacceptable for such security-centric project not to fix the problem ASAP.

    You know, this requires a full new release… It's not a matter of compiling/packaging one package and typing one command in a package manager, unlike those other reputable projects.



  • @doktornotor:

    You know, this requires a full new release… It's not a matter of compiling/packaging one package and typing one command in a package manager, unlike those other reputable projects.

    Yes, I know. But I haven't heard that it is being worked on yet… And pfsense.org site is still vulnerable, so someone can exploit and put a rogue download mirror... :(

    There is also a package to apply custom system patches - may be that can be used in interim to update openssl?

    And not having mechanism for quickly applying such security fixes is not really good approach to a security application, you know...


  • Banned

    Nice rant. However, you should focus that rant at the openssl guys who introduced this brainfart in the first place… Additionally, reading the source code, the entire openssl thing needs a rewrite from scratch.


  • Moderator

    "And pfsense.org site is still vulnerable, so someone can exploit and put a rogue download mirror"

    How should that magic happen? To set up a rogue download mirror one doesn't only need to have the certificate and key to let this site seem like the legit pfsense.org domain, but you also had to inject and poison dns queries for all the hosts to actually use your new rogue download mirror. So problem with SSL aside, let's not leap to wild speculations here and throw around FUD.

    Greets



  • @doktornotor:

    Nice rant. However, you should focus that rant at the openssl guys who introduced this brainfart in the first place… Additionally, reading the source code, the entire openssl thing needs a rewrite from scratch.

    That strategy isn't helpful for fixing the product that we're here to discuss, which is still vulnerable.

    I really like pfSense, but the response here is discouraging. This is being treated quite seriously and with high priority almost everywhere else, but here it seems the users are being lectured for even asking about a fix.



  • @JeGr:

    "And pfsense.org site is still vulnerable, so someone can exploit and put a rogue download mirror"

    How should that magic happen? To set up a rogue download mirror one doesn't only need to have the certificate and key to let this site seem like the legit pfsense.org domain, but you also had to inject and poison dns queries for all the hosts to actually use your new rogue download mirror. So problem with SSL aside, let's not leap to wild speculations here and throw around FUD.

    Greets

    "for all the hosts to actually use your new rogue download mirror"
    That doesn't have to be all the hosts, just the one you want to target.
    This is, of course, theoretical, however still a real security risk and I would not call this FUD - I know, similar things happen to other sites in the past.



  • There is a point to lack of response, as some of us have Gold support subscriptions. Granted, $99 a year is not a lot, but since the community has confirmed both 2.1 and 2.1.1 (the release that I just received a notification about) is vulnerable, you'd think at the very least we'd have a notification urging users not to upgrade to the latest version.

    Furthermore, http://filippo.io/Heartbleed/ shows that the portal.pfsense.org site is vulnerable. Not that this is particularly the same as the PfSense platform, but it does have some minor things such as credit card payment information…

    To be fair, 66% of the Internet is broken, but you'd expect some kind of announcement on the issue by now. Luckily, I'm a fairly lazy sysadmin and all my PfSense implementations are sitting at 2.0.3 :)



  • Has anyone successfully patched this by hand?


  • Banned

    No, of course not.@SectorNine50:

    Has anyone successfully patched this by hand?

    No. Patching by hand would require access to the t0p-s3cr3t pfSense©®™-tools  repo…



  • Quote from JimP 1 hour ago:

    It's known and we're already working on it.

    on RedMine issue https://redmine.pfsense.org/issues/3588
    It would be much easier if those managing this took 1 minute to post here so people would know that the fix is being worked on. That would save all this banter in the forum.



  • FYI… for those that haven't checked that bug report in the past hour... There is a reply to the redmine bug report as of around an hour ago saying it is already known and being worked on.



  • Thanks for letting the rest of us know!


  • Banned

    This looks like a huge heap of shit hitting the fan, BTW… Yahoo or Lastpass.com users might want to read these:



  • And this GitHub commit to mark 2.1.2-RELEASE shows that a new set of images is about to be built:
    https://github.com/pfsense/pfsense/commit/def5d042c9c51874e662a1175e94ce948224f2a7



  • @phil.davis:

    And this GitHub commit to mark 2.1.2-RELEASE shows that a new set of images is about to be built:
    https://github.com/pfsense/pfsense/commit/def5d042c9c51874e662a1175e94ce948224f2a7

    Sweet!!

    I just got done manually patching a OpenVPN server.  What a mess.  This isn't PfSense's fault.



  • 2.1.2 release is almost finished building. The PBI build run is in-progress, but takes quite a while. We had to restart it a couple times, taking longer. I'll post the location of 2.1.2 release before it's official and posted to the mirrors, just after I do basic testing. I'm cloning a package builder to another server in hopes I can get that building faster as it's going to take possibly another 12 hours to finish the PBI package run at its current rate and it's been running for around 7 hours now.

    Don't try to patch or upgrade OpenSSL, you'll more than likely just break things. Each PBI has its own copy, plus the base system.

    In the mean time:

    1. Don't open your web interface to the entire Internet. That's the bulk of the risk here.
    2. if you're using OpenVPN without "TLS Authentication" enabled, you might want to close that off from the Internet. I haven't seen a proof of concept against OpenVPN, but in theory minus TLS auth it's probably vulnerable. The default settings we use aren't vulnerable.
    3. there are a variety of other packages that depend on OpenSSL, but generally not ones that listen in a fashion that's exploitable in the way HTTPS is.


  • @cmb:

    2.1.2 release is almost finished building. The PBI build run is in-progress, but takes quite a while.

    This is great news, thanks for the effort to get it out quickly.

    @cmb:

    Don't try to patch or upgrade OpenSSL, you'll more than likely just break things. Each PBI has its own copy, plus the base system.

    Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(



  • @dgcom:

    @cmb:

    2.1.2 release is almost finished building. The PBI build run is in-progress, but takes quite a while.

    This is great news, thanks for the effort to get it out quickly.

    @cmb:

    Don't try to patch or upgrade OpenSSL, you'll more than likely just break things. Each PBI has its own copy, plus the base system.

    Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

    Well, this is a BIG one as it ties into alot of the inner workings so I don't blame them for having to entirely rebuild the whole thing to ensure every piece are updated.  Hang in there.



  • @El:

    I really like pfSense, but the response here is discouraging. This is being treated quite seriously and with high priority almost everywhere else, but here it seems the users are being lectured for even asking about a fix.

    There's nothing to think it's not high priority here. We have easily upwards of 30 man hours into this at this point. Our job here isn't nearly as simple as OSes for instance, who can throw out a minor update rather easily and not have to rebuild a slew of stuff. We've been working continually on this since it was publicly known.

    It appears we'll beat every other similar-scoped open source distribution, and probably nearly all similar commercial appliances, in issuing fixes. There are already several updated PBIs uploaded with the patched openssl.



  • @dgcom:

    Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

    It depends on the issue. This one's difficult because it requires recompiling a slew of PBIs, which is very time consuming, and building an entire release. If it were as simple as "here's a file, copy this and you're fixed", we would have provided that file 24 hours ago. It's also not something that's exploitable in the common uses of the system and where people are using reasonable security practices. Spend a lot more time looking at your web servers, mail servers, etc. right now, and follow my recommendations in the post above.



  • @cmb:

    @dgcom:

    Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

    It depends on the issue. This one's difficult because it requires recompiling a slew of PBIs, which is very time consuming, and building an entire release. If it were as simple as "here's a file, copy this and you're fixed", we would have provided that file 24 hours ago. It's also not something that's exploitable in the common uses of the system and where people are using reasonable security practices. Spend a lot more time looking at your web servers, mail servers, etc. right now, and follow my recommendations in the post above.

    I perfectly understand implications of this particular issue, and yes - it is not just a matter of replacing openssl executable… What I am saying is that recompiling everything is not very efficient. But, I guess, you know your product :)

    As for "not something that's exploitable in the common uses" - my major concern is web UI, which I would think is exposed often for remote management and packages like stunnel, HAProxy, Squid... Whatever deals with SSL frontend in any way - shouldn't build system be smart enough to recompile only if dependencies changed?

    I, personally, do not run anything, based on recent versions of openssl - except pfSense.


  • Moderator

    @cmb:

    It appears we'll beat every other similar-scoped open source distribution, and probably nearly all similar commercial appliances, in issuing fixes. There are already several updated PBIs uploaded with the patched openssl.

    And you are right with that. We just openend a case with Juniper about the SSL Issue and besides transferring ownerships of the ticket and "assuring us, it is already worked on and they are hard at work" nothing happened. So yes, it perhaps would have been nice to post a short statement here (or the forums generally) but the bug tracker showed, they were working on it. That's more than I can say about Juniper and others. ;)

    Thanks and kudos to the team.



  • moderator edit: don't do this, see the 2.1.2 link below.



  • @dgcom:

    @doktornotor:

    You know, this requires a full new release… It's not a matter of compiling/packaging one package and typing one command in a package manager, unlike those other reputable projects.

    Yes, I know. But I haven't heard that it is being worked on yet… And pfsense.org site is still vulnerable, so someone can exploit and put a rogue download mirror... :(

    There is also a package to apply custom system patches - may be that can be used in interim to update openssl?

    And not having mechanism for quickly applying such security fixes is not really good approach to a security application, you know...

    Forgive my ignorance, but why could a user not simply grab the new OpenSSL version, from FreeBSD and compile?


  • Banned

    @jsheed_sa:

    Forgive my ignorance, but why could a user not simply grab the new OpenSSL version, from FreeBSD and compile?

    I'd suggest reading the entire thread since it's already answered.



  • edit: scratch that. Another security fix was made earlier that the first build of 2.1.2 just missed. It's rebuilding.



  • @cmb:

    @dgcom:

    Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

    It depends on the issue. This one's difficult because it requires recompiling a slew of PBIs, which is very time consuming, and building an entire release. If it were as simple as "here's a file, copy this and you're fixed", we would have provided that file 24 hours ago. It's also not something that's exploitable in the common uses of the system and where people are using reasonable security practices. Spend a lot more time looking at your web servers, mail servers, etc. right now, and follow my recommendations in the post above.

    Forgive my ignorance, but why could a user not simply grab the new OpenSSL version, from FreeBSD and compile?



  • @jsheed_sa:

    @cmb:

    @dgcom:

    Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

    It depends on the issue. This one's difficult because it requires recompiling a slew of PBIs, which is very time consuming, and building an entire release. If it were as simple as "here's a file, copy this and you're fixed", we would have provided that file 24 hours ago. It's also not something that's exploitable in the common uses of the system and where people are using reasonable security practices. Spend a lot more time looking at your web servers, mail servers, etc. right now, and follow my recommendations in the post above.

    Forgive my ignorance, but why could a user not simply grab the new OpenSSL version, from FreeBSD and compile?

    Yes he could and did… but he apparently did something really bad according to this thread (still, his firewalls seem to be working fine with this approach, perhaps
    because he doesn't use pbi packages on them).



  • Packages on pfSense are in pbi-packages since 2.1 which means that each package that uses OpenSSL or other dependencies will have their own copy of the binaries. So if you have stunnel, squid or other packages that also use OpenSSL, the package pbi-package will have to be recompiled.



  • @developers: Thanks for working that fast on this problem.



  • Are there going to be updated 2.2 snapshots released to address this issue?