Patching/Upgrading OpenSSL

El Scorcho

Thanks for letting the rest of us know!

doktornotor

This looks like a huge heap of shit hitting the fan, BTW… Yahoo or Lastpass.com users might want to read these:

https://twitter.com/markloman/status/453502888447586304
https://twitter.com/markloman/status/453504687934017536

phil.davis

And this GitHub commit to mark 2.1.2-RELEASE shows that a new set of images is about to be built:
https://github.com/pfsense/pfsense/commit/def5d042c9c51874e662a1175e94ce948224f2a7

Darkk

@phil.davis:

And this GitHub commit to mark 2.1.2-RELEASE shows that a new set of images is about to be built:
https://github.com/pfsense/pfsense/commit/def5d042c9c51874e662a1175e94ce948224f2a7

Sweet!!

I just got done manually patching a OpenVPN server. What a mess. This isn't PfSense's fault.

cmb

2.1.2 release is almost finished building. The PBI build run is in-progress, but takes quite a while. We had to restart it a couple times, taking longer. I'll post the location of 2.1.2 release before it's official and posted to the mirrors, just after I do basic testing. I'm cloning a package builder to another server in hopes I can get that building faster as it's going to take possibly another 12 hours to finish the PBI package run at its current rate and it's been running for around 7 hours now.

Don't try to patch or upgrade OpenSSL, you'll more than likely just break things. Each PBI has its own copy, plus the base system.

In the mean time:

Don't open your web interface to the entire Internet. That's the bulk of the risk here.
if you're using OpenVPN without "TLS Authentication" enabled, you might want to close that off from the Internet. I haven't seen a proof of concept against OpenVPN, but in theory minus TLS auth it's probably vulnerable. The default settings we use aren't vulnerable.
there are a variety of other packages that depend on OpenSSL, but generally not ones that listen in a fashion that's exploitable in the way HTTPS is.

dgcom

@cmb:

2.1.2 release is almost finished building. The PBI build run is in-progress, but takes quite a while.

This is great news, thanks for the effort to get it out quickly.

@cmb:

Don't try to patch or upgrade OpenSSL, you'll more than likely just break things. Each PBI has its own copy, plus the base system.

Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

Darkk

@dgcom:

@cmb:

2.1.2 release is almost finished building. The PBI build run is in-progress, but takes quite a while.

This is great news, thanks for the effort to get it out quickly.

@cmb:

Don't try to patch or upgrade OpenSSL, you'll more than likely just break things. Each PBI has its own copy, plus the base system.

Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

Well, this is a BIG one as it ties into alot of the inner workings so I don't blame them for having to entirely rebuild the whole thing to ensure every piece are updated. Hang in there.

cmb

@El:

I really like pfSense, but the response here is discouraging. This is being treated quite seriously and with high priority almost everywhere else, but here it seems the users are being lectured for even asking about a fix.

There's nothing to think it's not high priority here. We have easily upwards of 30 man hours into this at this point. Our job here isn't nearly as simple as OSes for instance, who can throw out a minor update rather easily and not have to rebuild a slew of stuff. We've been working continually on this since it was publicly known.

It appears we'll beat every other similar-scoped open source distribution, and probably nearly all similar commercial appliances, in issuing fixes. There are already several updated PBIs uploaded with the patched openssl.

cmb

@dgcom:

Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

It depends on the issue. This one's difficult because it requires recompiling a slew of PBIs, which is very time consuming, and building an entire release. If it were as simple as "here's a file, copy this and you're fixed", we would have provided that file 24 hours ago. It's also not something that's exploitable in the common uses of the system and where people are using reasonable security practices. Spend a lot more time looking at your web servers, mail servers, etc. right now, and follow my recommendations in the post above.

dgcom

@cmb:

@dgcom:

Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

It depends on the issue. This one's difficult because it requires recompiling a slew of PBIs, which is very time consuming, and building an entire release. If it were as simple as "here's a file, copy this and you're fixed", we would have provided that file 24 hours ago. It's also not something that's exploitable in the common uses of the system and where people are using reasonable security practices. Spend a lot more time looking at your web servers, mail servers, etc. right now, and follow my recommendations in the post above.

I perfectly understand implications of this particular issue, and yes - it is not just a matter of replacing openssl executable… What I am saying is that recompiling everything is not very efficient. But, I guess, you know your product :)

As for "not something that's exploitable in the common uses" - my major concern is web UI, which I would think is exposed often for remote management and packages like stunnel, HAProxy, Squid... Whatever deals with SSL frontend in any way - shouldn't build system be smart enough to recompile only if dependencies changed?

I, personally, do not run anything, based on recent versions of openssl - except pfSense.

JeGr

@cmb:

It appears we'll beat every other similar-scoped open source distribution, and probably nearly all similar commercial appliances, in issuing fixes. There are already several updated PBIs uploaded with the patched openssl.

And you are right with that. We just openend a case with Juniper about the SSL Issue and besides transferring ownerships of the ticket and "assuring us, it is already worked on and they are hard at work" nothing happened. So yes, it perhaps would have been nice to post a short statement here (or the forums generally) but the bug tracker showed, they were working on it. That's more than I can say about Juniper and others. ;)

Thanks and kudos to the team.

casper4242

moderator edit: don't do this, see the 2.1.2 link below.

jsheed_sa

@dgcom:

@doktornotor:

You know, this requires a full new release… It's not a matter of compiling/packaging one package and typing one command in a package manager, unlike those other reputable projects.

Yes, I know. But I haven't heard that it is being worked on yet… And pfsense.org site is still vulnerable, so someone can exploit and put a rogue download mirror... :(

There is also a package to apply custom system patches - may be that can be used in interim to update openssl?

And not having mechanism for quickly applying such security fixes is not really good approach to a security application, you know...

Forgive my ignorance, but why could a user not simply grab the new OpenSSL version, from FreeBSD and compile?

doktornotor

@jsheed_sa:

Forgive my ignorance, but why could a user not simply grab the new OpenSSL version, from FreeBSD and compile?

I'd suggest reading the entire thread since it's already answered.

cmb

edit: scratch that. Another security fix was made earlier that the first build of 2.1.2 just missed. It's rebuilding.

jsheed_sa

@cmb:

@dgcom:

Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

It depends on the issue. This one's difficult because it requires recompiling a slew of PBIs, which is very time consuming, and building an entire release. If it were as simple as "here's a file, copy this and you're fixed", we would have provided that file 24 hours ago. It's also not something that's exploitable in the common uses of the system and where people are using reasonable security practices. Spend a lot more time looking at your web servers, mail servers, etc. right now, and follow my recommendations in the post above.

Forgive my ignorance, but why could a user not simply grab the new OpenSSL version, from FreeBSD and compile?

casper4242

@jsheed_sa:

@cmb:

@dgcom:

Any plans to get such patches easier in the future? As much as I like to hope, but I do not think this is the last one :(

It depends on the issue. This one's difficult because it requires recompiling a slew of PBIs, which is very time consuming, and building an entire release. If it were as simple as "here's a file, copy this and you're fixed", we would have provided that file 24 hours ago. It's also not something that's exploitable in the common uses of the system and where people are using reasonable security practices. Spend a lot more time looking at your web servers, mail servers, etc. right now, and follow my recommendations in the post above.

Forgive my ignorance, but why could a user not simply grab the new OpenSSL version, from FreeBSD and compile?

Yes he could and did… but he apparently did something really bad according to this thread (still, his firewalls seem to be working fine with this approach, perhaps
because he doesn't use pbi packages on them).

fragged

Packages on pfSense are in pbi-packages since 2.1 which means that each package that uses OpenSSL or other dependencies will have their own copy of the binaries. So if you have stunnel, squid or other packages that also use OpenSSL, the package pbi-package will have to be recompiled.

Hobby-Student

@developers: Thanks for working that fast on this problem.

ingmthompson

Are there going to be updated 2.2 snapshots released to address this issue?