Patching/Upgrading OpenSSL

CNLiberal · Apr 10, 2014, 9:15 PM

Cool. I'm proceeding. Poor ESF servers getting hammered hard.

EDIT: Finished. Looks good!

Fmstrat · Apr 10, 2014, 9:17 PM

Hi all, I see the founder tweeted that keys may be OK:

https://twitter.com/neelmehta/status/453625474879471616

However I also know that people have pulled keys from FreeBSD. For OpenVPN users, do we need to swap our keys if we were using PFsense 2.1.x?

Thanks.

dannyman · Apr 10, 2014, 10:13 PM

Thanks for getting 2.1.2 out!

This will be my first upgrade. I assume the firewall will need to reboot. Any quick advice or how long an upgrade takes, how long is the downtime, maybe these questions are already covered somewhere?

Does the secondary firewall upgrade when it sees it peer upgrade or do I trigger that manually?

I've announced 5+ minutes downtime in a 1 hour maintenance window. Looks like there's a backup / rollback option so I'm hoping that even if I hit a snag I won't be dead in the water!

Thanks,
-danny

BBcan177 · Apr 10, 2014, 10:17 PM

Upgraded one of my boxes from 2.0.3 to 2.1.2.

No issues to report! ;)

Except that Suricata lost all of its interface settings. Looks like a new install of that package. Snort was also installed and it came back 100%

Great Work Guys!

BBcan177 · Apr 10, 2014, 10:19 PM

What testing method is recommended to test the Bleeding Heart issue in pfSense?

Bluejay · Apr 10, 2014, 10:46 PM

I registered just to comment on what an amazing job you guys did in getting this patch out so quickly. There are commercial routers costing hundreds of dollars (if not thousands) that I imagine will be vulnerable for weeks…

th3r3isnospoon · Apr 10, 2014, 10:55 PM

Thank you Jim Pingle, Chris Buechler, all the other devs and the community members that helped make this happen! It's much appreciated!

I will be buying a gold subscription to show my thanks!

-th3r3isnospoon

phong · Apr 11, 2014, 12:02 AM

@dannyman:

Any quick advice or how long an upgrade takes, how long is the downtime

It took my pfSense box about 5-10 minutes for the whole process (which was without any problems).
The downtime was only 1 minute 30 seconds (the time I couldn't access internet or ping the pfSense box). My pfSense box is quite an old PC (Pentium 3 or 4).

Thank you admins for your work on this serious bug.

jasonlitka · Apr 11, 2014, 12:23 AM

@dannyman:

Thanks for getting 2.1.2 out!

This will be my first upgrade. I assume the firewall will need to reboot. Any quick advice or how long an upgrade takes, how long is the downtime, maybe these questions are already covered somewhere?

Does the secondary firewall upgrade when it sees it peer upgrade or do I trigger that manually?

I've announced 5+ minutes downtime in a 1 hour maintenance window. Looks like there's a backup / rollback option so I'm hoping that even if I hit a snag I won't be dead in the water!

Thanks,
-danny

You're using CARP on all interfaces? If so, the downtime will be basically zero.

Upgrade the backup first. Once it's back up and running disable CARP on the first box to force a failover. Once you're sure things are working on the backup, upgrade the primary. Once the upgrade is done everyone will fail back to the primary automatically.

gertk · Apr 11, 2014, 6:54 AM

What about windows OpenVPN client programs? I have to upgrade them too or not?

doktornotor · Apr 11, 2014, 7:13 AM

@gertk:

What about windows OpenVPN client programs? I have to upgrade them too or not?

https://forum.pfsense.org/index.php?topic=74948.0

Atomicslave · Apr 11, 2014, 12:19 PM

Just want to say a big Thanks to all the Devs who put time in on this, upgraded my home firewall last night and my work one this morning.

Thanks again guys Awesome Job!

Ecnerwal · Apr 11, 2014, 12:59 PM

Happily my "system where VPN runs and security matters" is still on 2.0.3, as I've I've had continuing nags with 2.1 where I did try it that made me not upgrade that one since 2.0.3 is nice and stable and I can put up with complaints from the users of the other system more easily as I try to work the 2.1 issues out there (or wait for them to be worked out, more likely.) Hopefully 2.1.2 will be better than 2.1, and if not, I guess there's always heading back to 2.0.3 on both systems.

adam65535 · Apr 11, 2014, 2:04 PM

@Jason:

You're using CARP on all interfaces? If so, the downtime will be basically zero.

Upgrade the backup first. Once it's back up and running disable CARP on the first box to force a failover. Once you're sure things are working on the backup, upgrade the primary. Once the upgrade is done everyone will fail back to the primary automatically.

Unless you are dependent on packages being installed like some routing daemons, squid, etc. The current disable carp feature on the carp status page does not survive reboots. The primary will take over immediately after the reboot before those get re-installed fully so there will be some downtime in that case. If you don't depend on any packages then it should be 0 downtime.

Someone has submitted a feature in 2.2 version that allows you to put a carp member in maintenance mode so that the member you put into maintenance mode will not take over when it boots up unless carp detects that the other member is down.

NOYB · Apr 11, 2014, 6:12 PM

See several OpenVPN connections from an unauthorized location in the firewall logs this morning.
Is this someone trying to exploit the Heartbleed bug?


Name:    hn.kd.ny.adsl
Address:  182.118.48.87

pfSense OpenVPN Log


Apr 11 07:01:20 openvpn[9425]: TCP connection established with [AF_INET]182.118.48.87:53956 
Apr 11 07:01:20 openvpn[9425]: 182.118.48.87:53956 WARNING: Bad encapsulated packet length from peer (5635), which must be > 0 and <= 1560 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possible active attack on the TCP link -- [Attempting restart...] 
Apr 11 07:01:20 openvpn[9425]: 182.118.48.87:53956 Connection reset, restarting [0]

BBcan177 · Apr 11, 2014, 6:19 PM

http://kb.bothunter.net/ipInfo/nowait.php?IP=182.118.48.87
http://www.ipvoid.com/scan/182.118.48.87/
http://www.senderbase.org/lookup/ip/?search_string=182.118.48.87
https://www.iblocklist.com/search.php?string=182.118.48.87

If you use pfBlocker you could block known malicious ip addresses.

adam65535 · Apr 11, 2014, 9:42 PM

@NOYB:

See several OpenVPN connections from an unauthorized location in the firewall logs this morning.
Is this someone trying to exploit the Heartbleed bug?


Name:    hn.kd.ny.adsl
Address:  182.118.48.87

pfSense OpenVPN Log


Apr 11 07:01:20 openvpn[9425]: TCP connection established with [AF_INET]182.118.48.87:53956 
Apr 11 07:01:20 openvpn[9425]: 182.118.48.87:53956 WARNING: Bad encapsulated packet length from peer (5635), which must be > 0 and <= 1560 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possible active attack on the TCP link -- [Attempting restart...] 
Apr 11 07:01:20 openvpn[9425]: 182.118.48.87:53956 Connection reset, restarting [0]

You get that error any time someone even just connects to the openvpn TCP port with a regular browser. It is not an indication specifically of a heartbeat attack. Certainly someone is scanning for SSL ports though for whatever reason.

EDIT: changed openssl to openvpn

Doktor Jones · Apr 12, 2014, 6:17 AM

Just to double-check – if I install pfSense 2.1.2, my OpenVPN server is safely patched correct?

What command could I run to check the version of the libraries (e.g. /usr/local/lib/libssl.so) to ensure it's a patched version?

Thanks!

doktornotor · Apr 12, 2014, 9:28 AM

Just in case people still think their private keys are safe:

Can You Get Private SSL Keys Using Heartbleed?
The Results of the CloudFlare Challenge

@CloudFlare:

While it takes effort, it is possible to extract private SSL keys. The challenge was solved by Software Engineer Fedor Indutny and Ilkka Mattila at NCSC-FL roughly 9 hours after the challenge was first published. Fedor sent 2.5 million requests over the course of the day and Ilkka sent around 100K requests. Our recommendation based on this finding is that everyone reissue and revoke their private keys. CloudFlare has accelerated this effort on behalf of the customers whose SSL keys we manage.

stephenw10 · Apr 12, 2014, 11:37 AM

@Doktor:

What command could I run to check the version of the libraries (e.g. /usr/local/lib/libssl.so) to ensure it's a patched version?

To check the OpenSSL version:

[2.1.2-RELEASE][root@pfsense.fire.box]/root(1): /usr/local/bin/openssl version
OpenSSL 1.0.1g 7 Apr 2014

To check the libraries use the strings command and grep for Openssl. E.g.
@https://redmine.pfsense.org/issues/3588:

[2.1.1-RELEASE][xxx@xxx]/root(1): strings /usr/pbi/haproxy-devel-amd64/lib/libssl.so.8 | grep OpenSSL
OpenSSLDie
SSLv2 part of OpenSSL 1.0.1g 7 Apr 2014
SSLv3 part of OpenSSL 1.0.1g 7 Apr 2014
TLSv1 part of OpenSSL 1.0.1g 7 Apr 2014
DTLSv1 part of OpenSSL 1.0.1g 7 Apr 2014
OpenSSL 1.0.1g 7 Apr 2014

However I don't appear to have the strings command in my Nano install here at home.

Steve

Edit: You can add strings as part of binutils using pkg_add but it's quite big.

[2.1.2-RELEASE][root@pfsense.fire.box]/root(11): strings /usr/local/lib/libssl.so | grep OpenSSL
OpenSSLDie
SSLv2 part of OpenSSL 1.0.1g 7 Apr 2014
SSLv3 part of OpenSSL 1.0.1g 7 Apr 2014
TLSv1 part of OpenSSL 1.0.1g 7 Apr 2014
DTLSv1 part of OpenSSL 1.0.1g 7 Apr 2014
OpenSSL 1.0.1g 7 Apr 2014