Patching/Upgrading OpenSSL

JeGr

@joltman: As I was reporting this, I agree to adams statement. Getting rid of autoconfbackup package before upgrading should completely avoid this. Afterwards it can be installed to it's new version (1.22) again without hiccups. Problem was completely related to the package, not the main distro.

Greets Jens

CNLiberal

Cool. I'm proceeding. Poor ESF servers getting hammered hard.

EDIT: Finished. Looks good!

Fmstrat

Hi all, I see the founder tweeted that keys may be OK:

https://twitter.com/neelmehta/status/453625474879471616

However I also know that people have pulled keys from FreeBSD. For OpenVPN users, do we need to swap our keys if we were using PFsense 2.1.x?

Thanks.

dannyman

Thanks for getting 2.1.2 out!

This will be my first upgrade. I assume the firewall will need to reboot. Any quick advice or how long an upgrade takes, how long is the downtime, maybe these questions are already covered somewhere?

Does the secondary firewall upgrade when it sees it peer upgrade or do I trigger that manually?

I've announced 5+ minutes downtime in a 1 hour maintenance window. Looks like there's a backup / rollback option so I'm hoping that even if I hit a snag I won't be dead in the water!

Thanks,
-danny

BBcan177

Upgraded one of my boxes from 2.0.3 to 2.1.2.

No issues to report! ;)

Except that Suricata lost all of its interface settings. Looks like a new install of that package. Snort was also installed and it came back 100%

Great Work Guys!

BBcan177

What testing method is recommended to test the Bleeding Heart issue in pfSense?

Bluejay

I registered just to comment on what an amazing job you guys did in getting this patch out so quickly. There are commercial routers costing hundreds of dollars (if not thousands) that I imagine will be vulnerable for weeks…

th3r3isnospoon

Thank you Jim Pingle, Chris Buechler, all the other devs and the community members that helped make this happen! It's much appreciated!

I will be buying a gold subscription to show my thanks!

-th3r3isnospoon

phong

@dannyman:

Any quick advice or how long an upgrade takes, how long is the downtime

It took my pfSense box about 5-10 minutes for the whole process (which was without any problems).
The downtime was only 1 minute 30 seconds (the time I couldn't access internet or ping the pfSense box). My pfSense box is quite an old PC (Pentium 3 or 4).

Thank you admins for your work on this serious bug.

jasonlitka

@dannyman:

Thanks for getting 2.1.2 out!

This will be my first upgrade. I assume the firewall will need to reboot. Any quick advice or how long an upgrade takes, how long is the downtime, maybe these questions are already covered somewhere?

Does the secondary firewall upgrade when it sees it peer upgrade or do I trigger that manually?

I've announced 5+ minutes downtime in a 1 hour maintenance window. Looks like there's a backup / rollback option so I'm hoping that even if I hit a snag I won't be dead in the water!

Thanks,
-danny

You're using CARP on all interfaces? If so, the downtime will be basically zero.

Upgrade the backup first. Once it's back up and running disable CARP on the first box to force a failover. Once you're sure things are working on the backup, upgrade the primary. Once the upgrade is done everyone will fail back to the primary automatically.

gertk

What about windows OpenVPN client programs? I have to upgrade them too or not?

doktornotor

@gertk:

What about windows OpenVPN client programs? I have to upgrade them too or not?

https://forum.pfsense.org/index.php?topic=74948.0

Atomicslave

Just want to say a big Thanks to all the Devs who put time in on this, upgraded my home firewall last night and my work one this morning.

Thanks again guys Awesome Job!

Ecnerwal

Happily my "system where VPN runs and security matters" is still on 2.0.3, as I've I've had continuing nags with 2.1 where I did try it that made me not upgrade that one since 2.0.3 is nice and stable and I can put up with complaints from the users of the other system more easily as I try to work the 2.1 issues out there (or wait for them to be worked out, more likely.) Hopefully 2.1.2 will be better than 2.1, and if not, I guess there's always heading back to 2.0.3 on both systems.

adam65535

@Jason:

You're using CARP on all interfaces? If so, the downtime will be basically zero.

Upgrade the backup first. Once it's back up and running disable CARP on the first box to force a failover. Once you're sure things are working on the backup, upgrade the primary. Once the upgrade is done everyone will fail back to the primary automatically.

Unless you are dependent on packages being installed like some routing daemons, squid, etc. The current disable carp feature on the carp status page does not survive reboots. The primary will take over immediately after the reboot before those get re-installed fully so there will be some downtime in that case. If you don't depend on any packages then it should be 0 downtime.

Someone has submitted a feature in 2.2 version that allows you to put a carp member in maintenance mode so that the member you put into maintenance mode will not take over when it boots up unless carp detects that the other member is down.

NOYB

See several OpenVPN connections from an unauthorized location in the firewall logs this morning.
Is this someone trying to exploit the Heartbleed bug?


Name:    hn.kd.ny.adsl
Address:  182.118.48.87

pfSense OpenVPN Log


Apr 11 07:01:20 openvpn[9425]: TCP connection established with [AF_INET]182.118.48.87:53956 
Apr 11 07:01:20 openvpn[9425]: 182.118.48.87:53956 WARNING: Bad encapsulated packet length from peer (5635), which must be > 0 and <= 1560 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possible active attack on the TCP link -- [Attempting restart...] 
Apr 11 07:01:20 openvpn[9425]: 182.118.48.87:53956 Connection reset, restarting [0]

BBcan177

http://kb.bothunter.net/ipInfo/nowait.php?IP=182.118.48.87
http://www.ipvoid.com/scan/182.118.48.87/
http://www.senderbase.org/lookup/ip/?search_string=182.118.48.87
https://www.iblocklist.com/search.php?string=182.118.48.87

If you use pfBlocker you could block known malicious ip addresses.

adam65535

@NOYB:

See several OpenVPN connections from an unauthorized location in the firewall logs this morning.
Is this someone trying to exploit the Heartbleed bug?


Name:    hn.kd.ny.adsl
Address:  182.118.48.87

pfSense OpenVPN Log


Apr 11 07:01:20 openvpn[9425]: TCP connection established with [AF_INET]182.118.48.87:53956 
Apr 11 07:01:20 openvpn[9425]: 182.118.48.87:53956 WARNING: Bad encapsulated packet length from peer (5635), which must be > 0 and <= 1560 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possible active attack on the TCP link -- [Attempting restart...] 
Apr 11 07:01:20 openvpn[9425]: 182.118.48.87:53956 Connection reset, restarting [0]

You get that error any time someone even just connects to the openvpn TCP port with a regular browser. It is not an indication specifically of a heartbeat attack. Certainly someone is scanning for SSL ports though for whatever reason.

EDIT: changed openssl to openvpn

Doktor Jones

Just to double-check – if I install pfSense 2.1.2, my OpenVPN server is safely patched correct?

What command could I run to check the version of the libraries (e.g. /usr/local/lib/libssl.so) to ensure it's a patched version?

Thanks!

doktornotor

Just in case people still think their private keys are safe:

Can You Get Private SSL Keys Using Heartbleed?
The Results of the CloudFlare Challenge

@CloudFlare:

While it takes effort, it is possible to extract private SSL keys. The challenge was solved by Software Engineer Fedor Indutny and Ilkka Mattila at NCSC-FL roughly 9 hours after the challenge was first published. Fedor sent 2.5 million requests over the course of the day and Ilkka sent around 100K requests. Our recommendation based on this finding is that everyone reissue and revoke their private keys. CloudFlare has accelerated this effort on behalf of the customers whose SSL keys we manage.