IPsec fails to renegotiate after loss of a peer

kapara · May 22, 2009, 12:18 AM

This is the version I am running on my Alix Box.

1.2.2
built on Thu Jan 8 23:09:11 EST 2009

jimp · May 28, 2009, 7:22 PM

Great news everybody, it looks like this has been fixed in the development version of ipsec-tools!

I just got ipsec-tools 0.8-alpha20090525+natt to work on my 2.0 test box and things are working as they should.

Previously, DPD was removing the ISAKMP-SA, but not the IPsec-SA that went along with it. Now it appears to be clearing them all out.

Now this is what I'm seeing:

The connection establishes:

2009-05-28 12:51:17: INFO: respond new phase 1 negotiation: x.x.x.41[500]<=>x.x.x.40[500]
2009-05-28 12:51:17: INFO: begin Aggressive mode.
2009-05-28 12:51:17: INFO: received broken Microsoft ID: FRAGMENTATION
2009-05-28 12:51:17: INFO: received Vendor ID: DPD
2009-05-28 12:51:17: NOTIFY: couldn't find the proper pskey, try to get one by the peer's address.
2009-05-28 12:51:17: INFO: ISAKMP-SA established x.x.x.41[500]-x.x.x.40[500] spi:d75d671612ae7e75:07456176d8b6652c
2009-05-28 12:51:17: INFO: received INITIAL-CONTACT
2009-05-28 12:51:18: INFO: respond new phase 2 negotiation: x.x.x.41[500]<=>x.x.x.40[500]
2009-05-28 12:51:18: INFO: IPsec-SA established: ESP x.x.x.41[500]->x.x.x.40[500] spi=118325718(0x70d81d6)
2009-05-28 12:51:18: INFO: IPsec-SA established: ESP x.x.x.41[500]->x.x.x.40[500] spi=224293038(0xd5e70ae)

And then when I unplug the cable:

2009-05-28 12:52:22: INFO: DPD: remote (ISAKMP-SA spi=d75d671612ae7e75:07456176d8b6652c) seems to be dead.
2009-05-28 12:52:22: INFO: purging ISAKMP-SA spi=d75d671612ae7e75:07456176d8b6652c.
2009-05-28 12:52:22: INFO: purged IPsec-SA spi=224293038.
2009-05-28 12:52:22: INFO: purged IPsec-SA spi=118325718.
2009-05-28 12:52:22: INFO: purged ISAKMP-SA spi=d75d671612ae7e75:07456176d8b6652c.
2009-05-28 12:52:23: INFO: ISAKMP-SA deleted x.x.x.41[500]-x.x.x.40[500] spi:d75d671612ae7e75:07456176d8b6652c

And at that point, setkey -D shows nothing in the SA database, which is miles ahead of what I saw previously.

Since I compiled it on an 8-CURRENT box I can't get that same set of ipsec-tools binaries to run on a 1.2.3-RC system. Once I get that going I can confirm it works on my other test cases. I'll have to hunt down another box to (ab)use for more testing, but this looks very promising.

focalguy · May 28, 2009, 7:36 PM

Great news jimp! Thanks for continuing to work on this.

drees · May 28, 2009, 8:21 PM

Awesome! Hopefully this fix can make it into the 1.2.3 release…

jimp · May 28, 2009, 11:36 PM

The test version of ipsec-tools should be making its way into the snapshots fairly soon.

I'll try to post again once I'm sure it's working in a snapshot so that others can try, too.

jimp · May 29, 2009, 3:58 AM

The new ipsec-tools is in the snapshot for 1.2.3-RC1 on FreeBSD 7.2 that can be found here:

http://snapshots.pfsense.org/FreeBSD_RELENG_7_2/pfSense_RELENG_1_2/updates/

So far with the testing I have been able to perform it reestablishes dropped tunnels perfectly with DPD.

I have tested this Full Update:
http://snapshots.pfsense.org/FreeBSD_RELENG_7_2/pfSense_RELENG_1_2/updates/pfSense-Full-Update-1.2.3-20090528-2046.tgz

And this ISO:
http://snapshots.pfsense.org/FreeBSD_RELENG_7_2/pfSense_RELENG_1_2/livecd_installer/pfSense-1.2.3-20090528-2038.iso.gz

And the proper ipsec-tools is in both, and appears to work.

It should be working its way into the 1.2.3-RC1 based on FreeBSD 7.1 overnight as well.

I would appreciate as much testing as anyone can give this. I know this particular bug is fixed but with any change like this there is the potential to break other things. Please test and report back any issues (Especially people who can test NAT-T). Don't be afraid to report apocalyptic failures or anything else that happens.

Jonb · Jun 29, 2009, 8:19 AM

Whoooooooooo so glad this has been looked at. Serious problems this cause me. Just need the traffic shaper fix now.

fastcon68 · Jul 6, 2009, 10:31 PM

Jimp,
Some new for you, if the partner vpn starts a ping the tunnel reconnects. But it if drops from my end it will not estiablish. Thought I let you know. I try t get some information on the typ of firewall they are using.
RC

jimp · Jul 6, 2009, 10:33 PM

@fastcon68:

Jimp,
Some new for you, if the partner vpn starts a ping the tunnel reconnects. But it if drops from my end it will not estiablish. Thought I let you know. I try t get some information on the typ of firewall they are using.
RC

At the moment we're in a holding pattern waiting on ipsec-tools 0.8 to get some fixes in, and I believe that's the way things are going to go.

There are too many of these issues to fix in ipsec-tools 0.7.2 with patches, unfortunately.

fastcon68 · Jul 6, 2009, 10:54 PM

That's cool. I not updating at the momment. I more concerned with a stable enviroment. I can't afford any more down time.

I am in the process of moving and have a ton of stuff going on.

RC

Rockets · Sep 23, 2009, 12:34 PM

@jimp:

At the moment we're in a holding pattern waiting on ipsec-tools 0.8 to get some fixes in, and I believe that's the way things are going to go.

There are too many of these issues to fix in ipsec-tools 0.7.2 with patches, unfortunately.

Now ipsec-tools 0.7.3 is out any thoughts on using it?

jimp · Sep 23, 2009, 12:38 PM

It was even worse off than 0.8 in some respects. We had to stay at 0.7.2 but drop NAT-T to get some semblance of stability.

Rockets · Sep 23, 2009, 12:57 PM

Is IPsec renegotiating properly now in 1.2.3-RC3?

jimp · Sep 23, 2009, 1:08 PM

Yes, it is working now as far as all my tests have shown both in actual tests and in running it at home and having some Internet stability issues. Seems to work fine as far as I can tell.

Rockets · Sep 23, 2009, 1:51 PM

Jimp wher'd I'd find RC3? It's not on the offical mirrors - only RC1. Or is RC3 a current snapshot? I'm using embedded.

jimp · Sep 23, 2009, 1:57 PM

@Rockets:

Jimp wher'd I'd find RC3? It's not on the offical mirrors - only RC1. Or is RC3 a current snapshot? I'm using embedded.

It's only in snapshots at the moment, but there will probably be an "official" cut of RC3 (or perhaps RC4?) before release.

Here are the NanoBSD (new embedded system) snapshots:
http://snapshots.pfsense.org/FreeBSD_RELENG_7_2/pfSense_RELENG_1_2/nanobsd/?C=M;O=D

bkm · Sep 28, 2009, 12:47 PM

I am seeing an issue that seems to be the same. I am testing RC 1.2.3 20090924. I have tunnels set up to two separate sites to a Netopia router on the other end. The tunnels are working when I leave work in the evening. When I get to work in the morning, they are not working. The IPSec status page (SAD) shows that the tunnels are up. If I restart raccoon, the tunnel status goes down. I then ping a site and everything gets renegotiated and it works again.
I currently have a 28800 lifetime for phase 1 and 86400 for phase 2.
I am willing to test a couple things for a day or two if someone has a suggestion. After that I will need to put my pfsense box into production without the tunnels and I will be limited in what I can try.

jimp · Sep 29, 2009, 1:37 PM

@bkm:

I am seeing an issue that seems to be the same. I am testing RC 1.2.3 20090924. I have tunnels set up to two separate sites to a Netopia router on the other end. The tunnels are working when I leave work in the evening. When I get to work in the morning, they are not working. The IPSec status page (SAD) shows that the tunnels are up. If I restart raccoon, the tunnel status goes down. I then ping a site and everything gets renegotiated and it works again.
I currently have a 28800 lifetime for phase 1 and 86400 for phase 2.
I am willing to test a couple things for a day or two if someone has a suggestion. After that I will need to put my pfsense box into production without the tunnels and I will be limited in what I can try.

It might help to know more about these tunnels, at least this much: Are they static tunnels or mobile clients? Are they using main mode or aggressive mode? Do you have DPD enabled? Keep Alive? What shows up in the logs when the tunnels are broken?

And anything else you can think of.

fairchild · Sep 29, 2009, 5:42 PM

im using the latest 2.0 snapshot, do you recommend leaving DPD enabled? I dont have access to the logs right now so i cant post them but it appears that when a tunnel goes down because of the internet connection on my end or the other end i have to restart the racoon service on both ends for the tunnel to reestablish, this is between 2 pfsense boxes… i dont even want to get started on my linksys vpn tunnel issues

cmb · Sep 29, 2009, 5:45 PM

@fairchild:

im using the latest 2.0 snapshot

Don't. That's not going to be stable. Pretty sure the 7.2/2.0 builds still use NAT-T which has renegotiation issues, and the 8 snapshots likely don't have a proper ipsec-tools either.