IPSEC Phase 2 issue (AES-256 related?)

wallacebw

All:

I'm running into an issue setting up a IPSEC tunnel between two 2.0 beta 5 boxes (both on Sat Feb 5 14:20:34 EST 2011 – latest as of this posting).

FYI: One is an embedded build running on alix and the other is a full install (AMD64)

Phase 1 completes fine (using AES-256 / SHA-1 / DH 5 1536). but when attempting phase 2 I run into problems.

Phase 2:

ESP / AES256 (only allowed cypher) / SHA1 (only allowed algorithm) / PFS key group 5.

in looking at the logs I see:

racoon: [name]: INFO: initiate new phase 2 negotiation: 1.2.3.4[500]<=>5.6.7.8[500]
racoon: ERROR: pfkey UPDATE failed: Invalid argument
racoon: ERROR: pfkey ADD failed: Invalid argument
racoon: [name]: ERROR 5.6.7.8 give up to get IPsec-SA due to time up to wait.

These events repeat at Approx. 4 sec intervals for 12+ attempts.

in researching the issue, it looks like this came up in the past 1.2.3RC1 and possibly compiling against the wrong kernel source. REF: http://www.network-builders.com/ipsec-tunnel-fails-pfkey-update-failed-invalid-argument-t57645.html

Any thoughts?

Also, thanks for the memory stick (usb) build, it helped alot on my HP DL360-G6 server with no optical drive. (For some reason mounting the iso through ilo was problematic) REF: http://forum.pfsense.org/index.php/topic,28759.msg149591.html

wallacebw

I just saw this post also: http://forum.pfsense.org/index.php?topic=23273.0

I'll try dropping phase 2 to aes-128 tomorrow to replicate the linked conditions. (I have to drive in for that).

cmb

AES256 works fine in 2.0. Granted I am using ipsec-tools 0.8.0, not sure if I've set it up on 0.7.3 that's currently in snapshots.

wallacebw

@cmb:

AES256 works fine in 2.0. Granted I am using ipsec-tools 0.8.0, not sure if I've set it up on 0.7.3 that's currently in snapshots.

Should the 0.80 tools work on amd64? Service won't start after upgrade (followed instructions in the link)

cmb

@wallacebw:

Should the 0.80 tools work on amd64? Service won't start after upgrade (followed instructions in the link)

Not that build. That's i386-only.

cmb

check http://forum.pfsense.org/index.php/topic,33010.0.html again, I added an amd64 build, though entirely untested.

azzido

Try disabling glxsb if you have it enabled on the Alix box. I had issues establishing IPSec tunnels with it enabled on my Alix box.

wallacebw

.80 amd64 installed successfully. still seem unable to stack AES-256 for both phases though. any thoughts?