IPSEC between pfSense and Cisco PIX 525 (pixos v8)

[bruor]

IPSEC gurus:

I have been trying for a few hours to get this working properly,  i believe my setup seems ok on the pix and the pfsense box,  on the pix i have the following relevant config items:

crypto ipsec transform-set coutts-transform-set esp-des esp-md5-hmac
crypto ipsec security-association lifetime seconds 3600

crypto map mymap 100 match address vpn
crypto map mymap 100 set peer 72.38.121.34
crypto map mymap 100 set transform-set coutts-transform-set
crypto map mymap interface T3
crypto isakmp identity address
crypto isakmp enable T3
crypto isakmp policy 11
 authentication pre-share
 encryption des
 hash md5
 group 2
 lifetime 86400
no crypto isakmp nat-traversal

tunnel-group 72.38.121.34 type ipsec-l2l
tunnel-group 72.38.121.34 ipsec-attributes
 pre-shared-key *

configuration on the pfsense matches this but is setup for the opposite end.  I don't know if i need an entry on the psk tab, but i have put one there with the identifier as the IP of the cisco,  and the psk in that box as well as in the ipsec properties.  The phase1 groups, lifetimes etc.  are all correct, and it seems like racoon is trying to come up, however, i get the following errors when i try to get the tunnel to come online.  i believe the bold lines are the relevant ones, but am having trouble tracking down any answers as  to why they are happening.

Apr 23 21:55:32 racoon: [USLV IPSEC]: INFO: ISAKMP-SA deleted 72.38.121.34[500]-198.183.167.100[500] spi:19788241af5d2232:4219ee0ce8b1cb16 
Apr 23 21:55:31 racoon: ERROR: phase2 negotiation failed due to phase1 expired. 19788241af5d2232:4219ee0ce8b1cb16:0000d114 
Apr 23 21:55:21 racoon: [USLV IPSEC]: INFO: ISAKMP-SA expired 72.38.121.34[500]-198.183.167.100[500] spi:19788241af5d2232:4219ee0ce8b1cb16 
Apr 23 21:55:21 racoon: ERROR: Message: '^ hBl 2 4 ( P &C5LP F E I e z M T\ \p $Nh @ a H-8+ 2 !gi f) d% 4Pd <{ xn' l U = H w ,| h B f$ nkx f < '. 
***Apr 23 21:55:21 racoon: ERROR: fatal INVALID-ID-INFORMATION notify messsage, phase1 should be deleted. 
Apr 23 21:55:21 racoon: [USLV IPSEC]: INFO: initiate new phase 2 negotiation: 72.38.121.34[500]<=>198.183.167.100[500] 
Apr 23 21:55:20 racoon: [USLV IPSEC]: INFO: ISAKMP-SA established 72.38.121.34[500]-198.183.167.100[500] spi:19788241af5d2232:4219ee0ce8b1cb16 
Apr 23 21:55:20 racoon: INFO: received Vendor ID: DPD 
Apr 23 21:55:20 racoon: INFO: received Vendor ID: draft-ietf-ipsra-isakmp-xauth-06.txt 
Apr 23 21:55:20 racoon: INFO: received Vendor ID: CISCO-UNITY 
***Apr 23 21:55:20 racoon: INFO: received broken Microsoft ID: FRAGMENTATION
Apr 23 21:55:20 racoon: INFO: begin Identity Protection mode. 
Apr 23 21:55:20 racoon: [USLV IPSEC]: INFO: initiate new phase 1 negotiation: 72.38.121.34[500]<=>198.183.167.100[500] 
Apr 23 21:55:20 racoon: [USLV IPSEC]: INFO: IPsec-SA request for 198.183.167.100 queued due to no phase1 found. 

Any help on this would be greatly appreciated,  I need to get this configuration done by friday or we may have to reinstall our crappy cisco 2611 in place of this lovely pfsense box.

[hoba]

http://doc.m0n0.ch/handbook-single/#id2608349

[bruor]

thanks for the link,  that is the doc i used as a guide to get troubleshooting underway.  the configs at that link also differ as they are for pixos6 and do not apply on 8.  8 will tell you that a command is deprected and you have to do something a little differently.

Is there anyone that can help with the racoon issues I am expeiencing?  is there a way to turn off isakmp fragmentation on the cisco?  It would be great to know exactly what is going wrong since it seems to want to bring up both layers of the tunnel.

[bruor]

I have an update on this.  it seems that my remote subnet entry was /16, while the actual remote subnet was /22…

the debugging on the cisco was way more helpful in determining the problem at the end of the day.  for those in a similar situation you will need to run the following on a PIX/ASA to see what you need.
debug crypto isakmp

THEN.  i got a ping ready on pfsene, to ping the inside address of the remote endpoint (after creating firewall rules) and did the following
terminal monitor
-execute ping on pfsense now.
-after you see the Group = xxxx  entry in the logs and think you have what you need
terminal no monitor

this will keep it from scrolling off your buffer until you can figure our what it going on.