Racoon stops without any cause

[kalu]

after chaning the below mentioned things it works for a little longer but after 1-2 hours it again stops i.e. the racoon service

pfs key group ->off
DPD->off
prefer old SA's->ticked

by the way i have a mix up of devices 3 is Trendnet and 1 is pfsense nanobsd RC.02
i'm quite sure there's something that i have missed.
any helping hands is greatly appreciated
thanks

[kalu]

i really can't figure out why racoon is stopping after running for few hours.

please help me pfsense gods and gurus
:)

[Metu69salemi]

Then you can count me off  ;)

Do you happen to have logs at this occurance time. It would be great to have also logs from another side of ipsec. If there could be answers

[kalu]

Thanks Metu69salemi
The other side has Trendnet VPN router firelwall and the log at trendnet says

[15:40:31]**** SENT OUT  FIRST MESSAGE OF MAIN MODE ****
[15:40:31] PAYLOADS: SA,PROP,TRANS,VID

In my opinion the problem lies in my pfsense. The racoon just automatically stops and ipsec is down as soon as i start the racoon service , everything is back to normal.
preety annoyed
thanks
kalu

[kalu]

hi here is the log file of my pfsense device

Aug 3 15:43:07 racoon: INFO: begin Identity Protection mode.
Aug 3 15:43:07 racoon: [IBP-LINK]: INFO: initiate new phase 1 negotiation: 202.79.51.215[500]<=>202.79.54.135[500]
Aug 3 15:43:07 racoon: [IBP-LINK]: INFO: IPsec-SA request for 202.79.54.135 queued due to no phase1 found.
Aug 3 15:42:52 racoon: INFO: delete phase 2 handler.
Aug 3 15:42:52 racoon: [IBP-LINK]: [202.79.54.135] ERROR: phase2 negotiation failed due to time up waiting for phase1 [Remote Side not responding]. ESP 202.79.54.135[0]->202.79.51.215[0]
Aug 3 15:42:24 racoon: ERROR: phase1 negotiation failed due to time up. 051917376d09ef4c:0000000000000000
Aug 3 15:42:21 racoon: [IBP-LINK]: [202.79.54.135] INFO: request for establishing IPsec-SA was queued due to no phase1 found.
Aug 3 15:42:05 racoon: INFO: delete phase 2 handler.
Aug 3 15:42:05 racoon: [IBP-LINK]: [202.79.54.135] ERROR: phase2 negotiation failed due to time up waiting for phase1 [Remote Side not responding]. ESP 202.79.54.135[0]->202.79.51.215[0]
Aug 3 15:41:59 racoon: INFO: purged IPsec-SA proto_id=ESP spi=4008569731.
Aug 3 15:41:56 racoon: INFO: purged IPsec-SA proto_id=ESP spi=3485435869.
Aug 3 15:41:34 racoon: INFO: begin Identity Protection mode.
Aug 3 15:41:34 racoon: [IBP-LINK]: INFO: initiate new phase 1 negotiation: 202.79.51.215[500]<=>202.79.54.135[500]
Aug 3 15:41:34 racoon: [IBP-LINK]: INFO: IPsec-SA request for 202.79.54.135 queued due to no phase1 found.
Aug 3 15:41:29 racoon: [ISP-LINK]: INFO: IPsec-SA established: ESP 202.79.51.215[500]->202.79.54.197[500] spi=3507038995(0xd1092b13)
Aug 3 15:41:29 racoon: [ISP-LINK]: INFO: IPsec-SA established: ESP 202.79.51.215[500]->202.79.54.197[500] spi=256939600(0xf509650)
Aug 3 15:41:29 racoon: [ISP-LINK]: INFO: respond new phase 2 negotiation: 202.79.51.215[500]<=>202.79.54.197[500]
Aug 3 15:41:28 racoon: [PAL-LINK]: INFO: ISAKMP-SA deleted 202.79.51.215[500]-202.79.50.88[500] spi:f5c9058ac6c52ffa:ddf88b39fda1db1a
Aug 3 15:41:28 racoon: [PAL-LINK]: INFO: ISAKMP-SA expired 202.79.51.215[500]-202.79.50.88[500] spi:f5c9058ac6c52ffa:ddf88b39fda1db1a
Aug 3 15:41:27 racoon: INFO: purged IPsec-SA proto_id=ESP spi=2417358319.
Aug 3 15:41:27 racoon: [PAL-LINK]: INFO: IPsec-SA established: ESP 202.79.51.215[500]->202.79.50.88[500] spi=3172222795(0xbd14474b)
Aug 3 15:41:27 racoon: [PAL-LINK]: INFO: IPsec-SA established: ESP 202.79.51.215[500]->202.79.50.88[500] spi=40673213(0x26c9fbd)
Aug 3 15:41:27 racoon: [PAL-LINK]: INFO: respond new phase 2 negotiation: 202.79.51.215[500]<=>202.79.50.88[500]
Aug 3 15:41:27 racoon: [PAL-LINK]: INFO: ISAKMP-SA established 202.79.51.215[500]-202.79.50.88[500] spi:f5c9058ac6c52ffa:ddf88b39fda1db1a
Aug 3 15:41:26 racoon: INFO: begin Identity Protection mode.
Aug 3 15:41:26 racoon: [PAL-LINK]: INFO: respond new phase 1 negotiation: 202.79.51.215[500]<=>202.79.50.88[500]
Aug 3 15:41:18 racoon: INFO: delete phase 2 handler.
Aug 3 15:41:18 racoon: [IBP-LINK]: [202.79.54.135] ERROR: phase2 negotiation failed due to time up waiting for phase1 [Remote Side not responding]. ESP 202.79.54.135[0]->202.79.51.215[0]
Aug 3 15:40:59 racoon: [ISP-LINK]: INFO: IPsec-SA expired: ESP/Tunnel 202.79.54.197[500]->202.79.51.215[500] spi=42990396(0x28ffb3c)
Aug 3 15:40:59 racoon: [ISP-LINK]: INFO: IPsec-SA expired: ESP 202.79.51.215[500]->202.79.54.197[500] spi=4008569731(0xeeedeb83)
Aug 3 15:40:58 racoon: ERROR: phase1 negotiation failed due to time up. f8d07fc084134232:0000000000000000
Aug 3 15:40:57 racoon: [PAL-LINK]: INFO: IPsec-SA expired: ESP/Tunnel 202.79.50.88[500]->202.79.51.215[500] spi=39787469(0x25f1bcd)
Aug 3 15:40:57 racoon: [PAL-LINK]: INFO: IPsec-SA expired: ESP 202.79.51.215[500]->202.79.50.88[500] spi=3485435869(0xcfbf87dd)
Aug 3 15:40:47 racoon: [IBP-LINK]: [202.79.54.135] INFO: request for establishing IPsec-SA was queued due to no phase1 found.
Aug 3 15:40:46 racoon: NOTIFY: the packet is retransmitted by 202.79.54.209[500] (1).
Aug 3 15:40:41 racoon: NOTIFY: the packet is retransmitted by 202.79.54.209[500] (1).
Aug 3 15:40:39 racoon: INFO: delete phase 2 handler.
Aug 3 15:40:39 racoon: [IBP-LINK]: [202.79.54.135] ERROR: phase2 negotiation failed due to time up waiting for phase1 [Remote Side not responding]. ESP 202.79.54.135[0]->202.79.51.215[0]
Aug 3 15:40:36 racoon: NOTIFY: the packet is retransmitted by 202.79.54.209[500] (1).
Aug 3 15:40:32 racoon: [IDP-LINK]: INFO: IPsec-SA established: ESP 202.79.51.215[500]->202.79.54.209[500] spi=4090473567(0xf3cfac5f)
Aug 3 15:40:32 racoon: [IDP-LINK]: INFO: IPsec-SA established: ESP 202.79.51.215[500]->202.79.54.209[500] spi=108652071(0x679e627)
Aug 3 15:40:32 racoon: INFO: received RESPONDER-LIFETIME: 300 seconds
Aug 3 15:40:32 racoon: [IDP-LINK]: INFO: initiate new phase 2 negotiation: 202.79.51.215[500]<=>202.79.54.209[500]
Aug 3 15:40:32 racoon: [IDP-LINK]: INFO: IPsec-SA expired: ESP/Tunnel 202.79.54.209[500]->202.79.51.215[500] spi=89231451(0x551905b)
Aug 3 15:40:27 racoon: [IDP-LINK]: INFO: IPsec-SA established: ESP 202.79.51.215[500]->202.79.54.209[500] spi=2299975968(0x8916d920)
Aug 3 15:40:27 racoon: [IDP-LINK]: INFO: IPsec-SA established: ESP 202.79.51.215[500]->202.79.54.209[500] spi=190570987(0xb5be1eb)
Aug 3 15:40:27 racoon: [IDP-LINK]: INFO: respond new phase 2 negotiation: 202.79.51.215[500]<=>202.79.54.209[500]
Aug 3 15:40:08 racoon: INFO: begin Identity Protection mode.
Aug 3 15:40:08 racoon: [IBP-LINK]: INFO: initiate new phase 1 negotiation: 202.79.51.215[500]<=>202.79.54.135[500]
Aug 3 15:40:08 racoon: [IBP-LINK]: INFO: IPsec-SA request for 202.79.54.135 queued due to no phase1 found.

[Metu69salemi]

Log just repeats, that phase 1 isn't correct nor found. have you triplechecked every single settings?

[kalu]

hi Metu69salemi
Yes i've checked and double checked the settings.
It works preety fine until racoon stops .
thanks
kalu

[Metu69salemi]

Is there system logs at the time?

[kalu]

The IPSec VPN link is running fine now after starting racoon again
don't know when will it get down by it self
Hi Metu69salemi
here is the system log

Aug 3 10:40:47 check_reload_status: Reloading filter
Aug 3 10:40:41 check_reload_status: Syncing firewall
Aug 3 10:40:37 check_reload_status: Syncing firewall
Aug 3 10:40:29 check_reload_status: Syncing firewall
Aug 3 10:40:27 check_reload_status: Syncing firewall
Aug 3 10:39:00 check_reload_status: Reloading filter
Aug 3 10:35:54 check_reload_status: Reloading filter
Aug 3 16:20:44 apinger: alarm canceled: WAN_GW(202.79.51.193) *** delay ***
Aug 3 10:35:12 check_reload_status: Reloading filter
Aug 3 16:20:02 apinger: ALARM: WAN_GW(202.79.51.193) *** delay ***
Aug 3 16:15:42 kernel: pid 10457 (racoon), uid 0: exited on signal 11 (core dumped)
Aug 3 10:24:23 check_reload_status: Syncing firewall
Aug 3 16:09:22 php: /vpn_ipsec_phase2.php: Reloading IPsec tunnel 'ISP-LINK'. Previous IP '202.79.54.197', current IP '202.79.54.197'. Reloading policy
Aug 3 16:06:17 php: /index.php: Successful webConfigurator login for user 'admin' from 10.49.32.162
Aug 3 16:06:17 php: /index.php: Successful webConfigurator login for user 'admin' from 10.49.32.162
Aug 3 16:04:48 php: /index.php: User logged out for user 'admin' from: 10.49.32.162
Aug 3 10:18:42 check_reload_status: Reloading filter
Aug 3 10:18:35 check_reload_status: Syncing firewall
Aug 3 16:03:35 php: /vpn_ipsec_phase2.php: Reloading IPsec tunnel 'IDP-LINK'. Previous IP '202.79.54.209', current IP '202.79.54.209'. Reloading policy
Aug 3 09:45:05 check_reload_status: Reloading filter
Aug 3 09:44:56 check_reload_status: Reloading filter
Aug 3 15:29:55 apinger: alarm canceled: WAN_GW(202.79.51.193) *** delay ***
Aug 3 15:29:46 apinger: ALARM: WAN_GW(202.79.51.193) *** delay ***
Aug 3 15:28:18 php: /status_services.php: Forcefully reloading IPsec racoon daemon
Aug 3 15:06:37 kernel: pid 60568 (racoon), uid 0: exited on signal 11 (core dumped)
Aug 3 15:05:17 php: /status_services.php: Forcefully reloading IPsec racoon daemon
Aug 3 15:03:42 kernel: pid 19710 (racoon), uid 0: exited on signal 11 (core dumped)
Aug 3 15:03:33 php: /status_services.php: Forcefully reloading IPsec racoon daemon
Aug 3 09:16:18 check_reload_status: Reloading filter
Aug 3 15:01:08 apinger: alarm canceled: WAN_GW(202.79.51.193) *** delay ***
Aug 3 09:16:05 check_reload_status: Reloading filter
Aug 3 15:00:55 apinger: ALARM: WAN_GW(202.79.51.193) *** delay ***
Aug 3 14:41:47 kernel: pid 33505 (racoon), uid 0: exited on signal 11 (core dumped)
Aug 3 14:38:44 sshlockout[27588]: sshlockout/webConfigurator v3.0 starting up
Aug 3 14:38:44 sshd[20376]: Accepted keyboard-interactive/pam for admin from 10.49.32.162 port 2498 ssh2
Aug 3 14:37:53 syslogd: kernel boot file is /boot/kernel/kernel
Aug 3 14:37:53 syslogd: exiting on signal 15
Aug 3 14:37:39 syslogd: kernel boot file is /boot/kernel/kernel
Aug 3 14:37:39 syslogd: exiting on signal 15
Aug 3 14:37:37 syslogd: kernel boot file is /boot/kernel/kernel
Aug 3 14:37:37 syslogd: exiting on signal 15
Aug 3 14:34:00 kernel: pid 46139 (racoon), uid 0: exited on signal 11 (core dumped)
Aug 3 14:33:48 kernel: arp: unknown hardware address format (0x1100)
Aug 3 14:27:46 kernel: arp: unknown hardware address format (0x4500)
Aug 3 14:23:14 php: /status_services.php: Forcefully reloading IPsec racoon daemon
Aug 3 14:22:05 apinger: /usr/local/bin/rrdtool respawning too fast, waiting 300s.
Aug 3 08:36:38 check_reload_status: Reloading filter
Aug 3 14:21:29 kernel: vr2: link state changed to DOWN
Aug 3 08:36:29 check_reload_status: Linkup starting vr2
Aug 3 14:21:28 apinger: alarm canceled: WAN_GW(202.79.51.193) *** delay ***

[Metu69salemi]

Kalu:
if you're sure that p1 & p2 settings are right, then this log doesn't say anything to me.
It just shows that it's not liking how the racoon itself works(or not) by killing that process. maybe some developers could help reading this log

[stemond]

i have same issue :(
Racoon stop withou causes and drops all IPSEC tunnels.

i am using Pfsense 2.0 RC3 and it happens without PPTP tunnel

this is my log, have you any hint ???

 Sep 8 09:58:48 	php: /status_services.php: Forcefully reloading IPsec racoon daemon
[b]Sep 8 09:52:28 kernel: pid 23362 (racoon), uid 0: exited on signal 11 (core dumped)[/b]
Sep 8 09:50:04 	kernel: arp: 192.168.126.13 moved from 00:01:02:f9:ea:55 to 00:08:02:45:32:42 on le0
Sep 8 09:30:04 	kernel: arp: 192.168.126.13 moved from 00:01:02:f9:ea:55 to 00:08:02:45:32:42 on le0
Sep 8 09:30:04 	kernel: arp: 192.168.126.13 moved from 00:08:02:45:32:42 to 00:01:02:f9:ea:55 on le0
Sep 8 09:10:04 	kernel: arp: 192.168.126.13 moved from 00:01:02:f9:ea:55 to 00:08:02:45:32:42 on le0

S.

[TheBlast]

Hi there,
same issue for me : ~160 ipsec tunnels get stopped after some hours.
Could someone just paste the magic script to restart racoon if it's stopped (cron inside) ?

edit : error message
Sep 17 08:33:49 pfsense kernel: pid 2238 (racoon), uid 0: exited on signal 11 (core dumped)

Edit 2: new crash
System log message : Sep 17 19:07:56 kernel: pid 10333 (racoon), uid 0: exited on signal 11 (core dumped)
Ipsec error message : Sep 17 19:07:56 racoon: [xxx]: [yyy.yyy.yyy.yyy] ERROR: phase1 negotiation failed.

Edit 3 : no crash since I disabled badly configured tunnel … will keep you informed and check with V2 Release this week.

So a badly configured tunnel seems to kill racoon ... Will this help ?

[dhatz]

You might also want to open a ticket with the ipsec-tools developers:
http://sourceforge.net/projects/ipsec-tools/

[podilarius]

@TheBlast:

So a badly configured tunnel seems to kill racoon … Will this help ?

What was badly configured? I have noticed that all that had the problem are showing core dumps. Could be bad memory or bad memory management by raccoon. Are there any other packages being used?

[TheBlast]

Hi,
when I said "badly" it was just a way to say that one side was using an ID and not the other side.
Anyway I disabled all the "misconfigured" tunnels but I still get the same problem, even with version 2.0 Release.
Racoon stops once or twice a day. Fortunately some kind of cron restarts it from time to time but looks like a bug.
Where can I find the core dump and who will be interested in debugging it ?

[TheBlast]

The problem remains the same : once or twice a day racoon crashes. Fortunately some king of script restarts it after a while. Is the a way to stop this ?

[stemond]

@theblast: Can you post your script restart ?

thank you!

[TheBlast]

Hi,
no because I don't know which script it is ! I just wanted to point out that a script does the job.

[podilarius]

Are you running snort or any other packages? What type of hardware do you have? Single/Multiple Core CPU and how much memory?

[TheBlast]

Hi,
Only VPN Client export package is installed.
The hardware :

• abit motherboard (2011) / Core I3 intel processor
• Ram 2 Go
• Network : Lan is intel PCI Express Gigabit adapter, others are DLINK DFE 530 Tx 100 mb