Racoon crashed, core dumped
-
Hello,
My racoon core dumped this morning, and all my IPSec tunnels went down. Disabling and reenabling IPSec in the GUI brought them back up.
May 9 02:00:43 192.168.xxx.xxx racoon: NOTIFY: the packet is retransmitted by xx.xx.xxx.xx[500] (1).
May 9 02:00:48 192.168.xxx.xxx pf: 00:00:05.206751 rule 1/0(match): block in on enc0: (tos 0x0, ttl 99, id 236, offset 0, flags [none], proto TCP (6), length 40)
May 9 02:00:48 192.168.xxx.xxx pf: 192.168.yy.xxx.80 > 192.168.xx.xxx.41609: Flags [.], cksum 0x437b (correct), ack 1, win 0, length 0
May 9 02:00:48 192.168.xxx.xxx kernel: pid 32526 (racoon), uid 0: exited on signal 11 (core dumped)Is there any way to find out why this happened?
Thanks,
Todd
-
I've submitted a bug report about this to FreeBSD :
http://www.freebsd.org/cgi/query-pr.cgi?pr=168104
Wouldn't it be best if racoon auto-restarted in case of crash?
Thanks,
Todd
-
Hard to say why that happened exactly but it's not unheard of.
You might try a 2.1 snapshot, it may be something we've already fixed or patched in racoon.
-
I'm experiencing the same issue with the latest 2.1 version from snapshots.pfsense.org as well. IPSec keeps crashing after a few seconds and will not come back up again unless doing so manually. Didn't have this issue with the 26th of November 2011 release. Tried to gitsync against the latest master branch to no avail.
I'm running:
2.1-DEVELOPMENT (amd64)
built on Sat May 26 23:51:29 EDT 2012
FreeBSD 8.3-RELEASE-p1In the logs it keeps reporting:
kernel: pid 1796 (racoon), uid 0: exited on signal 11 (core dumped)
-
Needs a newer binary, check the next new snapshot.
-
FYI, this has happened again, 40 days since previously.
I've noticed from the logs that immediately before both crashes it was establishing a VPN tunnel with a Netgear ProSafe VPN FVS114 Firewall, with tunnel settings on both sides being AES128/SHA1, with both Phase 1 and Phase 2 having 28800 lifetimes.
-
There have been many bugfixes commited to the ipsec-tools source tree since the last release 0.8.0 more than a year ago:
http://sourceforge.net/mailarchive/forum.php?forum_name=ipsec-tools-devel
http://ftp.netbsd.org/pub/NetBSD/NetBSD-current/src/crypto/dist/ipsec-tools/src/ -
FYI, upgrading hardware to faster CPU, more RAM, etc. seems to have resolved this problem.
-
This happened again (after about 6 months) on the newer hardware. Correlated the events this time to ISP outages/connectivity issues.
Submitted bug to Ipsec-tools:
https://sourceforge.net/tracker/index.php?func=detail&aid=3603844&group_id=74601&atid=541482#
I'm not running ipsec-tools 0.8.1 yet (pfSense 2.1), so I don't know if it happens there as well.
-
Today I've found that I had duplicate IPSec tunnels configured in pfSense, one disabled and the other enabled.
I've moved this tunnel elsewhere, and I've removed both from the pfSense config to see if this improves my racoon stability.
-
Stability is improved, but I'm still having racoon segfaults that coincide with loss of connectivity to endpoints.
I've noticed DPD is firing on these tunnels, should I disable DPD?
-
Stability is improved, but I'm still having racoon segfaults that coincide with loss of connectivity to endpoints.
I've noticed DPD is firing on these tunnels, should I disable DPD?
Define "loss of connectivity", does the endpoints' physical link go down ?
Btw iirc there's an exploit that makes it possible to remotely crash racoon.
My suggestion would be to upgrade to 2.1 if you can, which includes an updated and fully patched racoon.
-
Yes, the ISP providing internet to these remote routers is going down, usually at night for less than an hour (maybe afterhours maintenance). racoon segfaults almost immediately after this happens.
It always seems to be the same ISP, too, so I'm currently going over the configurations of these particular tunnels and changing them to 'known good' settings.
It looks like the vulnerability was for ipsec-tools 0.6.2 and earlier: http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2005-3732
I thought I just read about a recent build that was unstable, is 2.1 production ready?
-
The ISP replaced/repaired a DSL DSLAM and possibly some core routers as well.
racoon has been up stable now for several weeks since this change.
-
I'm assuming this isn't related to Broadcom NIC issues?
This is a small Dell server with two Broadcom NICs.
-
A FreeBSD developer is asking me for backtraces, but they don't seem to be that informative.
Aren't there separate binaries with debugging symbols that you are supposed to use when doing this?
GNU gdb 6.6 [GDB v6.6 for FreeBSD] Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-portbld-freebsd8.1"... (no debugging symbols found) (no debugging symbols found) Core was generated by `racoon'. Program terminated with signal 11, Segmentation fault. #0 0x080672a9 in ?? () from /libexec/ld-elf.so.1 (gdb) bt #0 0x080672a9 in ?? () from /libexec/ld-elf.so.1 #1 0x2854de48 in ?? () #2 0x00000000 in ?? () (gdb) quit