Kernel Panic

acherman · Feb 17, 2011, 2:43 PM

Hey Aubrey, changing my mtu's didn't fix anything. In fact it caused a panic on both boxes when trying to apply the changes. I don't think I will try that again.

jimp · Feb 17, 2011, 2:53 PM

@acherman:

Hey Aubrey, changing my mtu's didn't fix anything. In fact it caused a panic on both boxes when trying to apply the changes. I don't think I will try that again.

Did you get the panic/crash info when it did that? Also, are you on i386 or amd64? What snapshot date?

acherman · Feb 17, 2011, 2:58 PM

No, I didn't. Sorry. But I'm sure it will happen again soon and I have the camera ready. I am running i386 and build Mon Jan 31 07:16:37 EST 2011 right now (saw the same issues with Mon Feb 14 02:12:45 EST 2011 and I can't remember which one from the 15th).

jimp · Feb 17, 2011, 3:03 PM

Update to the most current snap ASAP and then test again. Testing on old snaps isn't likely to provide any useful feedback for this. There were patches after those dates to help prevent panics in other situations.

acherman · Feb 17, 2011, 3:26 PM

I was able to make my CARP slave fail again (seems reproducable) - I was changing the mtu's back to default, if making one change and applying it things seemed fine, when I changed a few of them and applied them but applied changes after a few (maybe 5) then it hung.

I will update to the latest snap and see if I can make it fail again…

acherman · Feb 17, 2011, 4:08 PM

I updated my CARP backup to the latest snap - Thu Feb 17 02:14:25 EST 2011 - and ran through similar motions that caused the panics before with no issues. I have updated my master as well and will update if things fail again.

eri-- · Feb 17, 2011, 10:11 PM

Can any of you with the pf_state_tree_id_RB_REMOVE_COLOR panic
try and set debug.pfpptpproxy=1 and see if they get the panic?

acherman · Feb 17, 2011, 11:12 PM

Sure, let me now how to do it (sorry) and if I get the panic running today's snapshot that can be my next step.

Feb 18, 2011, 6:16 AM

Remember messing with the GUI when this happened, and then it stopped working… and then that.

eri-- · Feb 18, 2011, 9:16 AM

Please give info on each screenshot you put in here.
Its absolutely required the date of the snapshot and how the box is cofnigured: carp, vlan, pptp,….

@ acherman
Go to system->Advanced->System tunables->Click +(at the end)

Tunable: debug.pfpptpproxy
Value: 1
Description: Testing pptp fix

Click save and try to see if it panics still.

acherman · Feb 18, 2011, 1:35 PM

Had 3 panics on my CARP master and 1 on my backup - I assume they were panics, lost comms to them but no physical access, just cycled power remotely. I just entered the tunable requested, I will update when/if something happens. I will upload a drawing of my network config when I get to the office, as well as more config details.

acherman · Feb 18, 2011, 3:08 PM

Had another panic on my master this morning as I was doing up a quick drawing - looks like the same situation, even with the pptp tunable noted by ermal. It ran fine for almost 14 hours yesterday before the first one last night (from 9:05 am to 10:56 pm). Anyway, screenshots from this mornings panic:

Now, here is a basic layout of my config:

Directly connected to the Sync interface is an identical box with all other interfaces running in parallel. Our WAN is a static connection. DHCP is served out to our corp LAN, remote office LAN, and a couple of our WiSP VLANs (625, 626) - I do not have the CARP backup DHCP turned on. The only VPN I have configured is OpenVPN (one server connection). The only package installed is the OpenVPN client config export utility. I do have a number of virtual IPs configured for inbound and outbound NAT. No traffic shaping turned on. I think that's it for special stuff, let me know if more is required, or I can upload my config files.

I am still running on the build from Thu Feb 17 02:14:25 EST 2011, with the pptp fix tunable noted by ermal.

eri-- · Feb 18, 2011, 4:30 PM

I just put a patch up.
The snapshot has to build but when it comes out try with that.

sammyfusionsleep.com · Feb 18, 2011, 6:20 PM

Thanks very much for the efforts…

2.0-BETA5 (i386)
built on Thu Feb 17 02:14:25 EST 2011
Dell Poweredge 860
Intel Pentium D CPU 2.80GHz
2 gigs ram.
Openvpn client export plugin is the only plugin installed.
pptp server for remote clients.
1 Wan, 1 LAN, no CARP, 4 IP Alias VIPs
1 ipsec tunnel to other another pfsense endpoint
2 ipsec tunnels to cisco routers

Crash happened during normal operations with no changes being made. I did have a browser tab connected to the GUI with the dashboard page up. there are two

salvor · Feb 18, 2011, 10:32 PM

@ermal:

I just put a patch up.
The snapshot has to build but when it comes out try with that.

I was taking a peek at the repository and noticed the last commit by ermal was on Feb 16.

Is this the correct URL to find out what went into the latest snapshot?
https://rcs.pfsense.org/projects/pfsense

This is the only commit on Feb 18 (as of 4:29 PM CST):
06:42 cmb pushed c7d56176 to pfsense/mainline on branch master

jimp · Feb 18, 2011, 10:33 PM

It's in the pfsense-tools repo, not just pfsense.

salvor · Feb 19, 2011, 2:16 AM

@jimp:

It's in the pfsense-tools repo, not just pfsense.

Thanks! Seems I was missing half the action!

acherman · Feb 19, 2011, 2:27 AM

Now I feel like I'm missing the action. I still can't find a new snapshot.

phospher · Mar 3, 2011, 6:16 PM

I just upgraded from 1.3 to the latest 2.0 RC1 snapshot and I'm getting the dreaeded malloc kernel panic. This is happening on two machines in a vmware esxi environment. It basically just keeps rebooting. I see there is a bug report about vmware tools causing the issue. How do I go about uninstalling vmtools?
Thanks!

acherman · Mar 3, 2011, 6:17 PM

My 2 CARP boxes have been running on the Beta5 build from Mon Feb 21 19:15:26 EST 2011 without any issues for 9 days, 14:12.