PfSense 2.0.1 locking up (hard)
-
I've just installed a new pfSense setup for my client using one of the SuperMicro Atom D525 motherboards.
It has 3 NIC's now. 2 on motherboard and one Intel NIC plugged in via PCIe.
It's been running flawlessly at home where I built a similar (but not completely same) model of his network.. but when I got it to the client… and we decided to add RIP to the system as it now sits between his netscreen firewall and his ExtremeNetworks core router, the unit would randomly lock up. It requires a hard reset. (local keyboard is completely unresponsive)
(or so it seems. I've turned off RIP for now since the network is pretty static)
And it's now the waiting game.
ALL this system is doing is Squid proxy. There's really no other rules for the FW/NAT except "manual NAT" and "allow all" for the firewall. The setup is pretty simple/no-frills.
Has anyone else seen this behavior at all?
(lock ups in general with things to avoid)?
-Ben
-
If the only difference is the NIC perhaps look at this:
http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_igb.284.29_and_em.284.29_CardsSteve
-
I'll take a look.
The card that faces into the network is an Intel card (but comes up as em(x) which still applies in that FAQ)
(I think the 2 on the MB are Intel as well sooo)Thanks!!
-Ben
-
I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.
-
I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.
Thanks for the note… it's good to hear from people with similar hardware.
I have the same MB at home (not for this install though, I got the D525 for the client) -- and THEN bought the D510 since I had parts for it laying around and don't need the mem speed for it.
Your proposition gives me the willies. What? Am I supposed to schedule a reboot every weekend? Turn on the BMC watchdog? (need to test that at home first) Yikes.
I do have 4GB in the unit -- but I am running pfSense-2.0.1-embedded_AM64 off an SSD... so RAM isn't exactly wild-free-and-open. But, I'll bump up the bufs later.
right not seems to have a fairly static 5574/25600 and the client is getting their full 50Mb/s BW to the internet. So overall, provided the system stays up, we're pretty happy.
As always, the open source community does some excellent stuff.
stay tuned,
-Ben
-
Theoretically the MBUF usage should level off at some point. Mine are at 34652/131072 and growing with 21 days of uptime. This is approximately the same uptime I had 21 days ago when one of the NICs stopped passing traffic, so I have yet to see my MBUF usage level off on this system. Mind you I have hundreds of users, so I can only guess that that is a factor, because my internet connection is less than 40 Mbps.
-
I should add that increasing your max MBUFS is not going to put much of a dent in 4G of RAM. My system has 4G and no swap and is currently reporting 10% memory usage. My current states are reporting at 11795/389000 but I've seen them as high as 34000. I think memory is allocated on your max rather than current, so you can see that increasing your nmbclusters isn't likely to be a problem on your system.
-
Theoretically the MBUF usage should level off at some point. Mine are at 34652/131072 and growing with 21 days of uptime. This is approximately the same uptime I had 21 days ago when one of the NICs stopped passing traffic, so I have yet to see my MBUF usage level off on this system. Mind you I have hundreds of users, so I can only guess that that is a factor, because my internet connection is less than 40 Mbps.
Hmmm…
This user has 50Mb/s and about 3,000 PC's (Mac/Windows Mix - but mostly probably 90% Mac).
I just need to watch it. So far, the CPU has been pretty sleepy according to RRD. Peaks of 25% with most daily usage under 20%.
(I'm assuming that's system total consider this is a dual-core w/HyperThreading mb)Very nice.
The state table I've seen peak around 20,000 of 390,000. So.Also nice.
Thanks for the data. I'll keep sharing as I watch this systems first day in "production" mode. (scaaaarrryyy... hahahah)
-Ben
-
I should add that increasing your max MBUFS is not going to put much of a dent in 4G of RAM. My system has 4G and no swap and is currently reporting 10% memory usage. My current states are reporting at 11795/389000 but I've seen them as high as 34000. I think memory is allocated on your max rather than current, so you can see that increasing your nmbclusters isn't likely to be a problem on your system.
Agreed. I didn't think adding more MBUFs would be a big impact either. (thanks for the note though)
Cheers,
-ben
-
BTW,
Does MBUF's recent over idle time or do they allocate and then (ideally) stick at some average level?
(i.e. what kind of behavior should I see if everything were running correctly?)
-Ben
-
I've never seen them decrease except with a reboot. I think best case is that they level off and stop increasing at some point.
-
K.
Good to know - I'll be watching.
Thanks,
-ben
-
I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.
Forgive my possibly noobish question, but how do I set this setting?
Do I just add it on the: "System -> Advanced -> System Tunables" page?
Does this setting require a reboot of the system?
If no, does changing the setting drop any current states?
Much obliged,
Matt -
I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.
Forgive my possibly noobish question, but how do I set this setting?
Do I just add it on the: "System -> Advanced -> System Tunables" page?
You have to look at the link above. There's a file in /boot you need to create and put those parms in.
Does this setting require a reboot of the system?
If no, does changing the setting drop any current states?Yes. it needs a reboot. (so that should answer the 2nd part of that question)
It worked for me with no issues.. and I'm seeing a level off at about 7200 (7174) of 131072 after a day over heavy operation.
Shout if you need more,
-Ben
-
Some settings can be set from the system tunables page, however this one cannot. You must add it to /boot/loader.conf.local (create the file if it does not exist) and then reboot for it to take effect. Do not add it to /boot/loader.conf, as this will be overwritten on the next system upgrade.
-
Hi!
I have similar problem.
About month ago, when I updated to 2.0.1. version I have got situations, when pfsense freeezees.
Pfsense is installed on old HP E800 series server with PIII 1ghz CPU, 512Mb RAM and 17Gb SCSI hdd.
Firstly I though, that it is from bad RAM or HDD, but after changing them, everything continues.
I setted up another hardware, simple pc with 1,7Ghz Celeron Intel CPU, 768Mb RAM and 40Gb hdd.
Two days everything was ok, but then in one day it stops 3 TIMES!!
Then I thought, that it has got something common with ACPI, even I have got UPS.
Turned off ACPI support in pfsense.
Few days everything was OK after turning of ACPI in pfsense, but this night again it stops.
And I am a little bit desperate.
If hardware change doesn't help, it could be MBUFs?
Right now it is = 0.
In pfsense first page it shows 518/24896 MBUFs usage.Any Ideas?