PfSense w/ DHCP has brought down our network twice
-
We started using pfSense for our small business about 9 months ago. Up until a couple weeks ago, it has run very well. But a couple weeks (and then again today) something happened with pfSense that brought our entire network to a halt.
We have been using pfSense's built in DHCP for out network. We use static leases for all servers and desktops.
When people got to work that morning, they found that no one could connect to the internet or even to the servers that were on the local network. At first glance, I thought it was the firewall. But then when I saw that people couldn't even connect to servers that were on the local network (ie, no firewall involved with that connection), then I thought maybe it was a bad switch. After more investigation I sat down at the firewall machine (not SSH'ing in) and discovered that the keyboard was non-responsive. The screen was at the menu, but I couldn't select shell or anything. I had to do a hard restart of the machine.
After pfSense was back up, low and behold, the network worked again. Because local machines could not contact other local machines, I figured that the problem had to be something with DHCP. Either pfSense is screwing something up, or pfSense's dhcpd service is messing something up.
Of course the logs are wiped on each restart, so I cannot go back and look at them.
1 - I'm working on setting up a syslog server for pfSense to send logs to. When this is setup, will the logs on the syslog server get wiped when I restart pfSense like they do now?
2 - Is there an option somewhere that I can choose to NOT wipe the logs when pfSense is restarted?
3 - Do I need to run pfSense with verbose logging or anything like that to discover where this problem is originating? Unfortunately I cannot reproduce the problem, so I just have to wait for it to happen again. Is running the server with verbose logging going to fill up the HDD quickly?
4 - Where should I start looking to discover where this problem originated?
Like I said, this incident happened again this morning. It's really annoying and makes me question using pfSense. I hope that it is only a problem with the DHCP service itself. I'm really leaning toward going back to just running DHCPd on one of our servers to use.
And before anyone asks, no we have not been running a secondary DHCP server to fall back on. That is one lesson learned from this problem (if in fact the DHCP service is the problem).
Anyways, I looking for any help and insight I can get here. I have really enjoyed using pfSense and I hope to continue to use it.
-
Sounds like the whole system froze. The DHCP service can't respond to requests if the system itself isn't working. What hardware are you running it on? Also, is it new hardware or some old system that you put it on?
-
It's hardware that is several years old. Because of budget at the time and because pfSense said it didn't require fancy new hardware, we just threw together something with what we had.
Is there any definitive way to determine if it was simply a hardware freeze? Any thing in logs or anything at all that could help?
-
i'd be curious if here's a log for tracking hardware lock up's as well…..
i'm in the same boat, been using pfsense for 13 months now, had to reboot it once, and then another time i had to reset to factory defaults just out of the blue.
make sure you have all the settings backed up, as i didn't the first time and i was kicking myself.
-
If the keyboard is non-responsive then it can only be one of 2 things - a kernel crash (highly unlikely) or a hardware problem (almost certainly). The usual things to do include checking cooling (open the case, check that the fans work and that the case isn't clogged with dust), that the RAM is ok (run memtest86).
-
jakobud,
You cant really blame pfSense when the hardware has 'hardlocked'. NO OS can function when the hardware is frozen solid.. I know this gets frustrating very quick,when you come into the work spot and everyone is crying the blues,,,"No net access".
We have been using pfSense-1.2.3-RELEASE for not quite a year and have never had to restart any one of the three pfSense machines on account of hardware lockups. This is with a 1,000 user base. Not sure what an average concurrent user number would be. All three are different hardware configurations but mostly 7-8 year old 1u server boxes.
A starting point is to do an 'dmesg' and see if anything here looks wonky such as maybe cpu throttling,possibly due to cooling problems or something similar.
An easy thing to eliminate in this picture is system memory. Boot up off of about any linux boot disk,and type in 'memtest' to check the system memory. If the memory is bad you will see red blocks,which makes bad stuff obvious.
You didn't mention if you are using pfSense version #.#.#.
Why don't you post what hardware config you are using on this machine. AKA: IDE drives,memory,cpu,nics etc.Take Care,
BarryTake Care,
Barry -
Another lockup this-morning (or this weekend sometime rather). So its happening more often now. The fans and everything are working fine on the machine. I'll get back to you soon with some spec's and memtest results.
-
A defective keyboard could theoretically generate spurious keystrokes that could eventually bring down a computer.
Have you tried a different, known-good keyboard? Have you tried the same keyboard on a different, known-good PC?
-
If you are still encountering the hard freezes, I'd say it's time to take a backup and restore to another machine. See if your problems stop and go from there. The whole process won't take much longer than a reboot.
-
It (still) sounds like failing hardware, particularly if it's happening faster and faster. Have you tried running any hardware diagnostics yet?
-
Have not been able to do hardware testing yet. The entire network is basically offline while the pfsense machine is offline so we have not had the chance to do this yet. It's like trying to change the tire on a moving car :-(
-
jakobud,
get to work early in the am,,get a bootable cd with memtest86 on it.
1. boot the (linux)cd and at command prompt type in memtest
let it run for a couple hours.
,,,if this results "OK"(no red blocks)
2. Put a different keyboard on this pfSense machine as someone else mentioned.
3. restart pfSense machine.Post your results so someone else will know what has been found.
Take Care,
BC
