PfSense 2.2 503 - Service Not Available
-
Hey guys
Since today we have the same issues. I shutdown the Firewall (no virtualisation / native installation with lagg) at friday and startet it at sunday. After the first start i got into the https webpage, had an issue with the apinger service and restart over ssh.
After the restart I can't connect to vpn, ssh and the webpage- same issue as all here.
-
Hey guys
Since today we have the same issues. I shutdown the Firewall (no virtualisation / native installation with lagg) at friday and startet it at sunday. After the first start i got into the https webpage, had an issue with the apinger service and restart over ssh.
After the restart I can't connect to vpn, ssh and the webpage- same issue as all here.
No SSH?`No VPN -> other Problem. Here are only People with no access to the webgui. Everything else is fine
-
Hey guys
I found the issue today. The system killed the group "wheel" and after this it was not possible for the system to create the
php-fpm.pid and php-fpm.socket file.Greetz
-
This is massively, massively bad. Like "blocker" hair on fire bad. One of the great things about pfSense has been the appliance-like reliability which permits confident headless operation and operation in challenging environments where power isn't reliable. It has always just come back from power failures.
Today our UPS went batshit. It happens a lot here (in Iraq) where the AC line voltage varies from 80-260V, 40-65Hz, and goes out about 6-8 times per day. A UPS just doesn't last long under that kind of abuse (and this is a tiny little logic supply fanless box running on a SmartUps 3000 rack-mount, so it should have at least a day's run-time, which it needs on generator service days).
Today I got the 503 Service Unavailable error after spending half the day replacing the UPS. I take the time to drag a monitor up to the data cabinet (sealed, air:air self-cooled) and I see the attached:
(searchable as)
[ERROR] [pool lighty] cannot get gid for group 'wheel'
[ERROR] FPM initialization failedAnd I tried restarting several times, restarting the webconfigurator (11), and restarting PHP-FPM (16), and then found this thread and realized it was to no avail. Time to reconfigure the network to permit download of the current version (this one has been UI upgraded for the last 3 years) and hope the last config backup has the DHCP assignments for the 60 or so machines that were added recently.
How far back would one have to downgrade to escape the over-eager fsck?
Bluethunder's suggestions are good, but in my case, it is the UPS that is the problem.
Migrating to boot from ZFS would escape FSCK completely. I boot from ZFS on my FreeBSD servers already and it is quite reliable and fairly easy to configure now that it is integrated into the 10.1 installer.
It might also help as a stop-gap to specify fsck_y_enable="NO" in /etc/rc.conf. You'd hang at startup, but at least you could intervene in the FSCK process and possibly prevent the system from eating itself.
-
We have the same issue with one in the Bahamas. Power issues over the weekend and now not able to access the router. It is running but not passing any traffic. No DHCP and error 503 on the gui. Unfortunately SSH is turned off.
-
If you can get console (or have someone do it) it is pretty easy to pull the config off```
/cf/conf/backupA remote KVM with virtual media adapters may be essential with newer versions of pfSense that are at risk until this is addressed (which may be never).
-
Until we figure out how to fix this (it's a FreeBSD/fsck issue) it might also be wise for those prone to multiple instances of it to keep a tarball of /etc somewhere… If it breaks then untar the file back over /etc, reboot, and keep going.
-
Is there any way to disable fsck altogether (without breaking non-interactive boot)? Since, this does more harm than good apparently.
-
No. If we disable the call to fsck it won't mount the slice and will drop to a console… and the fix is to run fsck. catch-22.
-
If it is possible to recover from a tarball, then perhaps a script that runs on startup that tests for some indication of this problem and automatically executes recovering /etc from the archive? An ugly hack, but the problem I could easily see for myself (pfSense instances running 20 hours of travel apart) is that the manual fix is not an easy talk-through for a non-technical hands-on person and if the system goes down and it is awfully hard to get in from the WAN side to do the work remotely.
I don't want to attempt this again unintentionally, but does SSH successfully start when this happens and are the rules that permit WAN side access working?
Otherwise a remote box pretty much necessitates a remote KVM on an accessible IP outside the firewall to give console access. Having a tarball of /etc/ squirreled away would save from reinstalling and I'll prepare my instances for the worst by doing that and making sure WAN side SSH works.
-
It may be possible to make an ugly hack like that, but it's not something we'd actually code up and put in the images (not that I can see happening anyhow) unless things got really desperate.
For those especially prone to this, you might also try adding "sync,noatime" (sans quotes) to the mount options for the disk in /etc/fstab – in my testing it still ran fsck and found errors but I didn't see any corruption. Though whether that was pure luck or due to the change is unclear yet. For example:
Before:
/dev/ufsid/552d6d027debc466 / ufs rw 1 1
After
/dev/ufsid/552d6d027debc466 / ufs rw,sync,noatime 1 1
Disk performance may take a slight hit for that but if it does help, it's worth the extra stability.
-
This seems like a sensible fix. It should help reduce the risk of corruption on data loss. The mitigants seem to me:
-
Make sure SSH access works from wherever one needs to manage a dead firewall from (probably WAN)
-
Backup /etc to someplace sensible
-
adjust /etc/fstab to trade performance for reliability
Hopefully this will get sorted.
I would think that moving to boot on ZFS would be a reasonable migration path. No more fsck.
-
-
zfs is more of a long term goal (and it is one of our goals, definitely) – not something we can implement fast or without lots of testing, and not an option for upgrades. So it is great for the future, but not what we need to fix right now.
-
I'm having this same issue at a remote site.
does it just kill the GUI or does it kill the routing as well? I can still ping the box but I'm really hoping it's still allowing traffic to flow through…...
-
I just ran into this issue on two VM instances of pfSense I was setting up. Iv been struggling to get CARP working between two KVM hosts and would reboot the hosts without shutting down the pfSense VM first (simulating power failure). Have now re-installed the pair 3 times. Trying the ",sync,noatime" option in /etc/fstab to see if it prevents needing to re-install.
-
yaplej: please report if it does help - seems like you're doing the right kind of testing to verify. I've made the changes on all my pfSense instances: fingers crossed power doesn't go out at a remote site and kill it.
-
I had this happen after a power outage. Version 2.2.0. I received 503 error on the web console, tried enabling ssh from the console with no luck, routing did work as long as I gave the workstation a static IP and and a DNS other than pfSense. I ended up reinstalling to fix the issue. I installed several instances this time to different partitions so I can at lease boot something. http://www.blog.unflap.com/2009/12/28/dual-boot-pfsense-for-testing-new-versions/ You can just choose to go back to the main menu instead of reboot and can install as my instances as you need.
Note for some reason a clean install of 2.2.2 would not boot on my Dell Optiplex 320. I would get the F1, F2, F3 selection and then reboot in an endless loop. It had listed ad0 for the drive. I finally went back to 2.1.0 and installed multiple instances without issue to ad4. Not sure why the drive letter change. All three instances updated to 2.2.2 and a config restore just fine.
-
I just had this happen after installing Bandwidthd on a new test 2.2.2 config that I spun up today. No power outage or anything else strange going on. I selected the LAN interface and clicked Save. Then I went to Access Bandwidthd and got:
Please start bandwidthd to populate this directory.
I went back to the Bandwidthd config page and again clicked Save, and that's when I got the 503. All WebGUI attempts now give 503 until I restart PHP-FPM. I have never, ever seen this before until now with bandwidthd.
With PHP-FPM restarted, everything appeared to be normal again. Then when I went to my dashboard, all the widgets were good except for the NTP widget which showed 503, and my LAN traffic gaph which shows
Cannot get data about interface vmx1
ntopng seems to have died as well.
-
That's likely a different root cause than the filesystem corruption others are seeing.
-
And the problem persists through a reboot. Now both squid and squidGuard, which were working fine, are crashing as fast as they can start. WebGUI gives the exact same display as my screencap, with 503 for the NTP widget and the LAN graph not able to talk to vmx1. Bizarre.
-
With the squid* censored, I's pretty much possible you are getting an unclean reboot – which is perfectly enough to screw the filesystem.
On that note, I wonder if anyone tested with fsck from some previous FBSD versions. The one in 10.1 is simply mad.
-
I didn't see any warnings about a dirty filesystem. The system never crashed or anything that might cause an obvious filesystem error. fsck comes up with about a dozen unreferenced inodes though.
-
I left it running overnight and came back to a dead GUI and an endless stream of:
swap_pager_getswapspace(n): failed.
I lost my VMware heartbeat around 1am and it didn't return for 45 minutes.
Hard reset allowed me to boot. The LAN graph is still borked but the NTP widget now seems to be working again.
-
Hi,
I just encontered the same problem today. I'm running pfSense in a VirtualBox on an old computer probably with HDD troubles.
PHP-FPM could not start.I had to reinstall and restore a backup of config.xml.
I hope not to experience this trouble to often.
-
Version 2.2.2 direct install (no vm).
Power cut yesterday and problems here too 503 & 500 errors, ssh wasn't working or dhcp.
-
For those who want to avoid having their filesystem corrupted, here's a quick shell one-liner to add sync to fstab and remount / with sync:
cp /etc/fstab /etc/fstab.orig; /usr/bin/sed -i '' 's/^\(\/.*[[:space:]]*\/[[:space:]]*ufs[[:space:]]*\)rw\([[:space:]]*[[:digit:]][[:space:]]*[[:digit:]]\)$/\1rw,sync\2/' /etc/fstab; mount -o sync /
(Be sure to copy the entire line)
This will happen automatically during the 2.2.3 upgrade, and new installs of 2.2.3 will have sync enabled.
-
I also started a wiki doc about it, with the commands there as well. https://doc.pfsense.org/index.php/Filesystem_Corruption_%28503_errors,_cannot_get_uid%29
-
The fix for this is quite simple
Go to the command prompt…
vi /etc/group
add a line at the end using the same syntax as the line above it. Just use wheel as the group name and use a GID number that is not already used in the file.
Save it and restart the PHP service. The error will go away and you will be golden. -
Another fix will be : get the repaired pfSense version 2.2.4 …
-
Another fix will be : get the repaired pfSense version 2.2.4 …
Sounds definitely better than messing with screwed /etc/group manually.
-
I am getting a 500 internal server error with the latest version - 2.2.4 RELEASE.
Restarting PHP-FPM solved the issue, but does anyone know the root cause of the 500 error? I am seeing a lot of older threads coming up with that error and a couple posts above this one Gertjan says that this released fixed the 503 error.
Hopefully you have more information.
-
I am getting a 500 internal server error with the latest version - 2.2.4 RELEASE.
Restarting PHP-FPM solved the issue, but does anyone know the root cause of the 500 error? I am seeing a lot of older threads coming up with that error and a couple posts above this one Gertjan says that this released fixed the 503 error.
Hopefully you have more information.
2.2.4 fixes possible file system corruption - if the corruption was such that the system would boot but PHP would have a problem or… then there might have been 503 issues that now are not a problem.
But there are still times when "something happened" - I think this phrase is now a Microsoft trade mark? :) - and PHP can need a kick. e.g. some interaction with bandwidthd has been a cause that has not been turned into a reproducible and then fixable problem yet.
-
I am getting a 500 internal server error with the latest version - 2.2.4 RELEASE.
Restarting PHP-FPM solved the issue, but does anyone know the root cause of the 500 error? I am seeing a lot of older threads coming up with that error and a couple posts above this one Gertjan says that this released fixed the 503 error.
Hopefully you have more information.
2.2.4 fixes possible file system corruption - if the corruption was such that the system would boot but PHP would have a problem or… then there might have been 503 issues that now are not a problem.
But there are still times when "something happened" - I think this phrase is now a Microsoft trade mark? :) - and PHP can need a kick. e.g. some interaction with bandwidthd has been a cause that has not been turned into a reproducible and then fixable problem yet.
Thanks for the response Phil!
-
I am getting a 500 internal server error with the latest version - 2.2.4 RELEASE.
Restarting PHP-FPM solved the issue, but does anyone know the root cause of the 500 error? I am seeing a lot of older threads coming up with that error and a couple posts above this one Gertjan says that this released fixed the 503 error.
Hopefully you have more information.
Same thin here, I run pfsense on a virtual machine (qnap) .
This is a clean and new install of 2.2.4 and suddenly 503 !!
Luckaly the rest php-fpm works temporarely
Is this something that was supposed to be fixed in this release?
(I too have bandwidthd package installed, is it really this package? will it be fixed if I remove it?)
-
I have started getting 503 errors on a 64-bit pfSense 2.2.4 host that was upgraded from 2.2.3.
I'm not using bandwidthd.
The UPS on this host tripped overnight and I don't think NUT managed to shut the host down properly before the UPS powered off.
I am going to rebuild it from a fresh CD later today. -
Working fine now after rebuild from CD.
-
I spoke too soon. The WebGUI operates normally for up to a day after a reboot of the host. If I want to use the GUI after that I usually have to first SSH to the box and select option 16 to restart PHP-FPM.
-
i have same problem but i am usuing pfsense 2.3.2 so can i apply same solution for 2.2.2 ?
thanks