WebGui Not Accessible - "Bootup complete" every 60-90 Minutes
-
Greetings,
I have a SG1100 and I can not access the WebGui from either WAN or internally once I connect via OpenVPN. I am not home right now. I can access everything on my network via OpenVPN. Outgoing connections are working fine. However, I am not able to access the WebGui and SSH is disabled. I can't access the console remotely. I do have the ability to restart the power remotely to force a reboot.
The only changes I made recently is I started to backup my server to a cloud backup. Therefore, I have had increased uploads the last few days.
I have been getting email notifications about every 40 minutes "Bootup complete." Below are all of the alerts I have received just today.
Any suggestions?
0:14:25 Bootup complete
1:02:34 Bootup complete
1:53:44 Bootup complete
2:38:46 Bootup complete
3:24:44 Bootup complete
4:13:59 Bootup complete
5:01:46 Bootup complete
5:49:48 Bootup complete
6:33:35 Bootup complete
7:21:42 Bootup complete
8:01:24 Bootup complete
9:28:44 Bootup complete
10:36:47 Bootup complete
11:09:32 Bootup complete
12:20:26 Bootup complete -
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
Any suggestions?
What happens just before the "Bootup complete" message ?
The power is safe ? Are you using an UPS ?
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
increased uploads
Even several hundreds of megabits in or out shouldn't pose any issues.
-
With reboots that often I would open a ticket with us: https://go.netgate.com/
It's either a power issue or a hardware problem I would guess. It could be something in the config causing an isuse. You will need to see the logs for that though.
Console output would be best for diagnosing it.
Steve
-
@Sparky17 : what are the packages you are using ?
-
@Gertjan said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
Any suggestions?
What happens just before the "Bootup complete" message ?
Unfortanetely, I can not see the logs ass I can't access the WebGui or Console right now. I just get the "Bootup complete" message as an email notification.
The power is safe ? Are you using an UPS ?
I am using a UPS for clean power, but I haven't fully tested the shutdown on power outage so it may of had an unclean shutdown during a power outage about 6 weeks ago. However, I have only been having issues for less than a week.
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
increased uploads
Even several hundreds of megabits in or out shouldn't pose any issues.
-
Thanks Steve. I opened a ticket.
-
Unfortunately I can't say with certainty. Since I can't login to the WebGUI I do not know exactly. However, going off of my memory here is what I think I have installed and running.
ACME
Avahi
Darkstat
OpenVPN Client Export
pfBlockerNG
PIMD
Service Watchdog -
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
Unfortanetely, I can not see the logs ass I can't access the WebGui or Console right now.
Enable the second to best pfSense User Interface : the SSH access.
The console comes first, the GUI is third. Do not leave home without a decent SSH access.@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
Service Watchdog
That one is for experts only.
And they - the expert - said they would never use a tool like that.
It never saves you, but it is excellent in creating does create havoc.
When you use this package, do not leave your system alone. Not even for a minute.@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
ACME
OpenVPN Client ExportThese do close to nothing.
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
pfBlockerNG
I've seen that one killing a quad core I7 16 Gbytes system in less then a minute. True, the guy was trying to include every DNSBL feed available planet earth.
pfBlockerNG should be set up "correctly", and there will be no issues.
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
PIMD
Don't know what PIMP is.
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
Darkstat
I didn't used that package for years.
From what I recall, it should be put under daily surveillance as it can eat up (disk and processor) resources on your system. It produces logs with the rate of the traffic.edit : Btw : take some time to finish the install of the most important package : NUT.
-
I am embarrassed to say the reason for not accessing the WebGUI is user error! Sorry, for wasting anyone's time. I should not make changes to my firewall late at night apparently.
I changed it to port 4443 and forgot about the change!
I was able to check the logs and my guess is this is a "bad thing" and might have been caused from not powering down correctly when the UPS ran out of battery.
Jul 27 09:08:35 pfSense kernel: Trying to mount root from ufs:/dev/ufsid/5c11689b7c8fb123 [rw,noatime]...
Jul 27 09:08:35 pfSense syslogd: sendto: Network is unreachable
Jul 27 09:08:35 pfSense kernel: WARNING: / was not properly dismounted
Jul 27 09:08:35 pfSense syslogd: sendto: Network is unreachable
Jul 27 09:08:35 pfSense kernel: WARNING: /: mount pending error: blocks 88 files 1 -
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
Jul 27 09:08:35 pfSense kernel: Trying to mount root from ufs:/dev/ufsid/5c11689b7c8fb123 [rw,noatime]...
Jul 27 09:08:35 pfSense syslogd: sendto: Network is unreachable
Jul 27 09:08:35 pfSense kernel: WARNING: / was not properly dismounted
Jul 27 09:08:35 pfSense syslogd: sendto: Network is unreachable
Jul 27 09:08:35 pfSense kernel: WARNING: /: mount pending error: blocks 88 files 1Yep, the file system went belly up.
As said, if you have an UPS, hook up that data cable, and set up NUT.
Right now, you're good for the fsck video (Youtube => Netgate) -
It will show that whenever the router is rebooted without being shutdown cleanly. It usually means only that the filesystem is not marked clean and will recover from that itself.
It obviously is doing so since it is still booting. That may not always be the case however.If the file system becomes sufficiently damaged it may not be able to recover itself and will require interaction at the console. If that happens you will lose connection to it if you're remote only.
Steve
-
Thanks for the feedback.
Should I force an automatic check? I am worried that i might loose remote access.
Forcing an Automatic Check
If the system is booting successfully but a filesystem issue is suspected, connect to the console or SSH and use the reboot menu option (5), followed by F to reboot and force a filesystem check.https://docs.netgate.com/pfsense/en/latest/hardware/troubleshooting-disk-check-errors-fsck.html
-
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
Forcing an Automatic Check
should typically be done on site, because the system would run in one-user mode, which means ; just the console access. No WebGUI, no SSH.
You could probably launch it from 'else where' but I wouldn't take that risk.look at the video, have some one logging in for you @site, and ask him to "video" the screen, and tell that person what to type in and when ;)
-
It's unlikely that will help prevent it rebooting given that is is still booting.
If you were local to it I would suggest you re-install it clean to be sure.
What can prevent damage to the file system is enabling RAM disks:
https://docs.netgate.com/pfsense/en/latest/book/config/advanced-misc.html#ram-disk-settingsHowever you would need to increase the size over the default (at least double) with that list of packages you have running. It will reboot if you enable them and if for any reason it fails to boot again.....
If it's still rebooting as often as your logs show I think it is inevitable it will fail to boot at some point if you do nothing unfortunately. I would prepare to lose access there if you have no alternative way to reach it. Enabling RAM disks, assuming it reboots that first time, will reduce the chances of losing that connection.
Steve
-
Hi Steve,
Thanks. Just to be clear you do not recommend forcing an automatic check (SSH, option 5, F).
However, you do recommend enabling ram disks? I can disable some packages to help.
This will just be "limp mode" to try to keep it up and running for the next two weeks. Once I get home I will re-install from scratch.
-
I don't believe forcing a filesystem check will help since it is still fully booting and not reporting errors, missing files etc.
Enabling RAM disks will substantially help prevent irrecoverable damage to the filesystem due to reboots.
It's hard to know what to set the disk sizes to given the package you have running. What does your ram usage look like currently?
I always use at least double the default, so 80 and 120MB. That is sufficient for mist things, including pfBlocker with only easylist loaded.I would disable darkstat for now.
What lists do you have pfBlocker loading?
What services is the watchdog checking?
Steve
-
Thanks Steve.
I am now able to access the WebGUI and can confirm my packages.
ACME
Avahi
ipsec-profile-wizard
nut (Setting up now!)
openvpn-client-export
pfBlockerNG
pimd (Currently disabled, but still installed)Right now it says 40% of 990MiB are in use for memory.
-
Ah, OK then you can afford to use those double default numbers for ram disks if you decide to enable them.
It's hard to recommend that with absolute certainty when you don't have direct access but if it's still rebooting after running the forced fsck I think I would do that.
If you can do anything to be prepared to lose connectivity you should do that either way.Steve
-
@Sparky17 said in WebGui Not Accessible - "Bootup complete" every 60-90 Minutes:
I do have the ability to restart the power remotely to force a reboot.
Hello!
What are you using to control the power?
I cant tell from the thread if you resolved your frequent reboot issue, but I have had power control devices (iboot, digital-loggers, etc...) "misbehave" and cause this sort of problem. I normally have these between the UPS and device, so power monitoring (NUT) doesnt help prevent a power cut and possible disk corruption.
John
-
I have a APC UPS and then a APC 7900 PDU connected between them.
I do have NUT setup and appears to be working right now. However, next time I am home I am going to manually cut the power and test just to double check.
So far so good, as it has now been over 3 hours without a reboot since I forced an automatic system check on reboot. However, I did make some other changes (disabled PIMD, etc.) that may affect the frequency of the reboot. I am fully backed up and hoping for the best, but prepared for the worst. I will reinstall from scratch and restore from backup once I am back on site.
I also plan on enabling RAM disks now as well.
