Webgui not accessible. nginx not running
-
My server died due to a psu issue and the psu was replaced. When I started it I could not access pfsense via the gui. It is running as a vm and I am doing all troubleshooting from the lan. I can ping the lan interface and can access the desktop of the pfsense vm and execute commands on the pfsense vm. pfsense is working as another vm using it as a firewall is able to access the internet fine. It was running fine for 6 months since it was built.
The ngingx log is showing logs from before the server died and nothing new. Check sockstat does not show any nginx process running.
Please could someone help me with some troubleshooting steps to see if i can ge the webgui working again. I do not want to rebuild as it has a lot of customization on it. Is there a way of getting the backups off it if I need to rebuild so I can restore?
Ver 2.7.2
sockstat | grep 'nginx' provides no feedback.
-
@MorrisFury I take it you tried restarting it via 11 on the console/ssh menu?
What errors or logs does that give?
-
I have restarted the pfsense vm multiple times during testing which should have the same effect as restarting the gui right?
I am not an expert on pfsense so the above is what i have been able to figure out to check thus far but i cannot get any info related to the getgrnam failure i am seeing and the faxt that the nginx process is not running. What other logs can I post?
-
@MorrisFury well the general system logs should tell you if something doesn't start..
if its a VM, just rollback to snapshot.. One the huge reasons to run it on a VM is ability to easy roll back to snap..
I would read that for some reason your wheel group is missing? get group name "wheel".. Is your group missing, first place I would look is simple cat of /etc/group
[24.11-RELEASE][admin@sg4860.home.arpa]/root: cat /etc/group # $FreeBSD$ # wheel:*:0:root daemon:*:1: kmem:*:2: sys:*:3: tty:*:4:Wheel should be there right at the top.
-
Unfortunately there are no snapshots. I need to investigate why.
There are backups which I can see when I want to do a restore on the desktop. I am trying to figure out how to get them off the vm so if I am forced to factory reset then I can recover from those.
The wheel group is missing. I do not understand how that can happen except for some kind of corruption that happened when the server died.
This is all I get when running that command.
I have tried recreating the group but have not been successful. Does this sound like OS corruption when the server died? It does not sound right that you could lose a group during a crash.
-
@MorrisFury said in Webgui not accessible. nginx not running:
Unfortunately there are no snapshots. I need to investigate why.
Oh, no need. Go talk to the person who has set up the VM.
Because the "wheel" user is missing, imho, don't search any further. Your entire /etc/group file is damaged. And this file isn't probably not the only one.
At best, the system file is severely damaged.Make use of the fact that you have a VM.
Create a new one, install pfSense, then import the latest good config.xml and you're back in business.
If you don't care about VM 'drive' make sure that the config.xml is backed up regularly (use acb ?!). -
I thought this is system corruption. The challenge now is to get the last backup off the vm. Without the gui this is going to be a challenge. Any ideas?
Then I will do a factory reset and restore and hopefully have everything back.
-
@MorrisFury said in Webgui not accessible. nginx not running:
Without the gui this is going to be a challenge. Any ideas?
A VM, right ? Major advantage (imho, the only one) : you have an easy console access.
When you see the menu, go option 8.
Not that much to do there if you don't know anything about the OS used etc, but .... like the PC 'cmd' box : just perfect to look at (and do things with) files.
It's here : /cf/conf/config.xmlPrevious backups are here : /cf/conf/backup/
-
Yes, I do have access to the console. I have been trying, with my limited knowledge of pfsense to repair it from there but not with much luck. Still figuring out how to get the files off.
I can see the config.xml there.
I can also see the backups.
-
You have a VM, so, afaik, you can 'mount' the virtual drive of the VM and access it with a file browser like "Windows Explorer" - the mounted virtual drive will pop up with a new driver letter.
Which means you can access = copy the file easily.Btw : Dos (Windows), MAC, Linux (and Unix before), FreeBSD and all the others, like your phone and so on : the filesystem is always, for decades now, very similar.
@MorrisFury said in Webgui not accessible. nginx not running:
my limited knowledge
Not really a valid reason
You didn't tell what VM you use. So I can't assist any further.I presume for now that you use the free, build in Windows "Hyper V" VM hypervisor.
I've used this VM just a couple of time in the past, as it was available with 'a click' just to play with it.
What about this question : I want to retrieve a a file from my virtual drive, and I know I use "Hyper V" (replace with yours if needed) :How to retrieve a file from the hyper V virtual drive ?
Click and you see the results.
remember, if you have a question, it's exterly rare that you are the very first that asks himself this question.
Millions have had the same question. Thus, and this is a law, Google (maybe other search engines also) knows already about it. So do you now ^^ -
@MorrisFury sp you have part of cert in your /etc/group file? that overwrote most of the file? Yeah something is way wrong..
If you have console access - the .xml are just text files you could copy and paste what is in them into a file that your consoling in from. Vs having to mount the drive, etc.
Did you have ACB set, if so your older configs are in the cloud and can just be loaded from the cloud once you get the replacement pfsense vm setup.
https://docs.netgate.com/pfsense/en/latest/backup/autoconfigbackup.html#restoring-a-configuration
This is the section you would be interested in
https://docs.netgate.com/pfsense/en/latest/backup/autoconfigbackup.html#bare-metal-restoration
But my guess is if you were not doing snaps, you prob don't have the device key and encryption key saved? But you may be able to pull those from the old system.. Have never looked into that because I have those saved off just in case, etc.
edit: Checklist after you're recovered
1: setup ACB saving required info key and password and hint
2: do a manual backup in acb. Prob want to do this at milestones of changes or now and then.
3: save off a manual copy of your config, again do at milestones or on some sort of schedule.
4: Make sure snapshots are taken of your VM on some schedule or manually every now and then update this snap. -
Thanks everyone for the help and info. Much appreciated and I have learned a bit more. It looks like I am going to have to do a full rebuild and re-educate myself on the custom configs and how to do them.
The vm is a generation 2 Microsoft hyper-v vm so I cannot copy it somewhere else and map it as a drive to access the config. It wil not allow me due to encryption and figuring out how to get access to it is going to make me old.
The config I had on this pfsense had two internet connections which were configured for auto failover and it took me months to get that working correct. So, learn again and do it again.
I am also facing issues with downloading the iso from Netgate. It is corrupt and I cannot extract it. It is probably due to the poor connectivity I have available in this area. So I am first going to solve that challenge.
I will definitely make sure that checkpoints and acb is configured on the next built. If I find something interesting I will post back here as I am not deleting the old build. I still think it can be fixed somehow.
-
@MorrisFury said in Webgui not accessible. nginx not running:
The vm is a generation 2 Microsoft hyper-v vm so I cannot copy it somewhere else and map it as a drive to access the config.
Look again at this :
https://youtu.be/OCgZAiIVP0k?si=eJdtGdV6iabsiZQc&t=364
@Gertjan said in Webgui not accessible. nginx not running:
How to retrieve a file from the hyper V virtual drive ?
Click and you see the results.
Here :
How to get files off your VM : https://youtu.be/OCgZAiIVP0k?si=llpTfcZpnzx3piNw&t=363
Btw : why doing things more complex then they need to be ?
Save a small small old desktop PC from the wastelands (just check if it has 64 bit CPU) - add a 5 $ NIC and you can forget about all the VM hassle. Go bare bone.@MorrisFury said in Webgui not accessible. nginx not running:
It is probably due to the poor connectivity I have available in this area
When you download stuff, TCP is used. If a packet comes in bad shape, it will get resend so what you receive will be a binary exact copy of what was send. CRC checksum are used all over the place, and these can't lie.
If bits were falling randomly, you couldn't even connect to this forum using HTTPS and post a message here ^^ A bad connection just becomes a somewhat slower connection, that's it.If your PC device can't write out the file, any file, an ISO file (then it can't neither maintain a correct vhdx == pfSense VM file system ) .... may by you should landfill it .... wait : transform it into pfSense ?!
-
I have created a snapshot of the faulty vm and I'm going to do a factory reset now. I am curious to see if that will work and if it is the entire vm that is corrupt.
I suspect I did config acb backups and if I did I will attempt a recovery from there.
-
@MorrisFury do you have or can you find your device key of the current broken vm, because if not - its not going to find it, that key is going to change when you create a new install.
Or do you atleast know the hint - possible netgate can find your backup from that, as long as you know what the password is on the encryption? @stephenw10 would be the guy I would go to for that sort of thing. If he can't do it himself he would know who or the steps required in a tac case, etc.
But if you have access via console or ssh - you can pull those old configs off, or even the current one - as long as they are not corrupted - for the life of me I not coming up with a scenario where the /etc/group file would get a cert or parts of cert file in it - without user intervention causing it.
-
So far I have been unsuccessful in getting the old configs off. I also do not know the hint.
I did the factory reset and the webconfigurator fails to start.
interestingly enough when the vm cam back i checked to see if the old backup configs were there for restore. They were all there but a restore did not fix the problem.
This looks like a proper corruption and i think even an acb restore will not fix this.
-
Some other things I tried:
I tried mounting to a folder on a network nas to copy the conf folder there. I could not as it required a username and password.
I tried using smb but samba is not installed and i was unable to install it as it could not find the package. I tried installing the password but sudo is required and it went absolutely belly up when i tried to install that. It went fully corrupt and I had to revert to a previous checkpoint.
I am not convinced this corruption was caused by a server switching off due to a failed psu. I am starting to wonder if I was hacked somehow. The chances are small as there is a natting router in front of the pfenses which only allowed one port inbound to a different device.
I am downloading the iso over another link and will try a new install.
-
There is something wrong with the Netgate ISO. I downloaded twice on two different links and using two different laptops and both are giving me the error below:
-
@MorrisFury did you validate the checksum?
the one for VM install should have this I believe - mine matches, I just downloaded it
633CAAD98DD86BA0DDA479CE1051A41B2E5DF64C41E29E82EFD7B1AE4A34B2A4https://www.netgate.com/hubfs/pfSense-plus-installer-checksums.txt
The name doesn't seem to match up - but the hash does.
Also use a different tool then built in windows extract, I use 7zip
-
@MorrisFury said in Webgui not accessible. nginx not running:
wrong with the Netgate ISO
Read the doc : Prepare a USB Memstick.
There is a tip :
