Cannot reboot pfsense hardware + software

Koploseus

Hello,

Since today, we tried to reboot the pfsense (soft + hardware) , but we keep get the same error :

ses0: SEMB SES Device
ses1 at ahciem1 bus 0 scbus3 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 1.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
mmcsd0: 8GB <MMCHC M32508 5.2 SN 1AC9FBB4 MFG 08/2018 by 112 0x0000> at mmc0 50.0MHz/8bit/65535-block
mmcsd0boot0: 4MB partion 1 at mmcsd0
mmcsd0boot1: 4MB partion 2 at mmcsd0
mmcsd0rpmb: 4MB partion 3 at mmcsd0
Trying to mount root from ufs:/dev/ufsid/5e3dcecfb68b7ca4 [rw]...
WARNING: / was not properly dismounted
uhub0: 8 ports with 8 removable, self powered

Does anyone has alread get this message ?? We have a netgate SG5100

viragomann

Obviously, the UFS filesystem is corrupt.

Boot into the single user mode and repair the filesystem by entering

fsck -y -t ufs

Koploseus

Yeah that a problem to.. while it's starting I can't choose the different choice on the menu :o it's starting automatically !

viragomann

You will have to connect a computer using a console cable and establish a console connection. Then you can interact with pfSense.

When you see this screen

just press 2.

Koploseus

Already the case, when I see this screen it's automatically running alone so I can press anything :/ that also the problem they I have , I use minicom with the default value described in the netgate doc

EDIT : i can of manage to press 2 ... but when i do the command that you gave me, its says : fsck not found
Witch is weird, because i can find it in the sbin folder

Gertjan

Then do what the manual says https://docs.netgate.com/pfsense/en/latest/hardware/troubleshooting-disk-check-errors-fsck.html ^^

Koploseus

Yeah already tried .. ^^ but no success cause its says "fsck not found" ...

Gertjan

You did cd to /sbin first ?
pwd confirmed that are your current dir is /sbin ?
ls -al fsck confirmes now that you 'see 'fsck' ?

Koploseus

cd not found either...
When i do ls /sbin/ i can see fsck

EDIT: For info my shell got an "OK" instead of a sh or bash stuff

Gertjan

@Koploseus said in Cannot reboot pfsense hardware + software:

EDIT: For info my shell got an "OK" instead of a sh or bash stuff

Ah. No, keep that one for when there is really nothing on the TV any more.
Reboot, and choose option2 - single user mode.
You'll be having a more classic shell = sh

Koploseus

@Gertjan yeah , i did and its keep giving me the "OK" shell

Gertjan

The OK prompts is the very limited boot loader shell.

That will be option 3.

You should use option 2.
It's the same as 1 - but no multi tasking.
It even proposes you to use the /bin/sh at the ned of the boot.
Which allows you to execute fsck and other classic commands.

edit : https://forum.netgate.com/topic/140531/how-to-install-pfsense-with-usb/7

Koploseus

Yeah i am pressing 2 ahah , but either it start on 1 or 3... or 4 but when i press 2 .. i get the "OK" shell ahah

Gertjan

I propose you take the shortcut solution :
Backup your config.
Contact Netgate support - ask for a up to date firmware version.
Install it - and enjoy.

You never will know what happened, but consider this : trying to figure out issue while booting is not an easy one.

Koploseus

Yeah thats a problem.. we had a cyber attack, and we wanted to know what happened, where did they came from and how.

We need those answer , that why i need to fix it! :)

and we don't have the support of netgate (we didn't get the premium support)

Gertjan

@Koploseus said in Cannot reboot pfsense hardware + software:

We need those answer

Does the 5100 has a removable storage ?
Consider backup your drive for later analyses.
I strongly advise you to contact Netgate. They are very capable of analyse your setup fast and give the best advise.

@Koploseus said in Cannot reboot pfsense hardware + software:

we didn't get the premium support

They always, without limit of time, support it by giving you a recent clean firmware.

Btw : being the victim of a cyber attack needs more as a premium support.
You have to have the knowledge 'in house' to counter this - or get this knowledge fast.
The zero €-$ solution is :
Keep all pfSense settings to default - with one exception : you are allowed to give it a name.
Only allow ONE device on LAN : the device you use to admin it.
All other devices on others OPTx LANs, and they should not be able to connect to the GUI and / or telnet and or SSH. This way, pfSense is save, even if you host zombie networks on your OPTx LANs.
Easy to apply.

Remember : the biggest dangers on your networks are :
The admin.
The LANs users.

Koploseus

No removable storage :/

the backup of the config is in the pfsense ahah

i think that the "only" problem right now it that the boot menu (for the sh shell) is "too quick" and that i don't have time to press 2.. or its reading everything else but not the button 2

stephenw10

It should wait 3 seconds at the boot loader menu, you don't have to be that quick. If you do reach the OK> prompt you should be able to enter boot -s there to reach single user mode.
It looks similar to what I might expect if the serial console is not set as primary. It would be set as primary in the SG-5100 unless you imported a config from a previous device?

Steve

Koploseus

boot -s didn't work.

I have an update, i have put pfsense livecd on a USB, boot on it, i have done fsck on the right disk... now i have this :

Trying to mount root from ufs:/dev/ufsid/5e3dcecfb68b7ca4 [rw]...
uhub0: 8 ports with 8 removable, self powered
random: unblocking device.
CPU: Intel(R) Atom(TM) CPU C3558 @ 2.20GHz (2200.07-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x506f1 Family=0x6 Model=0x5f Stepping=1
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x101<LAHF,Prefetch>
Structured Extended Features=0x2294e283<FSGSBASE,TSCADJ,SMEP,ERMS,NFPUSG,MPX,PQE,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE,SHA>
Structured Extended Features3=0xac000400<IBPB,STIBP,ARCH_CAP,SSBD>
XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
IA32_ARCH_CAPS=0x69<RDCL_NO>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics

and its freezing after that (never had that before)

and also, before i had 1 green light and 2 red ... now i have 2 green light.. and 1 down (power are green) but nothing for data i think

While i mount my disk on my usb key i was able to find my backups .. didn't get them tho.. didn't know how to do it on minicom

for the serial concole, no previous config, all default