Help Needed: Upgrade to 2.4.3: /var/run filesystem full

airhead00

Hi there, upgraded from 2.3.5 to 2.4.3 and am getting:


da0: Attempt to query device size failed: NOT READY, Medium not present 
da0: quirks=0x2 <no_6_byte>pid 119 (php-cgi), uid 0 inumber 7 on /var/run: filesystem full
Failed to write core file for process php-cgi (error 28)</no_6_byte>

The system locks up serial and I'm stuck.

Running on a pcengines apu with a 24gb msata.

Not quite sure how to troubleshoot outside of doing a mass reinstall. If that's necessary, I'll go for it but hoping someone can help me with a different approach. Also, backups were not taken pre-upgrade. If re-install is necessary how might I go about pulling configs?

Thanks.

stephenw10

See what's taking up space on /var/run. It's probably something left over that shouldn't be there, nothing very large should.

Try:

du -ah /var/run

Steve

airhead00

Thanks! How do I get into a shell if I get a locked up on boot?

stephenw10

Ah, so it did upgrade but only then showed that error…

It depends how far though the process you are. If you see the boot menu you can try booting to single user mode.

If not you might reach a shell from the error by escaping the script with ctl-c. No promises there though.

Steve

airhead00

Indeed. I tried booting into single user with```
boot -s

airhead00

Ah, found https://doc.pfsense.org/index.php/Can_I_restore_a_backup_without_the_WebGUI which answers my second question.

stephenw10

Yes the recover config script in 2.4 is much improved. A full /var/run would not stop the config being read.

The only worry is that the filesystem may not actually be full, a bad drive can also sometimes present like that.

Steve

airhead00

I'm having a ton of trouble mounting the disk. I'm starting to think it's hosed but I can still "ls" pfsense's boot (though I can't drop into a shell) any thoughts on how I might be able to get it to mount?

airhead00

all good. via: http://ghantoos.org/2009/04/04/mounting-ufs-in-readwrite-under-linux/

blex

So you did get the disc mounted. Since I have the same problem on one of 4 FWs I would bee interested to hear if you found out what took the space away or if it was another problem?

stephenw10

Yes, I'd be interested to see what rogue file filling /run.

Steve

blex

As for my nightshift. I have talked with the support. There is no good way finding what happened at update. The suggestion was always have a serial console log running when you do the upgrade. Then you have a chance on catchen what was wrong.

In my situation there where only haproxy and securitata installed.

I ended up grabbing the last konfig (since I but carp in mainanance) from the old installation while booting from a usb stick with 2.4.3 on it.
I then did a reinstall and restored the config. Everything is now working.