No clue what happened… unable to make any changes via GUI or SSH to box
-
It's been a long day…
This started out with some folks from Germany not being able to reach a server that I have here at my office. I use pfBlockerNG and figured it was that since I block Germany. Using geoscreenshot.com I could see that a connection from Germany wasn't getting in. I turned off pfBlockerNG (and Snort) and I was still having the issue. I could see the IP of the German folks getting blocked.
Couple this with the problem that I constant get the 502 Bad Gateway error and my crap day just got crappier.
WTH, I'll reboot... Bad idea. It took no less than 4 reboots to get the system running again (one of them forced a fsck which took a while).
I'm now functional but just barely. I can't make any gui changes at all. I've also lost the ability to SSH into the box. Anytime I try to make a change I receive this:
<error>Fatal error: Cannot create references to/from string offsets nor overloaded objects in /etc/inc/xmlparse.inc on line 103 Call Stack: 0.0002 124668 1. {main}() /usr/local/www/system_advanced_admin.php:0 0.0341 850152 2. write_config() /usr/local/www/system_advanced_admin.php:296 0.0948 979640 3. cleanup_backupcache() /etc/inc/config.lib.inc:593 1.9734 1371640 4. parse_xml_config() /etc/inc/config.lib.inc:859 1.9734 1371668 5. parse_xml_config_raw() /etc/inc/xmlparse.inc:178 2.0390 1737880 6. xml_parse() /etc/inc/xmlparse.inc:217 2.0390 1738072 7. startElement() /etc/inc/xmlparse.inc:217 PHP ERROR: Type: 1, File: /etc/inc/xmlparse.inc, Line: 103, Message: Cannot create references to/from string offsets nor overloaded objects</error>
I try to enable SSH from the console and it says it's already enabled so it prompts me to disable, I figure what the heck, disable and then re-enable. Trying this gets me a similar error.
<error>Fatal error: Cannot create references to/from string offsets nor overloaded objects in /etc/inc/xmlparse.inc on line 103...</error>
Thoughts? Corrupted config maybe? I have a pretty recent backup but I'm afraid to apply it. Will it interrupt internet access for my folks?
Thanks in advance for any guidance
-
I would guess failing storage.
-
Thanks for the prompt reply!
I'm using GEOM but I can't add the widget to the dashboard to see if the array is healthy. Is there a way to from the console?
-
They look fine in Diagnostics - GEOM Mirrors…
-
Hmmm. Something else must have gotten corrupted. About all I can think of doing is dropping to single user mode and running a few /sbin/fsck -y / commands to see if that helps and if not a reinstall / restore.
I would take a new config backup if that is working.
-
Thanks, I'll give that a try!
-
It still sounds like failing storage hardware.
All of the symptoms line up, and though it may have been able to rebuild the mirror, it's probably a ticking time bomb.
-
Thanks, I'll replace the drives and report back.
-
I broke the mirror and swapped out ada1. It finished the rebuild so I broke the mirror again and replaced ada0. After a rebuild I'm still unable to make changes to anything. Here's what I receive:
Fatal error: Cannot create references to/from string offsets nor overloaded objects in /etc/inc/xmlparse.inc on line 103 Call Stack: 0.0002 128780 1. {main}() /usr/local/www/status_logs_settings.php:0 0.0340 852800 2. write_config() /usr/local/www/status_logs_settings.php:195 0.0948 982232 3. cleanup_backupcache() /etc/inc/config.lib.inc:593 2.0669 1374200 4. parse_xml_config() /etc/inc/config.lib.inc:859 2.0669 1374228 5. parse_xml_config_raw() /etc/inc/xmlparse.inc:178 2.1324 1740504 6. xml_parse() /etc/inc/xmlparse.inc:217 2.1324 1740696 7. startElement() /etc/inc/xmlparse.inc:217 PHP ERROR: Type: 1, File: /etc/inc/xmlparse.inc, Line: 103, Message: Cannot create references to/from string offsets nor overloaded objects
Any additional thoughts?
TIA!
Dale
-
Yeah, reinstall the thing and restore a config backup. Rebuilding corrupted mirrors does no good.
-
Dang, I was afraid that was going to be the next step…
Thanks!
