"Corrupt XML" kills 2.60
-
After running flawlessly for 5 months my 2.60 on Protectli FW4C started having problems last week. I couldn't log in, and when I tried to boot into the console, it never finished booting. It gets to here and stops:
I figured the easiest way back was to just reinstall and restore a backup from March. I reinstalled just fine, but when I attempted to Restore my XML file from a browser, my browser window displayed:
Fatal error: Uncaught Exception: XML error: SSHDATA at line 3634 cannot occur more than once in /etc/inc/xmlparse.inc:89 Stack trace: #0 [internal function]: startElement(Resource id #6, 'SSHDATA', Array) #1 /etc/inc/xmlparse.inc(188): xml_parse(Resource id #6, 'WGUXTMry6ps4TWr...', false) #2 /etc/inc/xmlparse.inc(149): parse_xml_config_raw('/conf/config.xm...', Array, 'false') #3 /etc/inc/config.lib.inc(134): parse_xml_config('/conf/config.xm...', Array) #4 /etc/inc/config.inc(144): parse_config() #5 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...') #6 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...') #7 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...') #8 /etc/inc/config.gui.inc(39): require_once('/etc/inc/notice...') #9 /etc/inc/auth.inc(33): require_once('/etc/inc/config...') #10 /etc/inc/authgui.inc(27): include_once('/etc/inc/auth.i...') #11 /usr/local/www/guiconfig.inc(60): require_once('/etc/inc/authgu...') #12 /usr/local/www/index.php(46): require_once('/usr/local/www/...') #13 {m in /etc/inc/xmlparse.inc on line 89 PHP ERROR: Type: 1, File: /etc/inc/xmlparse.inc, Line: 89, Message: Uncaught Exception: XML error: SSHDATA at line 3634 cannot occur more than once in /etc/inc/xmlparse.inc:89 Stack trace: #0 [internal function]: startElement(Resource id #6, 'SSHDATA', Array) #1 /etc/inc/xmlparse.inc(188): xml_parse(Resource id #6, 'WGUXTMry6ps4TWr...', false) #2 /etc/inc/xmlparse.inc(149): parse_xml_config_raw('/conf/config.xm...', Array, 'false') #3 /etc/inc/config.lib.inc(134): parse_xml_config('/conf/config.xm...', Array) #4 /etc/inc/config.inc(144): parse_config() #5 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...') #6 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...') #7 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...') #8 /etc/inc/config.gui.inc(39): require_once('/etc/inc/notice...') #9 /etc/inc/auth.inc(33): require_once('/etc/inc/config...') #10 /etc/inc/authgui.inc(27): include_once('/etc/inc/auth.i...') #11 /usr/local/www/guiconfig.inc(60): require_once('/etc/inc/authgu...') #12 /usr/local/www/index.php(46): require_once('/usr/local/www/...') #13 {mAttempting to reload the webGUI or go Back in the browser resulted in similar messages.
I rebooted the FW4C, and pfsense never finishes booting, with the console displaying messages like:
/Users/steven/Pictures/Photos Library.photoslibrary/resources/derivatives/C/CADBB525-A0D6-41D0-84CA-0B23FC168EEC_1_105_c.jpegSomewhere in that string of error text I saw the words "corrupt" and "XML".
At this point, the box is dead. Entering 4 or 5 and then Enter results in another string of errors like the photo above.
Rebooting ends up with a similar display, and I never even get to the text menu shown above.
I reinstalled a few times, and every time I try to restore the March backup, I get the same result.
I eventually rolled all the way back to a December backup, so my office is up and running, but I'm going to have to repeat a ton of work that I did between December and March.
Is there any easy way to validate an XML backup before restoring it to pfsense?
-
After getting the December config up and running I attempted to restore only some sections of the March config, such as the Firewall Rules, and the browser displayed this text instead of the GUI:
Fatal error: Uncaught Exception: XML error: SSHDATA at line 3634 cannot occur more than once in /etc/inc/xmlparse.inc:89 Stack trace: #0 [internal function]: startElement(Resource id #6, 'SSHDATA', Array) #1 /etc/inc/xmlparse.inc(188): xml_parse(Resource id #6, 'WGUXTMry6ps4TWr...', false) #2 /etc/inc/xmlparse.inc(149): parse_xml_config_raw('/conf/config.xm...', Array, 'false') #3 /etc/inc/config.lib.inc(134): parse_xml_config('/conf/config.xm...', Array) #4 /etc/inc/config.inc(144): parse_config() #5 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...') #6 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...') #7 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...') #8 /etc/inc/config.gui.inc(39): require_once('/etc/inc/notice...') #9 /etc/inc/auth.inc(33): require_once('/etc/inc/config...') #10 /etc/inc/authgui.inc(27): include_once('/etc/inc/auth.i...') #11 /usr/local/www/guiconfig.inc(60): require_once('/etc/inc/authgu...') #12 /usr/local/www/index.php(46): require_once('/usr/local/www/...') #13 {m in /etc/inc/xmlparse.inc on line 89 PHP ERROR: Type: 1, File: /etc/inc/xmlparse.inc, Line: 89, Message: Uncaught Exception: XML error: SSHDATA at line 3634 cannot occur more than once in /etc/inc/xmlparse.inc:89 Stack trace: #0 [internal function]: startElement(Resource id #6, 'SSHDATA', Array) #1 /etc/inc/xmlparse.inc(188): xml_parse(Resource id #6, 'WGUXTMry6ps4TWr...', false) #2 /etc/inc/xmlparse.inc(149): parse_xml_config_raw('/conf/config.xm...', Array, 'false') #3 /etc/inc/config.lib.inc(134): parse_xml_config('/conf/config.xm...', Array) #4 /etc/inc/config.inc(144): parse_config() #5 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...') #6 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...') #7 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...') #8 /etc/inc/config.gui.inc(39): require_once('/etc/inc/notice...') #9 /etc/inc/auth.inc(33): require_once('/etc/inc/config...') #10 /etc/inc/authgui.inc(27): include_once('/etc/inc/auth.i...') #11 /usr/local/www/guiconfig.inc(60): require_once('/etc/inc/authgu...') #12 /usr/local/www/index.php(46): require_once('/usr/local/www/...') #13 {mThis one wasn't fatal, and the box is still running, but I did get this same notification in the pfsense Notices area until I cleared it.
-
That's a known issue: https://redmine.pfsense.org/issues/13132
You should be able to removed the spurious tags manually before restoring.
Steve
-
Thanks! At least I know I'm not crazy.
I can edit the XML file, but how do I know which <sshdata> section is the correct one to keep and which to delete?
One starts with:
<sshdata> <sshkeyfile> <filename>ssh_host_rsa_key</filename> <xmldata>ldfHjqxIF. . . .and also contains:
<filename>ssh_host_rsa_key.pub</filename>. . . . <filename>ssh_host_ed25519_key</filename>. . . . <filename>ssh_host_ed25519_key.pub</filename> . . . .and the other with:
<sshdata> <sshkeyfile> <filename>ssh_host_rsa_key</filename> <xmldata>ldfHrqw6. . . .and also contains:
<filename>ssh_host_rsa_key.pub</filename>. . . . <filename>ssh_host_ed25519_key</filename>. . . . <filename>ssh_host_ed25519_key.pub</filename> . . . .So both sections contain the same key/pub pairs, but with different data.
How can I tell which set is currently in use?
Thanks!
-
Also, if you know, does the bug trigger if a second <sshdata> section exists in the XML file, even if pfsense has been commanded to import only a small section of the XML file, e.g. the Firewall Rules, that doesn't include the <sshdata> tag?
My results from last night suggest the answer is yes, but I thought I'd verify!
-
Yes, it still tries to parse the full file and fails because have two sshdata sections is invalid.
You could probably decode the data using base64 and compare it to what you expect. I would probably just remove both sections so the rest of the config is valid. It will generate a new ssh key.
Steve
-
. . . or I could compare it to the December backup that imports successfully.
I'll try importing the edited version over the weekend.
-
How prevelant is this problem? I see a resolution is slated for 2.7.0 but how can the rest of us know if our devices on 2.6.0 will have the same problem when we go to restore our backups? Is there a specific package or something that causes this or a way to avoid it?
-
You can search the config for sshdata tags.
