CA is lost after update
-
For those of you still losing the CA, can you go into the config history as described earlier in this thread and do a diff between each revision and find the one that loses the CA again? I suspect the step is the same but I'm hoping after the changes I made that something is slightly different. (Be sure to edit out the CA's crt/key fields before posting the diff here)
-
OK I updated from a snapshot earlier yesterday to the latest snapshot (also dated yesterday, Jan 29th). I watched the package reinstall progress. After it says All Packages Reinstalled, I viewed the Cert Manager, and the CA is still there. However, every page in the GUI now has the "Packages are currently being reinstalled in the background. Do not make changes in the GUI until this is complete." message at the top upon page load, even after I hit Close and then reload. It doesn't matter which page I go to, and everything looks normal otherwise except if I go to the Packages page, then I see the same message but instead of packages I see "Please wait while packages are reinstalled in the background."
This is continuing even 15 minutes after All Packages Reinstalled showed up in the reinstall status page after the reboot.
The config history starting at the bottom with the entry made after clicking Upgrade:
1/30/11 16:06:16 admin: Installed OpenVPN Client Export Utility package. Current 1/30/11 16:05:54 admin: Intermediate config write during package install for OpenVPN Client Export Utility. 1/30/11 16:05:50 admin: Removed OpenVPN Client Export Utility package. 1/30/11 16:05:49 admin: Intermediate config write during package removal for OpenVPN Client Export Utility. 1/30/11 16:05:42 admin: Installed Open-VM-Tools package. 1/30/11 16:00:56 admin: Intermediate config write during package install for Open-VM-Tools. 1/30/11 16:00:52 admin: Removed Open-VM-Tools package. 1/30/11 16:00:51 admin: Intermediate config write during package removal for Open-VM-Tools. 1/30/11 15:58:43 admin: Creating restore point before package installation.There aren't really many changes if I Diff from top to bottom of the entries above in one fell swoop:
Configuration diff from 1/30/11 15:58:43 to 1/30/11 16:06:16 --- /conf/backup/config-1296421123.xml 2011-01-30 16:00:51.000000000 -0500 +++ /conf/config.xml 2011-01-30 16:06:16.000000000 -0500 @@ -1743,8 +1743,8 @@ <sequence>system_information-container:col1:show,captive_portal_status-container:col1:close,carp_status-container:col1:close,cpu_graphs-container:col1:close,gateways-container:col1:show,gmirror_status-container:col1:close,installed_packages-container:col1:show,interface_statistics-container:col1:show,interfaces-container:col2:show,ipsec-container:col2:close,load_balancer_status-container:col2:close,log-container:col2:show,picture-container:col2:close,rss-container:col2:close,services_status-container:col2:show,traffic_graphs-container:col2:show,openvpn-container:col2:none</sequence> <revision>- <time>1296421123</time> - + <time>1296421576</time> + <username>admin</username></revision> <openvpn>@@ -1993,6 +1993,8 @@ <config_file>http://www.pfsense.com/packages/config/openvpn-client-export/openvpn-client-export.xml</config_file> <configurationfile>openvpn-client-export.xml</configurationfile> + <menu> + <service><tab><name>Client Export</name> <tabgroup>OpenVPN</tabgroup> So now it's justā¦stuck. Though it's working :-) The OpenVPN Client Export tab shows up under VPN->OpenVPN, and I assume VMware Tools is installed as well (no reason to doubt but I haven't checked. It installs before OpenVPn Client Export though and generally doesn't disappear even when OpenVPN Export does). Both packages were installed before the upgrade. After trouble yesterday, I edited the backup file I had, removed the <installedpackages> section entirely, and restored to that version (after manually uninstalling all packages from the Package Manager page). Then I installed both the VMware Tools and OpenVPN Client Export packages manually after it rebooted, and the CA was intact. Removing <installedpackages> got rid of old config options from haproxy and such that were left over from a while ago, and those have not returned. I did nothing since that restore/package reinstall except to run the auto-upgrade procedure with the results above.</installedpackages></installedpackages></tab></service></menu></openvpn> -
That status message indicates that the packages didn't fully reinstall from the console. Did you also try to reinstall them from the GUI immediately after update? You may have connected to the GUI before the console was completely finished with the boot cycle, so it was trying to reinstall the packages in two places at onceā¦ (just a theory)
Doesn't look like you lost your CA though, so it probably isn't related to this bug.
-
Nope, I did reconnect to the GUI as soon as possible, but I haven't done anything. It hasn't changed yet so I just hit the Reinstall button and we'll see what happens. I would assume the GUI would just show progress of the reinstall that's part of the upgrade?
I wouldn't be too sure on the CA not being goneā¦in the past it's only disappeared after the "Packages are being reinstalledā¦" thing disappears after the packages reinstall (based on loading the GUI ahead of time and seeing the CA and then watching it disappear later). I assume (as I don't have any better info) that since this never went away, it never got to the "let's kill the CA" stage :-) Or maybe it is fixed; impossible to tell for sure.
After I clicked Reinstall packages from the Backup area, it appeared to finish again ("All Packages Reinstalled") but the "Packages are being installed..." is still there on every page, no change. Of course the CA is there too :-) Each page takes a long time to load as well now.
-
Yep I was right :-( Rebooting fixed the reinstall problem, andā¦the CA disappeared. The console was stuck at "Loading VMware driver" before I chose reboot from the GUI menu; rebooted and it's at the normal console but the CA is gone. Here's the configuration diff, it's the very newest diff in the config history, between the newest and the next-to-newest, and you can see where the CA was removed (blanked for privacy, but it was between the <crt>and <prv>tags):
1/30/11 18:34:05 (system): Intermediate config write during package removal for Open-VM-Tools. Current 1/30/11 18:11:47 admin: Intermediate config write during package install for OpenVPN Client Export Utility. 1/30/11 18:11:42 admin: Removed OpenVPN Client Export Utility package. 1/30/11 18:11:41 admin: Intermediate config write during package removal for OpenVPN Client Export Utility. 1/30/11 18:11:35 admin: Installed Open-VM-Tools package. 1/30/11 18:07:19 admin: Intermediate config write during package install for Open-VM-Tools. 1/30/11 18:07:14 admin: Removed Open-VM-Tools package. 1/30/11 18:07:13 admin: Intermediate config write during package removal for Open-VM-Tools. 1/30/11 18:05:43 admin: Creating restore point before package installation. 1/30/11 16:06:16 admin: Installed OpenVPN Client Export Utility package.Configuration diff from 1/30/11 18:11:47 to 1/30/11 18:34:05 --- /conf/backup/config-1296429107.xml 2011-01-30 18:11:59.000000000 -0500 +++ /conf/config.xml 2011-01-30 18:34:06.000000000 -0500 @@ -1304,7 +1304,8 @@ - <shaper>+ <shaper>+</shaper> <ipsec><client><enable>@@ -1743,9 +1744,9 @@ <sequence>system_information-container:col1:show,captive_portal_status-container:col1:close,carp_status-container:col1:close,cpu_graphs-container:col1:close,gateways-container:col1:show,gmirror_status-container:col1:close,installed_packages-container:col1:show,interface_statistics-container:col1:show,interfaces-container:col2:show,ipsec-container:col2:close,load_balancer_status-container:col2:close,log-container:col2:show,picture-container:col2:close,rss-container:col2:close,services_status-container:col2:show,traffic_graphs-container:col2:show,openvpn-container:col2:none</sequence> <revision>- <time>1296429107</time> - - <username>admin</username> + <time>1296430445</time> + + <username>(system)</username></revision> <openvpn><openvpn-server>@@ -1805,7 +1806,8 @@ <l7shaper><container></container></l7shaper> - <dnshaper>+ <dnshaper>+</dnshaper> <gateways><gateway_item><interface>wan</interface> @@ -1955,13 +1957,6 @@ <dhcrelay>- <ca>- <refid>4d42c37c64858</refid> - - - - <serial>3</serial> -</ca> <installedpackages><package><name>Open-VM-Tools</name> @@ -1993,7 +1988,12 @@ <config_file>http://www.pfsense.com/packages/config/openvpn-client-export/openvpn-client-export.xml</config_file> <configurationfile>openvpn-client-export.xml</configurationfile></package> - <menu> - <service>+ <tab>+ <name>Client Export</name> + <tabgroup>OpenVPN</tabgroup> + <url>/vpn_openvpn_export.php</url> +</tab> + +</service> </menu></installedpackages></dhcrelay></gateway_item></gateways></dnshaper></openvpn-server></openvpn></enable></client></ipsec></shaper> ```</prv></crt> -
Diff is in those line besides
2/1/11 10:38:06 (system): Removed OpenVPN Client Export Utility package.
1/25/11 18:56:11 admin: Creating restore point before package installation.Configuration diff from 1/25/11 18:56:11 to 2/1/11 10:38:06 --- /conf/backup/config-1295974571.xml 2011-02-01 10:38:06.000000000 +0200 +++ /conf/backup/config-1296549486.xml 2011-02-01 10:38:08.000000000 +0200 @@ -794,9 +794,10 @@ <sequence>system_information-container:col1:show,captive_portal_status-container:col1:close,carp_status-container:col1:close,cpu_graphs-container:col1:close,gateways-container:col1:close,gmirror_status-container:col1:close,installed_packages-container:col1:close,interface_statistics-container:col1:close,interfaces-container:col2:show,ipsec-container:col2:close,load_balancer_status-container:col2:close,log-container:col2:close,picture-container:col2:close,rss-container:col2:close,services_status-container:col2:close,traffic_graphs-container:col2:close</sequence> <revision>- <time>1295974571</time> - - <username>admin</username> + <time>1296549486</time> + <description>+]]></description> + <username>(system)</username></revision> <openvpn><openvpn-server>@@ -856,28 +857,8 @@ <service><tab>- <tab>- <name>Client Export</name> - <tabgroup>OpenVPN</tabgroup> - <url>/vpn_openvpn_export.php</url> -</tab> <menu> <package>- <name>OpenVPN Client Export Utility</name> - - <category>Security</category> - <depends_on_package_base_url>http://files.pfsense.org/packages/amd64/8/All/</depends_on_package_base_url> - <depends_on_package>p7zip-9.13.tbz</depends_on_package> - <depends_on_package>zip-3.0.tbz</depends_on_package> - <build_port_path>/usr/ports/archivers/p7zip</build_port_path> - <build_port_path>/usr/ports/archivers/zip</build_port_path> - <version>0.5</version> - <status>BETA</status> - <required_version>2.0</required_version> - <config_file>http://www.pfsense.com/packages/config/openvpn-client-export/openvpn-client-export.xml</config_file> - <configurationfile>openvpn-client-export.xml</configurationfile> -</package> - <package><name>Open-VM-Tools</name> <website>http://open-vm-tools.sourceforge.net/</website> @@ -894,13 +875,6 @@</package> <dhcrelay>- <ca>- <refid>4d2efa305ac2a</refid> - - <crt>(deleted)</crt> - <prv>(deleted)</prv> - <serial>2</serial> -</ca> <ppps><gateways></gateways></ppps></dhcrelay> </menu></tab></service></openvpn-server></openvpn> -
That's slightly different than it was before, it was lost at one of the "intermediate" steps before.
Was that an upgrade to the snapshot from late last night?
2.0-BETA5 (i386)
built on Mon Jan 31 23:05:36 EST 2011?
-
It was
From Tue Jan 25 07:56:16 EST 2011
To new version: Mon Jan 31 19:36:10 EST 2011 -
OK, I was finally able to reproduce this with a CF image supplied by someone who had the problem (thanks!)
Using that I was able to track down (and fix) where the loss occurred during the boot process. Funny thing, it had nothing to do with the packages except that the package reinstall caused a config write which resulted in the data loss from the bug.
The next new snapshot should have the fix - I just restarted the builders.
-
Yay! That's great to hear! I was about to say "hey, would you like a copy of my VM?" but it sounds like you got essentially the same thing. It didn't seem like a Packages bug to me directly either because the second box (the one I sent my config from with the other a while back) has been upgrading just fine the past few times, not losing the same two packages or the CAs, where it was at some point in the past, so the packages thing just seemed to be the most visible, sometimes-reproducible symptom. Looking forward to this one being gone for sure!
Would the data loss affect any other areas of the config file as well or just the CA? Should I restore to an older backup version?
-
I only noticed the loss with CAs.
It was a faulty function in the certificate handling that abused references. There could be other functions that are broken in the same way, but there isn't an easy way to track them down.
Just need to wait and see if anyone else reports similar issues.
-
Hi,
I updated my two boxes without problems the last few times but really nice to hear, that could reproduce the bug and hopefully fixed it :-) Great work!
-
updated to built on Wed Feb 2 00:06:58 EST 2011
CA is NOT lost
thanks jimp
-
Both boxes that had issues in the past upgraded with no issues, CA still there, packages still there. Thanks Jim, very awesome to have this fixed finally!
-
Good to hear. :-)
-
Multiple upgrades on multiple pfSense installs since last post, and still rock solid CAs and packages! I'm done posting in this thread unless I experience another problem with the same issueā¦definitely considering it closed, can't say I'll remember the troubleshooting process too fondly :-)
-
ā¦and I won't remember the debugging process too fondly!
:-)
-
Well, regardless of difficulty, thank you Jim for sticking with it and figuring it out! I know the fact that multiple people had the problem indicated that I wasn't crazy but it was a tough little bugger, and I and others I'm sure appreciate the fix very much!
