CA is lost after update
-
I pulled a backup file from the server and found the <ca>entries and pasted the <crt>and <prv>contents into the Import Existing CA fields in the Cert Manager, and it says "The following input errors were detected: This certificate does not appear to be valid." and won't import the CA. I have two of them on this server, both fail the same way, whether or not I use the private key portion.</prv></crt></ca>
-
The fields in the backup are base64 encoded. You could paste them into a config directly, but not import them in that way.
I've had a couple other reports of CAs disappearing but I cannot reproduce it here.
Do you mind sending me your before-and-after config.xml files that have the CA and then are missing the CA? You can send them to me privately, jimp (at) pfsense [dot] org.
It may be something about a specific config that is causing the loss.
-
Config files sent, Jim.
I first tried to restore just the cert stuff from backup using the Backup/Restore area, but there's no Cert-only option there. That's when I tried to paste the cert from the config to import it. Would be nice if it was possible to import certs from the config file without using a tool to convert them to PEM format; if the Cert Manager figured out the base64 encoding automatically or something :-) Anyway, I'll take care of that manually for now and wait to hear on the config files if you find anything. The systems both have the OpenVPN Client Export Utility and the Open-VPN-Tools packages installed, and no others. Both config files lost their CA config during an upgrade to a newer snapshot, between 1/3/2011 and 1/10/1011 for one box and between config changes 20 seconds apart on 1/14/2011 for the other. I'm not sure which snapshots the upgrades were to…but I upgrade one box every couple of weeks (3 or 4 at the most) and the other I usually keep updated every day to every few days.
-
Most people will never have to care about the base64 encoded version….
It's easy to get it back though
Diagnostics > Command, PHP Execute box:
$foo = "base64encodedstring"; echo base64_decode($foo);I think I may have tracked down what might be causing it, but I'm not 100% sure. You're the second person to show that it happened right after the config upgrade from 7.5 to 7.6, and the only addition there was adding a cron job. The way I added the cron job there was a little different than the other code in the upgrade process so I switched out the code to something more standard for that area.
The only way to test it for sure would be to restore back to a config that was just before that process and let it upgrade again, to see if it retains your data.
-
If I have time I will test restoring to old config once a new snapshot is out.
Thanks for the easy base64 conversion, worked great! I wouldn't have thought to use the PHP command area :-)
Once I restored the CAs I had to edit the OpenVPN configuration and save it, and then the service started fine. I assume the unique id for the CA changed on import and OpenVPN needed to save that change.
Interestingly, of the two boxes, when I went to the Cert Manager on the "west" one (vs "pf" hostname) there was still one of two CAs there, and I didn't think I'd restored one by that time unless I lost track among the copying/pasting/base_64_decoding :-) The "pf" box only had one CA and it was definitely gone though, but now back after importing it from the config.
-
It looks like it only removed one CA, it just happened to be the CA you needed most on that one box.
Looking deeper I'm not certain that the upgrade code would have caused that at all, and I really don't see anywhere in the OpenVPN client export package that could have removed it either. One of them hadn't even updated to config version 7.6 yet.
Are you sure that the configs you sent are the exact point where the CAs disappeared?
the diff function on the config history is handy for tracking that stuff down. -
Yes, I used the Config History and Diff function there to narrow down the before and after (right next to each other) so the diff actually showed the <ca>lines being removed, and I downloaded the two config files that I compared in the Diffs and sent those. I did that on both boxes. You'd be welcome to login to the boxes and compare yourself, but I've made enough changes since my last post that the specific change point is gone in the Config History now. However, I do have a Veeam Backup virtual machine backup of the "pf" unit that I can restore if you want. I upgraded that one via command line today because upgrading a couple of days ago brought the webgui down and it never came back; upgrading via CLI brought it back after the upgrade. But the full VM gets automatically backed up every day or two.</ca>
-
As long as I have the before-and-after configs it should be enough. I'll keep poking at it and see what I can find.
-
OK here's something interesting…the CA didn't match the certs after I reimported it so OpenVPN wouldn't reconned on the "pf" box. So I downloaded a current config file, pasted JUST the <ca>and <certs>sections from the backup that had them still there, and did a full config file restore. When it rebooted after the restore, the Cert Manager shows NO CA's installed even though it's right there in the config.xml file I modified and restored!</certs></ca>
-
Sure you got the tags exactly right?
Yeah the CA import would give it a new certref id so everything that used it would have to be updated to point at the 'new' CA. (or you could edit the config and change the certref to match the previous one) but really if you got the tags right they should be there.
-
The first time, I just copied <ca>and</ca> and everything in between from the old file over top of the same tag in the new file (which had the manually-imported cert already there when I downloaded it). The only thing I'm seeing that's different is the tags are in a slightly different order, and the <serial>4</serial> section doesn't appear to be there in the newly downloaded version but is there in the old version, inside the <ca>section (I left it in).
I just tried again, and I updated the <caref>to match the CA's <refid>everywhere in the file before uploading this time, rather than re-saving the OpenVPN config, and this time it does seem to have worked, the CA is there and the certs say they are from the proper CA-name, whereas before they were all showing "external" even after the CA was imported.
OK wait! When I FIRST logged in it was running through the Package Reinstall, which I let complete, and then I checked the Cert Manager. The CA was there and matched up to the certs! Then I went back there a couple of minutes later, making no changes (I visited OpenVPN first and it showed a no-CA error), and the CA was gone again! So it's originally importing fine and then apparently during some of the automated after-install processing it's getting deleted:
1/18/11 10:53:50 : Installed Open-VM-Tools package. Current 1/18/11 10:53:48 : made unknown change 1/18/11 10:53:47 : Removed Open-VM-Tools package. 1/18/11 10:53:46 : made unknown change 1/18/11 10:53:01 admin: /pkg_mgr_install.php made unknown change 1/18/11 10:52:57 admin: Removed Open-VM-Tools package. 1/18/11 10:52:56 admin: /pkg_mgr_install.php made unknown change 1/18/11 10:52:51 : Installed OpenVPN Client Export Utility package. 1/18/11 10:52:49 admin: Installed OpenVPN Client Export Utility package. 1/18/11 10:52:37 : made unknown change 1/18/11 10:52:33 admin: /pkg_mgr_install.php made unknown change 1/18/11 10:52:32 : Removed OpenVPN Client Export Utility package. 1/18/11 10:52:28 admin: Removed OpenVPN Client Export Utility package. 1/18/11 10:52:27 admin: /pkg_mgr_install.php made unknown change 1/18/11 10:52:22 admin: Creating restore point before package installation. 1/18/11 10:49:33 admin: /diag_backup.php made unknown changeThose are the changes saved in Config History from the point I hit Restore to restore the config file to current, with me having made no manual changes. Of those, this is where the <ca>section gets deleted, between these two based on using Diff (the 4th and 5th config changes from the top):
1/18/11 10:53:46 : made unknown change 1/18/11 10:53:01 admin: /pkg_mgr_install.php made unknown changeQuite odd. If you want to log in, let me know and I'll create a username for you. Keep in mind the two installed packages were first installed, then apparently the upgrade and restore processes are both uninstalling and installing, or at least installing over top of, the old packages, and it appears that somehow in this process a CA gets wiped out.</ca></refid></caref></ca>
-
What packages do you have installed? (OR should have installed, I should say.) If there are issues installing/reinstalling the packages I don't really want to trust what is listed in the config vs what you know should be there.
-
Just the Open VM Tools and the OpenVPN Client Export. That's all package manager shows and all I've ever installed on this box I think (I had to rebuild it and restore config at one point a couple of months ago, haven't installed other packages since then). Exact same two packages installed on the "west" box as well, which is running at a totally different location, still a VM (on ESXi 3.5 vs. ESXi 4 for "pf" box), different IPs, both have never really touched the other. The "west" box may have had other packages on it at some point but I don't think so, I think I rebuilt it a few times when testing some CARP failover (which is not currently configured) in the last couple months as well, and haven't used any other packages than those two since.
-
I manually uninstalled the OpenVPN Client Export Utility from the Packages screen. Then I removed the section about it from the <packages>area in the config file that I'd restored earlier, but otherwise left it the same. I restored it again. This time, the Cert Manager shows the cert and it's still there, 10 or 20 minutes later or more. So the issue definitely was somewhere in the reinstall of the OpenVPN Client Export Utility package after the restore/upgrade. I'm going to reinstall the package manually now and see how it goes.</packages>
-
Hmm, and nothing in the open-vm-tools package would touch the CAs.
I'll keep digging at the OpenVPN client export package and see if I can see any scenario where it might do something unusual.
-
OK, another try:
I manually installed the OpenVPN Client Export Utility package again (this is after the restore earlier after manually removing it and then restoring config file without it included, which worked), and it installed and worked, CA still there. Then I downloaded a new backup file including the OpenVPN Client Export Utility package, and immediately restored it without making any changes. Now, the CA is gone, AND the package failed to reinstall and is not listed in the Installed Packages any longer, even though it was installed before I restored and is listed in the config file I restored. Here's the config history list:
1/18/11 12:11:01 : Installed Open-VM-Tools package. Current 1/18/11 12:06:46 : made unknown change 1/18/11 12:06:43 : Removed Open-VM-Tools package. 1/18/11 12:06:42 : made unknown change 1/18/11 12:01:47 : made unknown change 1/18/11 11:57:11 admin: Installed OpenVPN Client Export Utility package. 1/18/11 11:57:02 admin: /pkg_mgr_install.php made unknown change 1/18/11 11:57:01 admin: Creating restore point before package installation.First you can see where I installed the OpenVPN Client Export Utility package manually, and then when I restored it at 12:01 that's when the <ca>section disappears, between these two:
1/18/11 12:01:47 : made unknown change 1/18/11 11:57:11 admin: Installed OpenVPN Client Export Utility package.I downloaded a new backup, and can see that the <package>entry for OpenVPN Client Export Utility is gone, and there is no <ca>section, and like I said the Export Utility is now uninstalled, NOT listed in the Package Manager, when it was before the restore.
I do see in the config file that there are some leftover old settings from mod_security and ha_proxy inside <installedpackages>but those packages themselves have not been installed on this VM in the past, this config was restored to a fresh install since then.</installedpackages></ca></package></ca>
-
Very interesting.
And to make it even more interesting, nothing in the OpenVPN client export code makes a write to the config.
I still need to see if I can track down what is causing the ": made unknown change" entries.
-
Let me know if remote web or SSH access to this box would be helpful in tracking down the issue. Are there logs I'm not seeing you could look at?
-
Nah what you've posted so far may be enough.
I have just checked in a bunch of things that, while they may not fix it, may at least improve the situation in terms of logging. Hopefully the next snap will behave a bit better.
-
That is strange …
did update
2.0-BETA5 (amd64)
from built on Wed Jan 12 23:13:34 EST 2011
to built on Tue Jan 18 13:16:28 EST 2011CA is NOT lost
earlier tried
2.0 BETA5 AMD64
From: Wed Jan 12 23:13:34 EST 2011
To new version: Mon Jan 17 23:09:19 EST 2011and CA was lost
