CA is lost after update

dszp

The first time, I just copied <ca>and</ca> and everything in between from the old file over top of the same tag in the new file (which had the manually-imported cert already there when I downloaded it). The only thing I'm seeing that's different is the tags are in a slightly different order, and the <serial>4</serial> section doesn't appear to be there in the newly downloaded version but is there in the old version, inside the <ca>section (I left it in).

I just tried again, and I updated the <caref>to match the CA's <refid>everywhere in the file before uploading this time, rather than re-saving the OpenVPN config, and this time it does seem to have worked, the CA is there and the certs say they are from the proper CA-name, whereas before they were all showing "external" even after the CA was imported.

OK wait! When I FIRST logged in it was running through the Package Reinstall, which I let complete, and then I checked the Cert Manager. The CA was there and matched up to the certs! Then I went back there a couple of minutes later, making no changes (I visited OpenVPN first and it showed a no-CA error), and the CA was gone again! So it's originally importing fine and then apparently during some of the automated after-install processing it's getting deleted:

		 1/18/11 10:53:50	 : Installed Open-VM-Tools package.	Current
		 1/18/11 10:53:48	 : made unknown change	 	 	 
		 1/18/11 10:53:47	 : Removed Open-VM-Tools package.	 	 	 
		 1/18/11 10:53:46	 : made unknown change	 	 	 
		 1/18/11 10:53:01	 admin: /pkg_mgr_install.php made unknown change	 	 	 
		 1/18/11 10:52:57	 admin: Removed Open-VM-Tools package.	 	 	 
		 1/18/11 10:52:56	 admin: /pkg_mgr_install.php made unknown change	 	 	 
		 1/18/11 10:52:51	 : Installed OpenVPN Client Export Utility package.	 	 	 
		 1/18/11 10:52:49	 admin: Installed OpenVPN Client Export Utility package.	 	 	 
		 1/18/11 10:52:37	 : made unknown change	 	 	 
		 1/18/11 10:52:33	 admin: /pkg_mgr_install.php made unknown change	 	 	 
		 1/18/11 10:52:32	 : Removed OpenVPN Client Export Utility package.	 	 	 
		 1/18/11 10:52:28	 admin: Removed OpenVPN Client Export Utility package.	 	 	 
		 1/18/11 10:52:27	 admin: /pkg_mgr_install.php made unknown change	 	 	 
		 1/18/11 10:52:22	 admin: Creating restore point before package installation.
		 1/18/11 10:49:33	 admin: /diag_backup.php made unknown change

Those are the changes saved in Config History from the point I hit Restore to restore the config file to current, with me having made no manual changes. Of those, this is where the <ca>section gets deleted, between these two based on using Diff (the 4th and 5th config changes from the top):

		 1/18/11 10:53:46	 : made unknown change	 	 	 
		 1/18/11 10:53:01	 admin: /pkg_mgr_install.php made unknown change

Quite odd. If you want to log in, let me know and I'll create a username for you. Keep in mind the two installed packages were first installed, then apparently the upgrade and restore processes are both uninstalling and installing, or at least installing over top of, the old packages, and it appears that somehow in this process a CA gets wiped out.</ca></refid></caref></ca>

jimp

What packages do you have installed? (OR should have installed, I should say.) If there are issues installing/reinstalling the packages I don't really want to trust what is listed in the config vs what you know should be there.

dszp

Just the Open VM Tools and the OpenVPN Client Export. That's all package manager shows and all I've ever installed on this box I think (I had to rebuild it and restore config at one point a couple of months ago, haven't installed other packages since then). Exact same two packages installed on the "west" box as well, which is running at a totally different location, still a VM (on ESXi 3.5 vs. ESXi 4 for "pf" box), different IPs, both have never really touched the other. The "west" box may have had other packages on it at some point but I don't think so, I think I rebuilt it a few times when testing some CARP failover (which is not currently configured) in the last couple months as well, and haven't used any other packages than those two since.

dszp

I manually uninstalled the OpenVPN Client Export Utility from the Packages screen. Then I removed the section about it from the <packages>area in the config file that I'd restored earlier, but otherwise left it the same. I restored it again. This time, the Cert Manager shows the cert and it's still there, 10 or 20 minutes later or more. So the issue definitely was somewhere in the reinstall of the OpenVPN Client Export Utility package after the restore/upgrade. I'm going to reinstall the package manually now and see how it goes.</packages>

jimp

Hmm, and nothing in the open-vm-tools package would touch the CAs.

I'll keep digging at the OpenVPN client export package and see if I can see any scenario where it might do something unusual.

dszp

OK, another try:

I manually installed the OpenVPN Client Export Utility package again (this is after the restore earlier after manually removing it and then restoring config file without it included, which worked), and it installed and worked, CA still there. Then I downloaded a new backup file including the OpenVPN Client Export Utility package, and immediately restored it without making any changes. Now, the CA is gone, AND the package failed to reinstall and is not listed in the Installed Packages any longer, even though it was installed before I restored and is listed in the config file I restored. Here's the config history list:

		 1/18/11 12:11:01	 : Installed Open-VM-Tools package.	Current
		 1/18/11 12:06:46	 : made unknown change	 	 	 
		 1/18/11 12:06:43	 : Removed Open-VM-Tools package.	 	 	 
		 1/18/11 12:06:42	 : made unknown change	 	 	 
		 1/18/11 12:01:47	 : made unknown change	 	 	 
		 1/18/11 11:57:11	 admin: Installed OpenVPN Client Export Utility package.	 	 	 
		 1/18/11 11:57:02	 admin: /pkg_mgr_install.php made unknown change	 	 	 
		 1/18/11 11:57:01	 admin: Creating restore point before package installation.

First you can see where I installed the OpenVPN Client Export Utility package manually, and then when I restored it at 12:01 that's when the <ca>section disappears, between these two:

 1/18/11 12:01:47	 : made unknown change	 	 	 
		 1/18/11 11:57:11	 admin: Installed OpenVPN Client Export Utility package.

I downloaded a new backup, and can see that the <package>entry for OpenVPN Client Export Utility is gone, and there is no <ca>section, and like I said the Export Utility is now uninstalled, NOT listed in the Package Manager, when it was before the restore.

I do see in the config file that there are some leftover old settings from mod_security and ha_proxy inside <installedpackages>but those packages themselves have not been installed on this VM in the past, this config was restored to a fresh install since then.</installedpackages></ca></package></ca>

jimp

Very interesting.

And to make it even more interesting, nothing in the OpenVPN client export code makes a write to the config.

I still need to see if I can track down what is causing the ": made unknown change" entries.

dszp

Let me know if remote web or SSH access to this box would be helpful in tracking down the issue. Are there logs I'm not seeing you could look at?

jimp

Nah what you've posted so far may be enough.

I have just checked in a bunch of things that, while they may not fix it, may at least improve the situation in terms of logging. Hopefully the next snap will behave a bit better.

myka

That is strange …

did update

2.0-BETA5 (amd64)
from built on Wed Jan 12 23:13:34 EST 2011
to built on Tue Jan 18 13:16:28 EST 2011

CA is NOT lost

earlier tried

2.0 BETA5 AMD64
From: Wed Jan 12 23:13:34 EST 2011
To new version: Mon Jan 17 23:09:19 EST 2011

and CA was lost

jimp

I checked in some changes to the OpenVPN Client Export package this afternoon. It's possible the fix was there and not what is coming from the snapshot being built now.

dszp

I left both packages installed and upgraded one of the two boxes today to the latest snapshot a few hours ago, and it did NOT delete the CA this time! Both packages remain installed. Will upgrade the other as I have time; been a bit busy today. I did set up pfSense as the new firewall at our main office today though, not just my office, and I'm deploying pfSense on two NetGate boxes to customers in the next two days as well :-)

myka

me too successfully upgraded without loosing CA to

2.0-BETA5 (amd64)
built on Wed Jan 19 20:58:29 EST 2011

myka

CA is lost when updated

2.0-BETA5 (amd64)
from built on Wed Jan 19 20:58:29 EST 2011
to built on Thu Jan 20 01:23:56 EST 2011

jimp

Nothing changed that would have affected that between those builds…

Anything in the system log? What does the config history show for the last few config revisions?

Nachtfalke

Hi,

my CA is lost, too.

Updated from:
2.0-BETA5 (i386) built on Sun Jan 23 02:03:12 EST 2011
to:
2.0-BETA5 (i386) built on Sun Jan 23 10:30:03 EST 2011

Just have "OpenVPN Client Export utility" installed.

I read this earlier posts but didn't fully understand all.
If you need some files/configs please let me know step by step what should I have to do to help you.

jimp

If you go to Diagnostics > Backup/Restore on the config history tab, if you do a diff between the config from before the update, and the current config, is the only difference the missing CA?

jimp

I've gone over the package code again and reviewed any place in the system that modifies the CA and came up empty yet again.

I tried several times in a row on a VM and an ALIX to reproduce it and still have never lost a CA when it upgrades…

Nachtfalke

@jimp:

If you go to Diagnostics > Backup/Restore on the config history tab, if you do a diff between the config from before the update, and the current config, is the only difference the missing CA?

Sorry, I don't know how to use this feature :(

1/23/11 21:07:02 	(system): Installed OpenVPN Client Export Utility package. 	Current
		1/23/11 21:06:40 	(system): Intermediate config write during package install for OpenVPN Client Export Utility. 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 21:06:37 	(system): Removed OpenVPN Client Export Utility package. 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:57:41 	admin: /system_advanced_admin.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:56:38 	admin: /firewall_nat.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:56:36 	admin: /firewall_nat_edit.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:43:20 	admin: /system_usermanager_settings.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:38:15 	admin: Deleted CRL Test-Liste. 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:32:04 	admin: Deleted Certificate pfsense webGUI from CRL Test-Liste 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:32:02 	admin: Deleted Certificate Remote-User-VPN from CRL Test-Liste 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:31:39 	admin: Revoked cert Remote-User-VPN in CRL Test-Liste. 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:31:32 	admin: Revoked cert pfsense webGUI in CRL Test-Liste. 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:31:04 	admin: Saved CRL Test-Liste 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:22:28 	admin: /firewall_rules_edit.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:22:20 	admin: /firewall_rules_edit.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:21:57 	admin: /firewall_rules_edit.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:21:33 	admin: /firewall_rules_edit.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:19:30 	admin: /system_certmanager.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:19:16 	admin: /system_usermanager.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:18:44 	admin: /vpn_openvpn_server.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:13:16 	admin: /system_certmanager.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:09:28 	admin: /system_certmanager.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:08:50 	admin: /system_camanager.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:08:10 	admin: /system_camanager.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:08:07 	admin: /system_camanager.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:07:57 	admin: /system_certmanager.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:07:34 	admin: /system_usermanager.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:07:12 	admin: /vpn_openvpn_server.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:06:51 	admin: /firewall_rules.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
		1/23/11 20:06:44 	admin: /firewall_rules.php made unknown change 	Revert to this configuration 	Remove this backup 	Download this backup
	  	1/23/11 20:03:21 	admin: /vpn_openvpn_server.php made unknown change

jimp

Doesn't tell me much, really. To use the diff feature, select the "old" config in the first column of radio buttons, and the "new" config in the second column. Then press the diff button and it will show what changed between those two configuration files.

So in your case, click the radio selector (circle button) in the first colmn next to "1/23/11 20:57:41" and click the topmost radio selector in the second column, then press 'diff'.