Fatal Error PHP Acme Certificates in Renew Column

jpvonhemel

Here is the crash report from pfSense. I did a reinstall to 22.04 and restored my backup config and the issue is continuing.

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:30:29 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/VDZvZksF/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBS

Crash report details:

PHP Errors:
[19-Feb-2023 05:33:58 America/Denver] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/acme/acme_certificates.php:261
Stack trace:
#0 {main}
thrown in /usr/local/www/acme/acme_certificates.php on line 261

No FreeBSD crash data found.

kapranos

I have the same proble as you, I can not delete the certificate, I have accessed by SSH and do not see it in the certificate folders, you were able to solve it? thanks

jpvonhemel

@kapranos Sorry to hear, but glad it wasn't just me. Unfortunately my post did not gain any traction with Netgate, so I figured it was a problem unique to me. After a few days of frustration with the unstable upgrade, I nuked the firewall install, installed a clean image and reconfigured the firewall.

I was able to import most of the settings from backup. For me, that was mostly DCHP reservations, nat and firewall rules, so it wasn't the end of the world. Going foward, I will be more careful with updates and backups. I believe there is a way to leverage the zfs file system to retain and boot from an old install, so I do need to look into that a bit more.

I honestly do not know how it happened, but chalked it up to some sort of corruption on update or a really odd bug.

kapranos

@jpvonhemel said in Fatal Error PHP Acme Certificates in Renew Column:

@kapranos Sorry to hear, but glad it wasn't just me. Unfortunately my post did not gain any traction with Netgate, so I figured it was a problem unique to me. After a few days of frustration with the unstable upgrade, I nuked the firewall install, installed a clean image and reconfigured the firewall.

I was able to import most of the settings from backup. For me, that was mostly DCHP reservations, nat and firewall rules, so it wasn't the end of the world. Going foward, I will be more careful with updates and backups. I believe there is a way to leverage the zfs file system to retain and boot from an old install, so I do need to look into that a bit more.

I honestly do not know how it happened, but chalked it up to some sort of corruption on update or a really odd bug.

Thanks, for the moment I will live with the error, it doesn't bother me so much, I hope it will be solved with an update.

thanks

jimp

Can you open an issue for this under https://redmine.pfsense.org/projects/pfsense-packages?

I'm not seeing that on anything here, so it must be something unique about the settings for that entry in your config.xml. If you could, please include the ACME sections of your config.xml when creating the redmine issue. You can redact whatever information you want just make sure you put something in place (e.g. place mypassword with xxx or similar so we can distinguish entries with data from empty entries!)

Something must be wrong in the domain list of that entry but it's not clear how it would be empty and yet still working.

kapranos

@jimp said in Fatal Error PHP Acme Certificates in Renew Column:

config.xml

Thanks @jimp ,raised ticket https://redmine.pfsense.org/issues/14199

jimp

How did your configuration get into that state? I can't even trick it to make it save an incomplete entry like you have in the configuration there. The domain list is empty so it would never have created a cert or renewed one.

I suppose it could be an old entry from before validation was added, but that still couldn't have been doing anything useful.

It's probably best if you download a config backup, edit out this whole entry and then restore it:

			<certificates>
				<item>
					<a_domainlist></a_domainlist>
					<a_actionlist></a_actionlist>
					<keypaste></keypaste>
					<name>wildcard</name>
					<descr><![CDATA[Wildcard Cert]]></descr>
					<status>active</status>
					<acmeaccount>HomeCert</acmeaccount>
					<keylength>2048</keylength>
					<ocspstaple></ocspstaple>
					<preferredchain></preferredchain>
					<dnssleep></dnssleep>
					<renewafter></renewafter>
				</item>
			</certificates>

kapranos

I don't quite remember, I was trying to create a certificate in ACME for the Pfsense webui and at some point it got saved wrong and stayed in that state.

At the moment I don't want to restore a backup because it doesn't bother me too much, I can live with this if it doesn't have a solution.

thanks

jimp

The code could be fixed to account for that sort of error, and likely will be, but whenever we do the next ACME update which could be a while yet. But since it's apparently difficult (and should be impossible) to get into that state with normal usage it isn't a priority.

If you are comfortable with vi and working in the shell you could ssh into the firewall and use viconfig to edit that problem entry out live. The backup method is safer, but more disruptive.

jpvonhemel

@jimp I uploaded the acme section of the xml to redmine a few days ago. I don't think this is from @kapranos . I do not remember seeing his to compare against.

ACME was setup in conjunction with SSL ofloading using HAProxy/LetsEncrypt following a Lawrence Systems youtube guide. HAProxy was working and the system was stable prior to the Feb update.

When I posted the original issue, I noticed no one else had the same problem, so figured I was unique and since pfsense would hang, decided to just nuke and pave, then reconfigure from the other parts of the XML. I went back through Tom's video on HAProxy and things are fine. I pulled and posted a today's Acme section redacted and I anticipate this is now normal.

I am not sure if there is any hope for @kapranos install/issue can be salvaged.

<acme>
			<certificates>
				<item>
					<lastrenewal>1677002752</lastrenewal>
					<a_domainlist>
						<item>
							<status>enable</status>
							<name>*.REDACTED</name>
							<method>dns_namecom</method>
							<dns_namecomnamecom_username>REDACTED</dns_namecomnamecom_username>
							<dns_namecomnamecom_token>REDACTED</dns_namecomnamecom_token>
							<_index></_index>
						</item>
					</a_domainlist>
					<a_actionlist>
						<item>
							<status>enable</status>
							<command>/usr/local/etc/rc.d/haproxy.sh restart</command>
							<method>shellcommand</method>
							<_index></_index>
						</item>
					</a_actionlist>
					<keypaste></keypaste>
					<name>wildcardredacted</name>
					<descr><![CDATA[Wildcard Cert REDACTED.com]]></descr>
					<status>active</status>
					<acmeaccount>HomeCert</acmeaccount>
					<keylength>2048</keylength>
					<ocspstaple></ocspstaple>
					<preferredchain></preferredchain>
					<dnssleep></dnssleep>
					<renewafter></renewafter>
				</item>
			</certificates>
			<accountkeys>
				<item>
					<accountkey>REDACTED</accountkey>
					<name>HomeCert</name>
					<descr><![CDATA[REDACTED.com Lets Encrypt Account]]></descr>
					<email><![CDATA[REDACTED]]></email>
					<acmeserver>letsencrypt-production-2</acmeserver>
					<renewafter></renewafter>
				</item>
			</accountkeys>
			<enable>yes</enable>
			<writecerts></writecerts>
		</acme>

jimp

If the error was the same, then it was probably similar enough. The line at which it failed was trying to process items in the domain list, which was empty. I still can't figure out how it might have ended up in that state, but it's good to know when you started over again it worked that time.

kapranos

Attached PHP error log PHP_errors (1).log.zip

kapranos

Redmine report tracker https://redmine.pfsense.org/issues/14199

kapranos

Problem continues after upgrading to Pfsense 23.05 and ACME 0.7.3_2, I haven't had any response in the Redmine thread either :(

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 06:35:01 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/amd64/LkEyii3W/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/sources/FreeBS

Crash report details:

PHP Errors:
[03-Jun-2023 14:28:36 Europe/Madrid] PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/acme/acme_certificates.php:261
Stack trace:
#0 {main}
  thrown in /usr/local/www/acme/acme_certificates.php on line 261



No FreeBSD crash data found.