ACME filling up disk
-
For (nearly) the second time now the ACME package has filled up the disk on my pfSense. A recap of some history:
- On August 8, the ACME package updated the certificate
- On October 7, I came home in the afternoon to find the internet not working. Eventually determined the disk was full and deleted some zfs filesystems associated with old versions. This got things working but the webgui was totally borked.
- On October 11, I finally did a factory reset, and restored a configuration backup from August 8, after the certificate update
Things were working fine until a couple of days ago when I started getting HSTS errors when trying to connect. My browser showed I was being served the August 8 certificate, which expired on November 6. The certificate seemed legit (as in, one that was issued to my pfSense) but for whatever reason a newer cert (that was present in the GUI) wasn't being used. I couldn't find any evidence of this August 8th cert in the GUI or configs, but that's not actually the point of the story.
While trying to figure this out, I noticed that /cf was using a large amount of space. This was one of the things that eventually led to the previous install crashing:
Poking around I found a config file every second in the backup directory:
Looking at the timestamps on these config files you can see about when it started:
The first config in that 1731701* bunch had this change, me forcing a certificate issuance in the hope that would fix something:
After that, there is an "identical" config every second looking like:
time is /pfsense/revision/time, and lastrenewal is /pfsense/installedpackages/acme/certificates/item/lastrenewal. The comment on every one of thos configs is the same as above, "(system): Services: Acme: Storing signed certificate". I assume ACME is somehow in a loop where modifying the config is causing it to update the config.
I ended up uninstalling ACME which stopped the problem, but once I re-installed ACME and forced another Issue/Renew, it started again. System Logs shows this loop:
This is with pfSense-pkg-acme-0.8_1 on a Netgate 1100 running 24.03-RELEASE