Limiters break ACME LetsEncrypt renewal/account key registration
-
I recently had an issue renewing LetsEncrypt certs via ACME package, and soon after discovered that account key registration was broken too.
I found that when I disabled my firewall floating 'match' rules (WAN> OUT direction), things worked again.
I'm using FQ_Codel with tail-drop AQM.
Is there something I could do to leave the rule in place and have the ACME package auto-renew the cert on schedule?Oddly, I've had these rules in place for several months (maybe years), and prior renewals with ACME were a success.
-
My limiter/firewall rule setup is based on this.
-
There isn't a general issue with that or it would be failing for me as well, and it's working fine here. It's possible your floating rules on WAN are matching more than they should.
Check your setup against this doc instead of that thread:
https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html
A much more common cause of failure is that in certain cases ACME will fail to reach the LE servers over IPv6, but IPv4 works fine. Usually it's a local IPv6 configuration problem as well, though the easiest general fix is to set the option under advanced options to make the firewall prefer IPv4.
-
This post is deleted! -
@jimp Thanks for replying.
I do have several floating rules (from pfBlockerNG-devel, and a couple manual rules for ICMP workaround from that giant thread), however, if I disable that one rule, ACME renewals start working again.I don't use IPv6 and have prefer IPv4 checked in advanced options.
I followed the linked doc and the main difference from what I had is: Action match vs pass, and Source ANY vs WAN address, and Gateway: Default vs "Must be set to the gateway for this WAN interface".
I made changes to mimic the document and ACME renewal/key registration still fails unless I disable this rule.
Here is my thread on Let's Encrypt forum. Someone mentioned the curl POST was failing.
I have the full log posted there. -
These rules :
right ?
Btw : as I have two WAN type interfaces,
one called WAN_DHCP which is a IPv4 only connection to my upstream ISP router
the other called HE_TUN_TUNNELV6, this is my IPv6 interface.
So, instead of 3 rules I have 6, one for each "WAN" set.The ICMP ('ping') are pas rules. All the others, the last are (must be !) 'match' rules.
The "Source" is always "any".
When the :
then the Gateway is set to Default.
This changes when the direction is set to "out" !! The gateway has to be set to the correct WAN type interface.
That's why you see "WAN_DHCP" and "HE_TUN_TUNNELV6" in my image / gateway column.
As already said, these 6 rules are present on my pfSense for many years.
As is acme.sh : I have my certs renewed every 30 days.
I didn't change any (floating) rules for ...... a very long time.
It was working fine using 2.4.5 and before - and 2.50 and 2.5.2 CE today.Btw : there was an issue / pitfall shown lately on the forum.
It's easy to trigger :
Install pfBlockerNG-devel first. This, by itself, is completely harmless.
Activate a feed that blocks IP and/or domain names used by what is used to renew "letsencrypt" certificates - or who ever acme.sh is using to renew certs (this is new to me - dono of the pfSense acme.sh also contains this new choice of CA).
Apparently, the issue was that people activate IP and DNSBL feeds without checking them.
Or, at least, during the first several days : checking if the new feeds 'triggers' - and who, like what device, triggers it, etc.
Feeds are free, and this doesn't mean they are perfect - or not just plain broken.
" an admin always has to double check - even the things he doesn't know about - the latter should be triple checked. " -
@gertjan Yes, those rules.
I thought that changing the Protocol from ANY to TCP/UDP for the 'out' rule is going to do the trick, but no, it still fails.
Doesn't your setup contradict the document jimp linked, namely the "match" vs "pass" in floating rule for WAN out?
-
@bartkowski said in Limiters break ACME LetsEncrypt renewal/account key registration:
Doesn't your setup contradict ....
Yep.
I actually just discovered this new ? pfSEnse documentation page : https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html
Great ! Finally some good info from a source that doesn't need the 'fact check mode' to be set at 100 %. Thanks, jimp.I was pretty sure - reading the thread where I saw (to) many step by step how-toos - that it should be "match" and not "pass". Not that I understand the real difference between the two of them. I "thought" that "match" was the one to use as it is used to force traffic though a given gateway - using the supplied limiters, the up and own codel queues. This is my 'in brain' explanation ;)
I changed my 2 (2 + 2 actually) to "pass" instead of "match" and did a test on my favorite http://www.dslreports.com/speedtest test. The result was the same or slightly better. -
@gertjan A match rule will match and queue traffic without blocking or passing it, meaning a pass rule would allow traffic. Ergo, just be careful a pass rule isn't allowing traffic you don't want to allow.
Match is also "last match wins" by default as noted on that page, in the Quick section.
-
I switched the In / Out pipe queue order around in my WAN>out rule, and I was able to get ACME to register account key, but of course the result of Download was now matching my upload speed (I have 300/10 from Comcast) set by the WANOut_Q limiter queue (10500 Kbit/s).
-
Side note: I managed to crash my pfSense while changing some parameters on the Limiter, while also switching back to Kbit/s from Mbit/s.
-
In preparation for 22.01 update, I discovered that my Auto Config Backups were not being created. To my surprise, disabling my shaper rule allowed the backup to complete. What could this be?!
-
@bartkowski said in Limiters break ACME LetsEncrypt renewal/account key registration:
What could this be?!
That's what I was asking myself the entire afternoon.
Updated @home (virtual machine) to pfSense CE 2.6.0 : RAS.
@work : unbound ..... to use simple words : refuses to resolve ....
Switching to a pfSense minmal default config : all was fine. So, nice, its something in my setup.
[ ..... 2 hours later .... ]It looks like my limiters and/or floating firewall rules that uses these limiters are the issue.
I ditched them (floating rules first, then my limiters). pfSense was now fine.
I re reconstructed the limiters as per https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html
Added a firewall (just one ?) as per instruction son the same page.
I wasn't looking at the console but pfSense was rebooting - that's the first time for me in years.
I investigating. -
@gertjan said in Limiters break ACME LetsEncrypt renewal/account key registration:
I wasn't looking at the console but pfSense was rebooting
Did you crash? That's what happened to me if you read an earlier post, just unexpectedly.
-
Hey @gertjan, have you figured out anything?
I'm still scratching my head - I have two services that fail to work: ACME and Auto Backup. -
@bartkowski
Noop.
Not using limiters right now.
"They" know the issue exists, something good will come out ... soon ...
