Port Forwarding not honered for .well-known/acme-challenge

jgrabner

I am brand new with pfsense, so sorry if my explanation is not optimal.

I have a public IP from my ISP for my hobby webserver. I recently moved, so I have a new ISP. After speaking with tech support from this ISP, I got scared they did not know what they were doing, so I purchased a 4200 to firewall my LAN and webserver. I plugged it in last week and can't get Let's Encrypt working on my server (cert-manager in microk8s with nginx ingress).

Pfsense WAN is assigned my public IP address (150.220.136.18).
LAN side pfsense does DHCP for the PCs and my webserver. Port ranges 192.168.10.x

I port forward ports 80 and 443 to 30080 and 30443 on 192.168.10.12 (my webserver). Webserver is able to get HTTP (80) traffic from the internet, but
let's encrypt HTTP calls to <my DNS name>/.well-known/acme-challenge/<token>
get intercepted by pfsense and not port-forward.

It's a little confusing since the Microk8s ingress uses Nginx, but the following log is from Pfsense, so I'm guessing Pfsense is also doing Let's Encrypt and Nginx.

Here is the log from pfsense

Oct 12 19:00:51	nginx		2024/10/12 19:00:51 [error] 74022#100800: *23216 open() "/usr/local/www/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8 HTTP/1.1", host: "ancient-script.org", referrer: "http://ancient-script.org/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8"

This does not reach my server. My server does respond to this message if I put it into curl.

Is there any way to tell pfsense to either not steal /.well-known/acme-challenges or to be selective and only steal the ones with a token that it created?

I hope the web interface is not available from the WAN side by default.

Regards,
John

viragomann

@jgrabner
Disable the redirecting of port 80 in System > Advanced > Administration.

Also you should configure a different port for the web configurator.

SteveITS

@jgrabner pfSense can only forward ports not paths.

What exactly is seen if you request the entire URL externally? Hopefully not the pfSense web login page.

Is the LE request using port 30080 maybe?

jgrabner

@viragomann thank you.

When you say webconfig, are you refering to the web-server that is presented on 192.168.1.1:80 (by default) or in my case 192.168.10.1:80 , i.e. LAN ip address?

Why would changing the webserver port from 192.168.10.1:80 (LAN) to 192.168.1.1:90 (LAN) have any bearing on 150.220.136.18:80 (WAN) being port forwarded to 192.168.10.12:30080?

I will try your suggestion, but would appreciate some education, as i am new to pfsense.

Regards,
John

viragomann

@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:

When you say webconfig, are you refering to the web-server that is presented on 192.168.1.1:80 (by default) or in my case 192.168.10.1:80 , i.e. LAN ip address?

The setting is called "WebGUI redirect".

Why would changing the webserver port from 192.168.10.1:80 (LAN) to 192.168.1.1:90 (LAN) have any bearing on 150.220.136.18:80 (WAN) being port forwarded to 192.168.10.12:30080?

The pfSense web server is bound to all interfaces. You cannot specify a certain interface.

jgrabner

@viragomann

Thank you again.

So the webserver is bound to the wan port but forward what it gets to the port forward address, since my webserver is reachable from the cloud through pfsense, but does not do that for the acme messages from lets encrypt.

Sounds a little inconsistent?

Also, kind of risky binding to bind the wan interface even if by default the webserver is disable on this interface?

Does pfsense advertize that it uses lets encrypt on the wan interface?

Again, thank you for your expertize and educating me. I am just trying to get my head around this thing.

jgrabner

@jgrabner

One more thougth: i assume the processor in pfsense routers to be generaly quite small, relying mostly on hw for oacket routing. If so, having a webserver bound to the WAN port, even if only used for acme lets encrypt, would open the door for a denial of service attach. Ie small amount of traffic on wan could overload processor.

Is there an off switch for acme message processing from the WAN side?

SteveITS

@jgrabner default WAN rules allow no traffic. And, none are necessary there for port forwards.

jgrabner

@SteveITS but this default rule is not preventing acme messages from getting to the pfsense server as demonstrated by the log.

SteveITS

@jgrabner Post your WAN rules? and NAT rules?

It is not possible for http://150.220.136.18 to get to one web server and http://150.220.136.18/path to get to another, since those are both port 80, unless you have a Source set on your NAT rule (therefore, are using two rules).

http://150.220.136.18 gets me to an nginx 404 page.

jgrabner

@SteveITS Then how do you explain the logs in pfsense

Oct 12 20:01:05	nginx		2024/10/12 20:01:05 [error] 73972#100268: *24689 open() "/usr/local/www/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo HTTP/1.1", host: "www.ancient-script.org", referrer: "http://www.ancient-script.org/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo"
Oct 12 20:01:08	nginx		2024/10/12 20:01:08 [error] 74022#100800: *24691 open() "/usr/local/www/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8 HTTP/1.1", host: "ancient-script.org", referrer: "http://ancient-script.org/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8"
Oct 12 20:01:15	nginx		2024/10/12 20:01:15 [error] 74022#100800: *24693 open() "/usr/local/www/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo HTTP/1.1", host: "www.ancient-script.org", referrer: "http://www.ancient-script.org/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo"

The cert-manager I was using in microk8s on my server was talking to let's encrypt and let's encrypt was trying to reach my server, but somehow these logs were in pfsense GUI logs and not my server, but my server was seeing the other HTTP messages.

I don't understand pfsence, but in cert-manager in kubernetes, it installed a hook to pull just this message away from the regular web-server running on my server. I supposed something similar could be happening on pfsense.

I made the changes recommended by @viragomann , now pfsense is not logging anything.

I would be happy to put pfsense back to the bad configuration and try again if this would help an expert understand what is going on in pfsense. Just let me know what to collect, but recall I am new to pfsense.

SteveITS

@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:

192.168.10.12

Is this a PC on LAN? Do you have reflection enabled on the NAT rule?

jgrabner

@SteveITS Not sure what reflection is, do not think so.

here is my nat dump

<nat>
<outbound>
<mode>automatic</mode>
</outbound>
<separator/>
<rule>
<source>
<any/>
</source>
<destination>
<network>wanip</network>
<port>443</port>
</destination>
<ipprotocol>inet</ipprotocol>
<protocol>tcp/udp</protocol>
<target>192.168.10.12</target>
<local-port>30443</local-port>
<interface>wan</interface>
<descr>
<![CDATA[ https for ancient ]]>
</descr>
<associated-rule-id>nat_6702a46dc11a03.71140884</associated-rule-id>
<created>
<time>1728226413</time>
<username>
<![CDATA[ admin@192.168.10.10 (Local Database) ]]>
</username>
</created>
<updated>
<time>1728761303</time>
<username>
<![CDATA[ admin@192.168.10.12 (Local Database) ]]>
</username>
</updated>
</rule>
<rule>
<source>
<any/>
</source>
<destination>
<network>wanip</network>
<port>80</port>
</destination>
<ipprotocol>inet</ipprotocol>
<protocol>tcp/udp</protocol>
<target>192.168.10.12</target>
<local-port>30080</local-port>
<interface>wan</interface>
<descr>
<![CDATA[ http for ancient script ]]>
</descr>
<associated-rule-id>nat_6702a4121ed802.99471913</associated-rule-id>
<created>
<time>1728226322</time>
<username>
<![CDATA[ admin@192.168.10.10 (Local Database) ]]>
</username>
</created>
<updated>
<time>1728761322</time>
<username>
<![CDATA[ admin@192.168.10.12 (Local Database) ]]>
</username>
</updated>
</rule>
</nat>

My web server and other servers are on the LAN side (192.168.10.x). I am now accessing pfsense via 192.168.10.1:90, it was 192.168.10:80 when I observed pfsense logs with my acme messages.

viragomann

@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:

Then how do you explain the logs in pfsense

Oct 12 20:01:05	nginx		2024/10/12 20:01:05 [error] 73972#100268: *24689 open() "/usr/local/www/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo HTTP/1.1", host: "www.ancient-script.org", referrer: "http://www.ancient-script.org/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo"

pfSense is running an Nginx webserver for the UI and obviously your certification manager sends it's request to the public IP (public domain) of your webserver.
Note that the client in this log is 192.168.10.12 and the requested host is the public FQDN.

So this request is coming from LAN, where pfSense was catching requests on port 80, since WebGUI redirect was not disabled.

Don't know, why your cert manager is requesting your own web server. I don't know this behavior from certbot.

Best to add a host override for your web server to your local DNS. However, it should also work if you enable NAT reflection in the port forwarding rule.

jgrabner

@viragomann Thank you. I was seeing what I was expecting, not what it was.

I did not realize that the cert-manager running in my kubernetes node in the LAN validates the reachability of the DNS before asking the let's encrypt servers in the cloud to do the real validation from the cloud. i.e. a check to prevent wasting let's encrypt server processing.

So the NGINX in this log is from my ingress, not from pfsense.

I have been hunting a problem with my cert-manager for a couple of day. Now I see this pre-check was failing because I did not enable NAT reflection.

Thank you so much @viragomann . You saved me a lot of additional debugging time.

viragomann

@jgrabner
Glade that your acme is working now.

Gertjan

@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:

Here is the log from pfsense

Oct 12 19:00:51 nginx 2024/10/12 19:00:51 [error] 74022#100800: *23216 open() "/usr/local/www/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8 HTTP/1.1", host: "ancient-script.org", referrer: "http://ancient-script.org/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8"

This does not reach my server.

Read the log line again.
It, nginx ! (your web server !) says :
Error on opening a file. This file :

/usr/local/www/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8

It is close to impossible that some random visitor was visiting your web server, and asking for the folders in folder ".well-known/acme-challenge" (but this can happen) and asking for a very specific file name "1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" and now I'm ready to bet that this veryb that file name was given by Lets-encrypt to the acme;sh script so it creates a file with that name.
And in the file a random text should be placed, as given to the acme.sh by Lets-encrypt.

No more talks, congrats, Lets-encrypt did reach out to your web server and was asking for the challenge file, to get the challenge code (the content of the file).

But the file wasn't there.

The thing is : your acme.sh isn't set up correctly, as it did not create the file with the name "1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" in the root folder of the web server, in the folder/folder (with the also special content).

Its time to have a look at the very detailed acme.sh log file.
It told you were it was : /tmp/acme/YOUR-ACCOUNT-NAME/acme_issuecert.log

stephenw10

That log is from the pfSense nginx instance it uses for the webgui. The request was hitting that because it was coming from the LAN and without NAT reflection that does not hit the port forward.

Gertjan

@stephenw10

That request came from LAN ?
Such a request would be initiated from Letsencrypt, as that's the file it want to 'see'.
Omg ... it was written :

Here is the log from pfsense

The needed NAT rule :

and from now on, everything that comes into the WAN using port 80 will get redirected to 192.168.10.12 port 30080.

jgrabner

I was not expecting the cert-manager to do a local, from the LAN, validation of the reachability of the .well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8 path before telling the let's encrypt to do the real challenge from the internet server i.e. over the WAN. So when I saw this in the pfsense log, I assumed incorrectly PFsense was intercepting from the WAN the above for it's own use. I was in error.

NAT reflection made everything work.