Port Forwarding not honered for .well-known/acme-challenge

jgrabner

@SteveITS but this default rule is not preventing acme messages from getting to the pfsense server as demonstrated by the log.

SteveITS

@jgrabner Post your WAN rules? and NAT rules?

It is not possible for http://150.220.136.18 to get to one web server and http://150.220.136.18/path to get to another, since those are both port 80, unless you have a Source set on your NAT rule (therefore, are using two rules).

http://150.220.136.18 gets me to an nginx 404 page.

jgrabner

@SteveITS Then how do you explain the logs in pfsense

Oct 12 20:01:05	nginx		2024/10/12 20:01:05 [error] 73972#100268: *24689 open() "/usr/local/www/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo HTTP/1.1", host: "www.ancient-script.org", referrer: "http://www.ancient-script.org/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo"
Oct 12 20:01:08	nginx		2024/10/12 20:01:08 [error] 74022#100800: *24691 open() "/usr/local/www/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8 HTTP/1.1", host: "ancient-script.org", referrer: "http://ancient-script.org/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8"
Oct 12 20:01:15	nginx		2024/10/12 20:01:15 [error] 74022#100800: *24693 open() "/usr/local/www/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo HTTP/1.1", host: "www.ancient-script.org", referrer: "http://www.ancient-script.org/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo"

The cert-manager I was using in microk8s on my server was talking to let's encrypt and let's encrypt was trying to reach my server, but somehow these logs were in pfsense GUI logs and not my server, but my server was seeing the other HTTP messages.

I don't understand pfsence, but in cert-manager in kubernetes, it installed a hook to pull just this message away from the regular web-server running on my server. I supposed something similar could be happening on pfsense.

I made the changes recommended by @viragomann , now pfsense is not logging anything.

I would be happy to put pfsense back to the bad configuration and try again if this would help an expert understand what is going on in pfsense. Just let me know what to collect, but recall I am new to pfsense.

SteveITS

@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:

192.168.10.12

Is this a PC on LAN? Do you have reflection enabled on the NAT rule?

jgrabner

@SteveITS Not sure what reflection is, do not think so.

here is my nat dump

<nat>
<outbound>
<mode>automatic</mode>
</outbound>
<separator/>
<rule>
<source>
<any/>
</source>
<destination>
<network>wanip</network>
<port>443</port>
</destination>
<ipprotocol>inet</ipprotocol>
<protocol>tcp/udp</protocol>
<target>192.168.10.12</target>
<local-port>30443</local-port>
<interface>wan</interface>
<descr>
<![CDATA[ https for ancient ]]>
</descr>
<associated-rule-id>nat_6702a46dc11a03.71140884</associated-rule-id>
<created>
<time>1728226413</time>
<username>
<![CDATA[ admin@192.168.10.10 (Local Database) ]]>
</username>
</created>
<updated>
<time>1728761303</time>
<username>
<![CDATA[ admin@192.168.10.12 (Local Database) ]]>
</username>
</updated>
</rule>
<rule>
<source>
<any/>
</source>
<destination>
<network>wanip</network>
<port>80</port>
</destination>
<ipprotocol>inet</ipprotocol>
<protocol>tcp/udp</protocol>
<target>192.168.10.12</target>
<local-port>30080</local-port>
<interface>wan</interface>
<descr>
<![CDATA[ http for ancient script ]]>
</descr>
<associated-rule-id>nat_6702a4121ed802.99471913</associated-rule-id>
<created>
<time>1728226322</time>
<username>
<![CDATA[ admin@192.168.10.10 (Local Database) ]]>
</username>
</created>
<updated>
<time>1728761322</time>
<username>
<![CDATA[ admin@192.168.10.12 (Local Database) ]]>
</username>
</updated>
</rule>
</nat>

My web server and other servers are on the LAN side (192.168.10.x). I am now accessing pfsense via 192.168.10.1:90, it was 192.168.10:80 when I observed pfsense logs with my acme messages.

viragomann

@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:

Then how do you explain the logs in pfsense

Oct 12 20:01:05	nginx		2024/10/12 20:01:05 [error] 73972#100268: *24689 open() "/usr/local/www/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo HTTP/1.1", host: "www.ancient-script.org", referrer: "http://www.ancient-script.org/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo"

pfSense is running an Nginx webserver for the UI and obviously your certification manager sends it's request to the public IP (public domain) of your webserver.
Note that the client in this log is 192.168.10.12 and the requested host is the public FQDN.

So this request is coming from LAN, where pfSense was catching requests on port 80, since WebGUI redirect was not disabled.

Don't know, why your cert manager is requesting your own web server. I don't know this behavior from certbot.

Best to add a host override for your web server to your local DNS. However, it should also work if you enable NAT reflection in the port forwarding rule.

jgrabner

@viragomann Thank you. I was seeing what I was expecting, not what it was.

I did not realize that the cert-manager running in my kubernetes node in the LAN validates the reachability of the DNS before asking the let's encrypt servers in the cloud to do the real validation from the cloud. i.e. a check to prevent wasting let's encrypt server processing.

So the NGINX in this log is from my ingress, not from pfsense.

I have been hunting a problem with my cert-manager for a couple of day. Now I see this pre-check was failing because I did not enable NAT reflection.

Thank you so much @viragomann . You saved me a lot of additional debugging time.

viragomann

@jgrabner
Glade that your acme is working now.

Gertjan

@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:

Here is the log from pfsense

Oct 12 19:00:51 nginx 2024/10/12 19:00:51 [error] 74022#100800: *23216 open() "/usr/local/www/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8 HTTP/1.1", host: "ancient-script.org", referrer: "http://ancient-script.org/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8"

This does not reach my server.

Read the log line again.
It, nginx ! (your web server !) says :
Error on opening a file. This file :

/usr/local/www/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8

It is close to impossible that some random visitor was visiting your web server, and asking for the folders in folder ".well-known/acme-challenge" (but this can happen) and asking for a very specific file name "1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" and now I'm ready to bet that this veryb that file name was given by Lets-encrypt to the acme;sh script so it creates a file with that name.
And in the file a random text should be placed, as given to the acme.sh by Lets-encrypt.

No more talks, congrats, Lets-encrypt did reach out to your web server and was asking for the challenge file, to get the challenge code (the content of the file).

But the file wasn't there.

The thing is : your acme.sh isn't set up correctly, as it did not create the file with the name "1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" in the root folder of the web server, in the folder/folder (with the also special content).

Its time to have a look at the very detailed acme.sh log file.
It told you were it was : /tmp/acme/YOUR-ACCOUNT-NAME/acme_issuecert.log

stephenw10

That log is from the pfSense nginx instance it uses for the webgui. The request was hitting that because it was coming from the LAN and without NAT reflection that does not hit the port forward.

Gertjan

@stephenw10

That request came from LAN ?
Such a request would be initiated from Letsencrypt, as that's the file it want to 'see'.
Omg ... it was written :

Here is the log from pfsense

The needed NAT rule :

and from now on, everything that comes into the WAN using port 80 will get redirected to 192.168.10.12 port 30080.

jgrabner

I was not expecting the cert-manager to do a local, from the LAN, validation of the reachability of the .well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8 path before telling the let's encrypt to do the real challenge from the internet server i.e. over the WAN. So when I saw this in the pfsense log, I assumed incorrectly PFsense was intercepting from the WAN the above for it's own use. I was in error.

NAT reflection made everything work.

kiokoman

@jgrabner
all fine and good, but now that you know that nat reflection works for you, you should consider setting up a split DNS instead if you can

https://docs.netgate.com/pfsense/en/latest/nat/reflection.html
NAT reflection is a hack as it loops traffic through the firewall when it is not necessary

A preferable alternative to NAT reflection is deploying a split DNS infrastructure

Gertjan

@kiokoman said in Port Forwarding not honered for .well-known/acme-challenge:

https://docs.netgate.com/pfsense/en/latest/nat/reflection.html
NAT reflection is a hack as it loops traffic through the firewall when it is not necessary

I've an example that shows why NAT refection is 'strange' :

@jgrabner

You work for a company, the sales department. You just received a big order, and you want to contact the shipping department to check if they have enough stock.
In your company, internal phone numbers 310 to 320 are the sales numbers, 350 to 360 are the phones of shipping department.
Suzanne is holding the front desk, and she answers all the incoming calls from all the clients and everybody else out there, when calling your company.

Let's say : 310->320 and 350->360 are your LANs.
Suzanne is your pfSense.

So, what do you do ?
Do you really call 1-123-456-7890 ? (and thus you will have Suzanne answering, an you have to ask here to put shipping through ? )
Or do you call 'shipping' directly by dialing for example 350 ?

If you persist on doing the first option, Suzanne will have a talk with you, explaining you doing it wrong (again). That your methods are maybe not hacky, bit still wrong.

viragomann

@kiokoman said in Port Forwarding not honered for .well-known/acme-challenge:

you should consider setting up a split DNS instead if you can

You can not. Since you're doing port translation, you need the NAT rule on pfSense.

However , I'm wondering why your server use non-default ports fot HTTP/S.
With default ports you could go with local host overrides and get rid of NAT reflection.

jgrabner

@viragomann

Reason for non-standard ports on server:

my webserver is a set of pods in kubernetes. I am using microk8s implementation. While it supports hostPort on ingres (le lets me use port 80 and 443), it is highly discouraged in documentation. Documentation recomends nodePort configuration (ie 30000-). The only reason i could figure is that using hostPort would limit kubernetes to one webserver and kubernetes is designed for scale.
somewhere i read it was less secure to use 80, ie target of more hacking if someone got access to your lan. I suppose this only applies to lower skill hackers.

kiokoman

@viragomann said in Port Forwarding not honered for .well-known/acme-challenge:

@kiokoman said in Port Forwarding not honered for .well-known/acme-challenge:

you should consider setting up a split DNS instead if you can

You can not. Since you're doing port translation, you need the NAT rule on pfSense.

However , I'm wondering why your server use non-default ports fot HTTP/S.
With default ports you could go with local host overrides and get rid of NAT reflection.

you can use haproxy in this scenario listening on wan and lan instead of opening ports/creating a nats for each pod in Kubernetes, well if you have a couple of pods it doesn't really matter but since I have 50 services running in test / 50 in staging / 50 in production on Kubernetes behind pfsense it would be unmanageable without haproxy for me