Port Forwarding not honered for .well-known/acme-challenge
-
@viragomann thank you.
When you say webconfig, are you refering to the web-server that is presented on 192.168.1.1:80 (by default) or in my case 192.168.10.1:80 , i.e. LAN ip address?
Why would changing the webserver port from 192.168.10.1:80 (LAN) to 192.168.1.1:90 (LAN) have any bearing on 150.220.136.18:80 (WAN) being port forwarded to 192.168.10.12:30080?
I will try your suggestion, but would appreciate some education, as i am new to pfsense.
Regards,
John -
@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:
When you say webconfig, are you refering to the web-server that is presented on 192.168.1.1:80 (by default) or in my case 192.168.10.1:80 , i.e. LAN ip address?
The setting is called "WebGUI redirect".
Why would changing the webserver port from 192.168.10.1:80 (LAN) to 192.168.1.1:90 (LAN) have any bearing on 150.220.136.18:80 (WAN) being port forwarded to 192.168.10.12:30080?
The pfSense web server is bound to all interfaces. You cannot specify a certain interface.
-
Thank you again.
So the webserver is bound to the wan port but forward what it gets to the port forward address, since my webserver is reachable from the cloud through pfsense, but does not do that for the acme messages from lets encrypt.
Sounds a little inconsistent?
Also, kind of risky binding to bind the wan interface even if by default the webserver is disable on this interface?
Does pfsense advertize that it uses lets encrypt on the wan interface?
Again, thank you for your expertize and educating me. I am just trying to get my head around this thing.
-
One more thougth: i assume the processor in pfsense routers to be generaly quite small, relying mostly on hw for oacket routing. If so, having a webserver bound to the WAN port, even if only used for acme lets encrypt, would open the door for a denial of service attach. Ie small amount of traffic on wan could overload processor.
Is there an off switch for acme message processing from the WAN side?
-
@jgrabner default WAN rules allow no traffic. And, none are necessary there for port forwards.
-
@SteveITS but this default rule is not preventing acme messages from getting to the pfsense server as demonstrated by the log.
-
@jgrabner Post your WAN rules? and NAT rules?
It is not possible for http://150.220.136.18 to get to one web server and http://150.220.136.18/path to get to another, since those are both port 80, unless you have a Source set on your NAT rule (therefore, are using two rules).
http://150.220.136.18 gets me to an nginx 404 page.
-
@SteveITS Then how do you explain the logs in pfsense
Oct 12 20:01:05 nginx 2024/10/12 20:01:05 [error] 73972#100268: *24689 open() "/usr/local/www/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo HTTP/1.1", host: "www.ancient-script.org", referrer: "http://www.ancient-script.org/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo" Oct 12 20:01:08 nginx 2024/10/12 20:01:08 [error] 74022#100800: *24691 open() "/usr/local/www/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8 HTTP/1.1", host: "ancient-script.org", referrer: "http://ancient-script.org/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" Oct 12 20:01:15 nginx 2024/10/12 20:01:15 [error] 74022#100800: *24693 open() "/usr/local/www/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo HTTP/1.1", host: "www.ancient-script.org", referrer: "http://www.ancient-script.org/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo"The cert-manager I was using in microk8s on my server was talking to let's encrypt and let's encrypt was trying to reach my server, but somehow these logs were in pfsense GUI logs and not my server, but my server was seeing the other HTTP messages.
I don't understand pfsence, but in cert-manager in kubernetes, it installed a hook to pull just this message away from the regular web-server running on my server. I supposed something similar could be happening on pfsense.
I made the changes recommended by @viragomann , now pfsense is not logging anything.
I would be happy to put pfsense back to the bad configuration and try again if this would help an expert understand what is going on in pfsense. Just let me know what to collect, but recall I am new to pfsense.
-
@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:
192.168.10.12
Is this a PC on LAN? Do you have reflection enabled on the NAT rule?
-
@SteveITS Not sure what reflection is, do not think so.
here is my nat dump
<nat>
<outbound>
<mode>automatic</mode>
</outbound>
<separator/>
<rule>
<source>
<any/>
</source>
<destination>
<network>wanip</network>
<port>443</port>
</destination>
<ipprotocol>inet</ipprotocol>
<protocol>tcp/udp</protocol>
<target>192.168.10.12</target>
<local-port>30443</local-port>
<interface>wan</interface>
<descr>
<![CDATA[ https for ancient ]]>
</descr>
<associated-rule-id>nat_6702a46dc11a03.71140884</associated-rule-id>
<created>
<time>1728226413</time>
<username>
<![CDATA[ admin@192.168.10.10 (Local Database) ]]>
</username>
</created>
<updated>
<time>1728761303</time>
<username>
<![CDATA[ admin@192.168.10.12 (Local Database) ]]>
</username>
</updated>
</rule>
<rule>
<source>
<any/>
</source>
<destination>
<network>wanip</network>
<port>80</port>
</destination>
<ipprotocol>inet</ipprotocol>
<protocol>tcp/udp</protocol>
<target>192.168.10.12</target>
<local-port>30080</local-port>
<interface>wan</interface>
<descr>
<![CDATA[ http for ancient script ]]>
</descr>
<associated-rule-id>nat_6702a4121ed802.99471913</associated-rule-id>
<created>
<time>1728226322</time>
<username>
<![CDATA[ admin@192.168.10.10 (Local Database) ]]>
</username>
</created>
<updated>
<time>1728761322</time>
<username>
<![CDATA[ admin@192.168.10.12 (Local Database) ]]>
</username>
</updated>
</rule>
</nat>My web server and other servers are on the LAN side (192.168.10.x). I am now accessing pfsense via 192.168.10.1:90, it was 192.168.10:80 when I observed pfsense logs with my acme messages.
-
@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:
Then how do you explain the logs in pfsense
Oct 12 20:01:05 nginx 2024/10/12 20:01:05 [error] 73972#100268: *24689 open() "/usr/local/www/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo HTTP/1.1", host: "www.ancient-script.org", referrer: "http://www.ancient-script.org/.well-known/acme-challenge/1A6AQqAwJKY0rqmLaiV77vnSirvIYBhq5pPmalv0iZo"pfSense is running an Nginx webserver for the UI and obviously your certification manager sends it's request to the public IP (public domain) of your webserver.
Note that the client in this log is 192.168.10.12 and the requested host is the public FQDN.So this request is coming from LAN, where pfSense was catching requests on port 80, since WebGUI redirect was not disabled.
Don't know, why your cert manager is requesting your own web server. I don't know this behavior from certbot.
Best to add a host override for your web server to your local DNS. However, it should also work if you enable NAT reflection in the port forwarding rule.
-
@viragomann Thank you. I was seeing what I was expecting, not what it was.
I did not realize that the cert-manager running in my kubernetes node in the LAN validates the reachability of the DNS before asking the let's encrypt servers in the cloud to do the real validation from the cloud. i.e. a check to prevent wasting let's encrypt server processing.
So the NGINX in this log is from my ingress, not from pfsense.
I have been hunting a problem with my cert-manager for a couple of day. Now I see this pre-check was failing because I did not enable NAT reflection.
Thank you so much @viragomann . You saved me a lot of additional debugging time.
-
@jgrabner
Glade that your acme is working now. -
@jgrabner said in Port Forwarding not honered for .well-known/acme-challenge:
Here is the log from pfsense
Oct 12 19:00:51 nginx 2024/10/12 19:00:51 [error] 74022#100800: *23216 open() "/usr/local/www/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" failed (2: No such file or directory), client: 192.168.10.12, server: , request: "GET /.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8 HTTP/1.1", host: "ancient-script.org", referrer: "http://ancient-script.org/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8"
This does not reach my server.
Read the log line again.
It, nginx ! (your web server !) says :
Error on opening a file. This file :/usr/local/www/.well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8
It is close to impossible that some random visitor was visiting your web server, and asking for the folders in folder ".well-known/acme-challenge" (but this can happen) and asking for a very specific file name "1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" and now I'm ready to bet that this veryb that file name was given by Lets-encrypt to the acme;sh script so it creates a file with that name.
And in the file a random text should be placed, as given to the acme.sh by Lets-encrypt.No more talks, congrats, Lets-encrypt did reach out to your web server and was asking for the challenge file, to get the challenge code (the content of the file).
But the file wasn't there.
The thing is : your acme.sh isn't set up correctly, as it did not create the file with the name "1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8" in the root folder of the web server, in the folder/folder (with the also special content).
Its time to have a look at the very detailed acme.sh log file.
It told you were it was : /tmp/acme/YOUR-ACCOUNT-NAME/acme_issuecert.log -
That log is from the pfSense nginx instance it uses for the webgui. The request was hitting that because it was coming from the LAN and without NAT reflection that does not hit the port forward.
-
That request came from LAN ?
Such a request would be initiated from Letsencrypt, as that's the file it want to 'see'.
Omg ... it was written :Here is the log from pfsense
The needed NAT rule :
and from now on, everything that comes into the WAN using port 80 will get redirected to 192.168.10.12 port 30080.
-
I was not expecting the cert-manager to do a local, from the LAN, validation of the reachability of the .well-known/acme-challenge/1A9j2r1QaH4qQ8igoBlYEde3YC8_TgorjDIUJIb9bC8 path before telling the let's encrypt to do the real challenge from the internet server i.e. over the WAN. So when I saw this in the pfsense log, I assumed incorrectly PFsense was intercepting from the WAN the above for it's own use. I was in error.
NAT reflection made everything work.
-
@jgrabner
all fine and good, but now that you know that nat reflection works for you, you should consider setting up a split DNS instead if you canhttps://docs.netgate.com/pfsense/en/latest/nat/reflection.html
NAT reflection is a hack as it loops traffic through the firewall when it is not necessaryA preferable alternative to NAT reflection is deploying a split DNS infrastructure
̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
Please do not use chat/PM to ask for help
we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
Don't forget to Upvote with the 👍 button for any post you find to be helpful. -
@kiokoman said in Port Forwarding not honered for .well-known/acme-challenge:
https://docs.netgate.com/pfsense/en/latest/nat/reflection.html
NAT reflection is a hack as it loops traffic through the firewall when it is not necessaryI've an example that shows why NAT refection is 'strange' :
You work for a company, the sales department. You just received a big order, and you want to contact the shipping department to check if they have enough stock.
In your company, internal phone numbers 310 to 320 are the sales numbers, 350 to 360 are the phones of shipping department.
Suzanne is holding the front desk, and she answers all the incoming calls from all the clients and everybody else out there, when calling your company.Let's say : 310->320 and 350->360 are your LANs.
Suzanne is your pfSense.So, what do you do ?
Do you really call 1-123-456-7890 ? (and thus you will have Suzanne answering, an you have to ask here to put shipping through ? )
Or do you call 'shipping' directly by dialing for example 350 ?If you persist on doing the first option, Suzanne will have a talk with you, explaining you doing it wrong (again). That your methods are maybe not hacky, bit still wrong.
-
@kiokoman said in Port Forwarding not honered for .well-known/acme-challenge:
you should consider setting up a split DNS instead if you can
You can not. Since you're doing port translation, you need the NAT rule on pfSense.
However , I'm wondering why your server use non-default ports fot HTTP/S.
With default ports you could go with local host overrides and get rid of NAT reflection.
