PfBlockerNG: IPv4 Lists - List.txt Download from internal web server fails
-
pfBlockerNG 1.06 Observation: when configuring IPv4 Alias / List configuration to block or permit IP's using a custom list.
I wanted to down load a custom text file containing a list of ip's from a dedicated internal webserver (not the pfsense box itself).
I was using a simple txt file format and a http link like…. h-t-t-p://www.anyoldserver.co.uk/pfsense/mylist.txtI was suprised that it failed!
A little testing showed me that it will not work if the web server hosting the file is on the internal lan interface.
(note: web server and domain all work fine... not an issue with web server or network access.)However it works fine if the target txt file is hosted out on the WAN (internet).
looked using wire shark sniffer and no request is sent on the lan side.Any one got a work around or can confirm this as an issue or put me out of my misery and explain why this would be a feature?
Could it be somthing to do with inbound and out bound interface settings or its just programed to send the request out the WAN side only.
any assistance is appreciated.
Thanks in advance. -
Hi EdwardLand,
If you open a shell to your pfSense box, can you fetch the file? If not, then there is an issue in your network preventing fetching the file from your webserver.
cd /tmp
fetch -o testfile.txt "h-t-t-p://www.anyoldserver.co.uk/pfsense/mylist.txt"Open the testfile.txt in vi or any reader and see if the contents are correct?
Please ensure that its in the correct format x.x.x.x or x.x.x.x/xx or x.x.x.x-x.x.x.x and one entry per line. -
Hi EdwardLand,
If you open a shell to your pfSense box, can you fetch the file? If not, then there is an issue in your network preventing fetching the file from your webserver.
cd /tmp
fetch -o testfile.txt "h-t-t-p://www.anyoldserver.co.uk/pfsense/mylist.txt"Thanks BBcan177
I did a quick check and tested using your suggestion. It produces and operation time out. when run from the console of the pfsense.
using any machine on the lan…
The server delivers the text file to the fire fox browser just fine... dns works, file is returned and shown with content as expected.I think i may be somthing to do with resolving the server name back to the wan IP when the request orriginates on the same WAN interface.
ill do a little more investigation and get back with some more info about it.
Thanks.
-
Maybe try by IP : "http://x.x.x.x/pfsense/mylist.txt"
-
Maybe try by IP : "http://x.x.x.x/pfsense/mylist.txt"
BBcan177 in reply..
Tested this by adding the text file to the default website folder and using the servers IP LAN address in the URL.
It works.
The text file is retrieved from the webserver and contents are as expected.i also tested to eliminate dns issues:
Using the pfsense console (ping option 7) i can enter the external domain name and it resolves to its correct WAN ip of the router and pings ok.so its looking like a program / coding issue.
I think the pfblocker only expects the URL to be hosted on an external WAN interface.
i just need to get this confirmed so i can seek a work around. Any one else encounter this behaviour?
i have seen this limitation in some routers where internal users cant access internal webservers using resolved external lookups. Usually due to a NAT issue. (Note: all my lan machines work ok and can access the internal webserver ok using external dns domain name lookups)
ideas of what it could be are welcome from all…
I will keep looking... -
So you are trying to download a file from an internal server using your pfSense's WAN IP? Why? The problem you are seeing is NAT reflection. Just simply use the internal hostname or IP of the webserver.
-
I think you have a DNS or NAT issue as Fragged said above… If the fetch command fails in the shell then pfBlockerNG will never download the file. Maybe try split DNS and create a DNS entry for your http://www.anyoldserver.co.uk and reference the Internal Lan IP.
You can also use this command in the pfSense Diagnostics to do a similar test as the shell command you tried before.
In the "Diagnostics:Execute Shell Command:'
var_dump(file("h-t-t-p://www.anyoldserver.co.uk/pfsense/mylist.txt"));
That should print the file to the screen if pfSense can resolve that DNS name. But if its working with using the IP address instead just leave it, unless you are having other issues?
-
So you are trying to download a file from an internal server using your pfSense's WAN IP? Why? The problem you are seeing is NAT reflection. Just simply use the internal hostname or IP of the webserver.
Good point fragged… but i have to use the external domain name as the webserver uses the host header names to direct the request to the correct website on the server.
pfsense is port forwarding port 80 to the internal LAN IP of the web server. The domain name is used to direct which hosted site the webserver responds to.
The webserver is fully configured and working with several domains associated with it.
the default webserver is usually disabled, i only enabled it for teting this issue. So dumping a txt file on the default website is not possible for live work.just seen BBcan177's reply too....
stand by i will just test using the url with host name, doamin and IP address etc.
to see what works and what does not.oh... for reference 1:1 NAT is disabled
console Results for: fetch -o testfile.txt "URL"
----Test Group 1----
using ip URL= h-t-t-p://x.x.x.x/pfsense/mylist.txt works ok
using Hostname h-t-t-p://localservername/pfsense/mylist.txt works ok
using Hostname and domain h-t-t-p://localservername.mydomain.co.uk/pfsense/mylist.txt works ok----Test Group 2----
using domain h-t-t-p://mydomain.co.uk/pfsense/mylist.txt Fails.
using www.domain h-t-t-p://www.mydomain.co.uk/pfsense/mylist.txt Fails.----Test Group 3----
ping domain ping mydomain.co.uk resolves ok.
ping www.domain ping www.mydomain.co.uk resolves ok.Test group 1 all point to internal IP of webserver.... these all work so internal DNS lookup fine.
Test group 2 all point to WAN IP of router.... these all fail when used from the router console.
Test group 3 all point to WAN IP of router.... these all work ok when used from the router console ping.**Moment of Inspiration!
Added "www" as a host pointing to the internal lan ip of the webserver in the DNS forwarder.
I can now resolve www.mydomain.co.uk to the internal ip of the web server.**I think the key here is that the web server uses host headers to identify the website to access.
it expects to see "www.mysite.co.uk" in full.
so "mysite.co.uk" will not return a result.
hence the first test in group 2 failed and now works with "www" added.
pfsense now resolves www as an internal ip and at the same time has the full correct host header.If this had not of worked my next step would have been..
Setup DNS to return a different result internally to externally, split-horizon DNS as BBcan177 suggested.found this in the forum if this helps anyone….
https://doc.pfsense.org/index.php/Why_can%27t_I_access_forwarded_ports_on_my_WAN_IP_from_my_LAN/OPTx_networks%3FThank you very much for the info and advice... very much helped keep the brain cells working.
Thank you all.18gr .22 800fps ::)