Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PfBlockerNG: IPv4 Lists - List.txt Download from internal web server fails

    Scheduled Pinned Locked Moved pfSense Packages
    8 Posts 3 Posters 3.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      EdwardLand
      last edited by

      pfBlockerNG 1.06 Observation: when configuring IPv4 Alias / List configuration to block or permit IP's using a custom list.

      I wanted to down load a custom text file containing a list of ip's from a dedicated internal webserver (not the pfsense box itself).
      I was using a simple txt file format and a http link like…. h-t-t-p://www.anyoldserver.co.uk/pfsense/mylist.txt

      I was suprised that it failed!
      A little testing showed me that it will not work if the web server hosting the file is on the internal lan interface.
      (note: web server and domain all work fine... not an issue with web server or network access.)

      However it works fine if the target txt file is hosted out on the WAN (internet).
      looked using wire shark sniffer and no request is sent on the lan side.

      Any one got a work around or can confirm this as an issue or put me out of my misery and explain why this would be a feature?

      Could it be somthing to do with inbound and out bound interface settings or its just programed to send the request out the WAN side only.

      any assistance is appreciated.
      Thanks in advance.

      1 Reply Last reply Reply Quote 0
      • BBcan177B
        BBcan177 Moderator
        last edited by

        Hi EdwardLand,

        If you open a shell to your pfSense box, can you fetch the file? If not, then there is an issue in your network preventing fetching the file from your webserver.

        cd /tmp
          fetch -o testfile.txt "h-t-t-p://www.anyoldserver.co.uk/pfsense/mylist.txt"

        Open the testfile.txt in vi or any reader and see if the contents are correct?
        Please ensure that its in the correct format  x.x.x.x  or  x.x.x.x/xx  or  x.x.x.x-x.x.x.x and one entry per line.

        "Experience is something you don't get until just after you need it."

        Website: http://pfBlockerNG.com
        Twitter: @BBcan177  #pfBlockerNG
        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

        1 Reply Last reply Reply Quote 0
        • E
          EdwardLand
          last edited by

          @BBcan177:

          Hi EdwardLand,

          If you open a shell to your pfSense box, can you fetch the file? If not, then there is an issue in your network preventing fetching the file from your webserver.

          cd /tmp
            fetch -o testfile.txt "h-t-t-p://www.anyoldserver.co.uk/pfsense/mylist.txt"

          Thanks BBcan177

          I did a quick check and tested using your suggestion. It produces and operation time out. when run from the console of the pfsense.

          using any machine on the lan…
          The server delivers the text file to the fire fox browser just fine... dns works, file is returned and shown with content as expected.

          I think i may be somthing to do with resolving the server name back to the wan IP when the request orriginates on the same WAN interface.

          ill do a little more investigation and get back with some more info about it.

          Thanks.

          1 Reply Last reply Reply Quote 0
          • BBcan177B
            BBcan177 Moderator
            last edited by

            Maybe try by IP :  "http://x.x.x.x/pfsense/mylist.txt"

            "Experience is something you don't get until just after you need it."

            Website: http://pfBlockerNG.com
            Twitter: @BBcan177  #pfBlockerNG
            Reddit: https://www.reddit.com/r/pfBlockerNG/new/

            1 Reply Last reply Reply Quote 0
            • E
              EdwardLand
              last edited by

              @BBcan177:

              Maybe try by IP :  "http://x.x.x.x/pfsense/mylist.txt"

              BBcan177  in reply..

              Tested this by adding the text file to the default website folder and using the servers IP LAN address in the URL.

              It works.
              The text file is retrieved from the webserver and contents are as expected.

              i also tested to eliminate dns issues:
              Using the pfsense console (ping option 7) i can enter the external domain name and it resolves to its correct WAN ip of the router and pings ok.

              so its looking like a program / coding issue.

              I think the pfblocker only expects the URL to be hosted on an external WAN interface.

              i just need to get this confirmed so i can seek a work around. Any one else encounter this behaviour?

              i have seen this limitation in some routers where internal users cant access internal webservers using resolved external lookups. Usually due to a NAT issue. (Note: all my lan machines work ok and can access the internal webserver ok using external dns domain name lookups)

              ideas of what it could be are welcome from all…
              I will keep looking...

              1 Reply Last reply Reply Quote 0
              • F
                fragged
                last edited by

                So you are trying to download a file from an internal server using your pfSense's WAN IP? Why? The problem you are seeing is NAT reflection. Just simply use the internal hostname or IP of the webserver.

                1 Reply Last reply Reply Quote 0
                • BBcan177B
                  BBcan177 Moderator
                  last edited by

                  I think you have a DNS or NAT issue as Fragged said above… If the fetch command fails in the shell then pfBlockerNG will never download the file. Maybe try split DNS and create a DNS entry for your http://www.anyoldserver.co.uk and reference the Internal Lan IP.

                  You can also use this command in the pfSense Diagnostics to do a similar test as the shell command you tried before.

                  In the "Diagnostics:Execute Shell Command:'

                  var_dump(file("h-t-t-p://www.anyoldserver.co.uk/pfsense/mylist.txt"));

                  That should print the file to the screen if pfSense can resolve that DNS name. But if its working with using the IP address instead just leave it, unless you are having other issues?

                  "Experience is something you don't get until just after you need it."

                  Website: http://pfBlockerNG.com
                  Twitter: @BBcan177  #pfBlockerNG
                  Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                  1 Reply Last reply Reply Quote 0
                  • E
                    EdwardLand
                    last edited by

                    @fragged:

                    So you are trying to download a file from an internal server using your pfSense's WAN IP? Why? The problem you are seeing is NAT reflection. Just simply use the internal hostname or IP of the webserver.

                    Good point fragged… but i have to use the external domain name as the webserver uses the host header names to direct the request to the correct website on the server.

                    pfsense is port forwarding port 80 to the internal LAN IP of the web server. The domain name is used to direct which hosted site the webserver responds to.

                    The webserver is fully configured and working with several domains associated with it.
                    the default webserver is usually disabled, i only enabled it for teting this issue. So dumping a txt file on the default website is not possible for live work.

                    just seen BBcan177's reply too....

                    stand by i will just test using the url with host name, doamin and IP address etc.
                    to see what works and what does not.

                    oh...  for reference 1:1 NAT is disabled

                    console Results for: fetch -o testfile.txt "URL"

                    ----Test Group 1----
                    using ip                            URL= h-t-t-p://x.x.x.x/pfsense/mylist.txt                                                works ok
                    using Hostname                        h-t-t-p://localservername/pfsense/mylist.txt                                works ok
                    using Hostname and domain    h-t-t-p://localservername.mydomain.co.uk/pfsense/mylist.txt      works ok

                    ----Test Group 2----
                    using domain                            h-t-t-p://mydomain.co.uk/pfsense/mylist.txt                                  Fails.
                    using www.domain                  h-t-t-p://www.mydomain.co.uk/pfsense/mylist.txt                        Fails.

                    ----Test Group 3----
                    ping  domain                      ping mydomain.co.uk                                  resolves ok.
                    ping  www.domain              ping www.mydomain.co.uk                        resolves ok.

                    Test group 1 all point to internal IP of webserver.... these all work so internal DNS lookup fine.
                    Test group 2 all point to WAN IP of router.... these all fail when used from the router console.
                    Test group 3 all point to WAN IP of router.... these all work ok when used from the router console ping.

                    **Moment of Inspiration!

                    Added "www" as a host pointing to the internal lan ip of the webserver in the DNS forwarder.
                    I can now resolve www.mydomain.co.uk to the internal ip of the web server.**

                    I think the key here is that the web server uses host headers to identify the website to access.
                    it expects to see "www.mysite.co.uk" in full.
                    so "mysite.co.uk" will not return a result.
                    hence the first test in group 2 failed and now works with "www" added.
                    pfsense now resolves www as an internal ip and at the same time has the full correct host header.

                    If this had not of worked my next step would have been..
                    Setup DNS to return a different result internally to externally, split-horizon DNS as BBcan177 suggested.

                    found this in the forum if this helps anyone….
                    https://doc.pfsense.org/index.php/Why_can%27t_I_access_forwarded_ports_on_my_WAN_IP_from_my_LAN/OPTx_networks%3F

                    Thank you very much for the info and advice... very much helped keep the brain cells working.
                    Thank you all.

                    18gr .22 800fps  ::)

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.