Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Captive portal and allowed hosts problem

    Scheduled Pinned Locked Moved 2.0-RC Snapshot Feedback and Problems - RETIRED
    12 Posts 4 Posters 15.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      clevenp
      last edited by

      Hello

      I have been using pfsense 2.0 for some time now and occassionally i updated the software (about once per month).

      recently I had to move my email servers to the google servers and I requested what to do and a new version came out with "allowed hosts: in the captive portal section
      I was able to configure this for the users who had no login on the captive portal so the machines were updated able to go to some sites (e.g. the avast servers)

      now I wanted to move all my clients to google mail and I needed to make sure the imap.googlemail.com/imap.gmail.com/pop.googlemail.com/pop.gmail.com/smtp.googlemail.com/smtp.gmail.com were accessable for all users (even those who are not allowed to use the internet).

      I have users who don't need authorisation (the management), some need to use the captive portal authorisation and most don't have the authorisation.

      I updated the pfsense software to the latest version. Additional packages I have installed are : squid, litesquid for reporting, cron

      I moved one client to the new email servers

      My client has problems as now sometimes the email client (both thunderbird as well as outlook) cannot connect to the imap server or to the smtp server or to both …
      I made sure the (from all to is selected in the captive portal allowed hosts) for the sites mentioned above.

      a quick test I did :
      I configured the allowed hosts on captive portal :
      www.google.com
      www.google.co.th

      those hosts should be accessible from all machines
      BUT
      when I type the address the login screen of captive portal is displayed instead of going to the website

      see copied list of allowed hosts

      Hostname Description

      smtp.gmail.com smtp server gmail  
        smtp.googlemail.com smtp to alternative google mail  
        imap.gmail.com imap on gmail server  
        imap.googlemail.com imap to alternative google mail servers  
      any  www.avast.com avast   
      any  mail.colbree.com colbree mailserver  
      any  smtp.totisp.net tot  
      any  www.no-ip.com update for dyn ip  
      any  www.whatismyip.com check external ip address  
      any  www.watchmyip.com    
      any  philiphome.dyndns.org    
      any  www.dyndns.com    
      any  auto.myip.is    
      www.no-ip.org    
      any  mail.colbreeasia.com local webmail  
      any  www.google.com google  
      pop.gmail.com pop mail for colbreeasia clients  
      pop.googlemail.com alternative pop for colbreeasia clients  
      any  mail.google.com mail via web  
      any  google.co.th th google

      remark : the little symbol (little green arrow) did not copy correct

      I postponed the transition of all mail clients from a host in the UK to google mail/apps as the users cannot access consistently their email on the google hosts.

      I tested this and consistently got the same error across all machines in the factory...

      version

      2.0-RC3 (i386)
      built on Thu Jul 7 22:58:43 EDT 2011

      Name pfsense.colbreeasia
      Version 2.0-RC3 (i386)
      built on Thu Jul 7 22:58:43 EDT 2011

      Update available. Click Here to view update.
      Platform pfSense
      CPU Type Intel(R) Pentium(R) D CPU 2.80GHz
      Uptime
      Current date/time
      Sat Jul 9 11:02:57 ICT 2011
      DNS server(s) 203.113.7.130
      203.113.5.130
      Last config change Sat Jul 9 10:38:46 ICT 2011
      State table size
      Show states
      MBUF Usage 131 /1155


      please help
      a committed user of pfsense

      1 Reply Last reply Reply Quote 0
      • C
        cmb
        last edited by

        Allowed hosts is not suitable for sites such as Google that frequently resolve to different IPs and return multiple IPs. What happens is the firewall will get one set of IPs when it does its lookup to allow those IPs, and then all the clients are likely to get completely different IPs when they do the same lookup. It was created specifically for scenarios where the hostname resolves to one IP and doesn't frequently change.

        1 Reply Last reply Reply Quote 0
        • C
          clevenp
          last edited by

          When this is the case, services like google apps (including email (or microsoft 365 services) in combination with pfsense is not possible …
          it kills the whole idea of using open source or free services with services in the cloud and the use of pfsense where users are sitting behind the portal (as they are restricted in accessing the internet)

          another example is the update of avast ... avast also has a set of IP addresses that are rotating .... I think most large companies offering on line services are using rotating IP addresses (rotating, load balancing, traffic dependent routing etc)

          not sure if this answer is satisfactory ...

          Without the portal, it also means pfsense is not able to work correctly? as the name is resolved after it goes through the portal or is the portal trying to resolve the IP address? I am not sure but I do not understand the answer given as it does not make a lot of design sense

          I thought (logically)
          the portal checks the host name against a table and allows/disallows the passing through without authorisation,
          then pfsense main engine will pick up the URL and resolve the address (after it passes through the portal)
          for each user it is possible the same URL can give a different address (that is why I have the host name in squid white list so it is NOT cached)

          if the solution of allowed hostnames is only resolving to 1 IP address then there is no need to use it as the allowed IP address tab is sufficient ... the pfsense admin just needs to look up the IP address of that particular site ....again the answer does not make any sense or the "allow hosts name" does not make any sense

          if the allow host tab in captive portal only can be used for sites with 1 IP address then the solution becomes very difficult to maintain for pfsense admins who are now considering using google mail/apps of Microsoft 365 services for small companies as pfsense cannot support this (same applies for for updates of virus scanners such as avast)

          Please confirm the answer is the only answer or consider the problem as a feature request/bug please

          Philip Van Cleven

          PS I was just considering to move school in Thailand to Google apps in conjunction with pfsense .... about 200 PC and about 1000 users (administration, teachers and students, each with different access rights)

          PS 2
          I requested this in February and I got this email from pfsense

          ermal
          Administrator
          Hero Member

          Offline

          Posts: 2182

          Re: gmail as domain email server and captive portal
          « Reply #1 on: February 21, 2011, 07:44:46 am »
          Quote
          Now you can allow by dns name access in CP.
          This is on 2.0 latest snapshots.

          you can find this when you google

          gmail as domain email server and captive portal

          « on: February 19, 2011, 07:01:02 am »

          1 Reply Last reply Reply Quote 0
          • C
            cmb
            last edited by

            It does what it was designed to do, no it doesn't meet every requirement but it wasn't designed to do so.

            1 Reply Last reply Reply Quote 0
            • C
              clevenp
              last edited by

              :(
              thanks for the response

              it seems that the 2 admins are not talking the same language
              it also seems clear from your response that the solution will not work and it is not considered as a bug ?
              I worked many years in the software/telco industry and when I launched my request in Feb I explicitely asked if the system would support the setup as described (with google hosting the email services)
              I lost 4 months and now I will loose a lot of goodwill at my clients when i will tell them the solution (I praised pfsense Open Source group for fast response on bugs and requests) is not going to work as the admins are giving 2 different answers and are now saying it will not work at all….

              by the way ... as I said again

              service providers such as AVAST and others (including Google) have load balancing around the globe and the IP addresses are changing when you initially contact them but to my knowledge no service will change the IP address within an open session. The argument given is a strange as the only thing the portal needs to do is to monitor the host name and to my knowledge it is the firewall engine that will do the DNS lookup (once it cleared the portal).
              If I am wrong then maybe there is a design flaw as portal and firewall are 2 different beasts (one is doing authorisation and host list/Ip list verification from the lists whereas the firewall is doing all the rules once the requests passed the portal. The portal is not an essential part of the firewall but an addon with the functionality described as above ....
              Maybe I am wrong as a software architect ...
              if the portal is doing DNS lookups I still don't know why "www.google.com" as an URL will not work when it is in the whitelist as that is a simple enough URL or are you telling me (and the board) that www.google.com never can work in a pfsense environment with a captive portal even if the URL is in the whitelist (allowed host list)?
              what about wikipedia (does not work either although it is in my whitelist)... In the school I want to give the students all access to wikipedia but only grade 6 has access to the internet ...

              Please confirm it is a feature that is not in the system now (as described above) and will not be implemented in this release and there is no timeline for this feature? or it is a bug that will be solved ?

              Philip

              1 Reply Last reply Reply Quote 0
              • C
                cmb
                last edited by

                It's a limitation of the existing implementation design and is not a bug. We have no plans of adding support for such a scenario at this time. Most of the work here happens because someone is funding it for some purpose, and that purpose may not always suit every possible imagined desired usage. Usually we devote more time to ensure it's as widely usable as possible, to the extent we lose money or at best break even on most all open source development projects. But in this case it would have been 10 times the work, and we simply couldn't afford to dedicate that to it. It's no different from FQDNs in firewall aliases (and uses the same back end basically), aliases can't be reliably used with FQDNs that resolve to a different IP every time you query them. But they work great for the majority of usage cases for that functionality.

                1 Reply Last reply Reply Quote 0
                • C
                  clevenp
                  last edited by

                  Dear CMB

                  Thanks for your reply.

                  After a review of your answer, I will need to find a work around or an alternative for pfsense to make it work.

                  it looks pfsense cannot work reliably in an environment where the capitve portal service is used and where cloud services (using loadbalancing mechanisms across multiple sites  (therefor they cannot use a virtual IP address)) are required from users who are not authorised to access the internet except for those sites that are defined.

                  It cannot resolve addresses that are used under global URL format such as mail.google.com / www.google.com / smtp.gmail.com etc

                  Thanks again for your very informative answer and it is great to hear how committed the pfsense team is.

                  Philip

                  1 Reply Last reply Reply Quote 0
                  • N
                    Nachtfalke
                    last edited by

                    @clevenp

                    Perhaps this is a possible solution for you:
                    You could try squid and squidguard and allow all sites with squidguard and deny all others which should not to be allowed.

                    I am using this on an extra interface/VLAN and just allow sites for updating windows, kaspersky, adobe, java.

                    1 Reply Last reply Reply Quote 0
                    • C
                      clevenp
                      last edited by

                      Thanks Nachtfalke

                      I will check later this week how to use Squidguard and try it out
                      I already have Squid and lightsquid running

                      Philip

                      1 Reply Last reply Reply Quote 0
                      • C
                        clevenp
                        last edited by

                        I found the solution

                        followed some advice from another board from Untangle : drop the portal and program the firewall to allow smtps, imaps, pop3s and dns for everyone.
                        Block in the proxy everything except those who are registered (with fixed IP)

                        downloaded the latest pfsense RC3 version and ran into trouble with stability and when it was stable it did not do what I expected : outgoing traffic was still blocked (IMAP.googlemail.com could not be resolved)

                        at the end I downloaded the community supported Endian firewall and guess what

                        1. the ports for googlemail etc work perfect … I got everyone blocked but mail is arriving and can be sent (just had to add port 465 in the allowed ssl ports). The settings are base settings for the firewall and some of them are already in place (good as example too!)
                        2. enabled the transparant proxy
                          created the rules for those IP who could not go to the internet (just deny of 80, 8080) --- worked perfect and they still enjoy their email
                          created the rules for the others to allow everything  -- worked perfect (my dhcp users are transparant (so I can still log and make sure they cannot go to certain sites) and they can access the internet)

                        sorry developers of Pfsense ... I enjoyed the software for over 2 years but I was forced to look for a solution. I was shocked with the response I got when i posted my questions. The endian solution worked straight out of the box ...and it was pretty easy to learn how to make the rules ... I do have to admit that it takes a bit better hardware (a second generation P4 with 1 gig) but that is not a real issue to pick this up anywhere for less than 100$)

                        as I have multiple sites to support I will move all sites to the Endian firewall solution ...

                        Philip Van Cleven

                        sysadmin at various small factories in Thailand

                        1 Reply Last reply Reply Quote 0
                        • E
                          eri--
                          last edited by

                          Whatever makes you have a solution.

                          Be aware that in 3 posts you have expressed 3 different targets and what you have done with Endian can be done with pfSense.
                          For some things more work is needed in pfSense, especially third party packages, to configure though the end result is the same.

                          1 Reply Last reply Reply Quote 0
                          • C
                            clevenp
                            last edited by

                            a little reply

                            my objective was simple
                            everyone is  on a domain email hosted by google
                            some users have access to the internet
                            some users have access to the internet using a portal
                            some users have no access
                            all users have email using the googlemail settings (ssl in and out)

                            environment
                            fw + squid proxy + proxylite + portal
                            proxy in transparent mode
                            old P4 (early model) with 1 gig of ram and 40 gig HD

                            I tried the suggestion to use the new version (as I was still on 1.2.3) as that has white pages for the portal
                            but that information was not complete (it was well intended)

                            with rc2, it kind of worked but my users were complaining that often the email gave an error (unresolved address)
                            then I found out that the white list in the portal was not really meant for what I wanted as google uses multiple IP (and not 1 virtual IP)

                            then had a heated discussion with no result

                            downloaded rc3 (last saterday) to try again

                            then I tried to just open the ports for outgoing traffic for mail (again ssl google definition) while blocking http traffic
                            but that did not work as I got an error about dns
                            I opened port 53 to resolve dns problems but problem still happened

                            whatever I did, email was not going out or in

                            at the same time RC3 was giving me me grief (machine hung at random times)

                            I tried many different combinations but all failed on the basic problem : email coming in for all users even when they are not allowed to use the internet

                            it was a desperate step to even look at other Firewalls
                            maybe I did not configure the fw correctly but I used the outgoing rules to open ports 53,465,993 and 995 (DNS,SMTP,POP and IMAP for google)
                            and this for any IP on the lan network with as destination anywhere

                            when I tried the same with endian … endian was already preconfigured to receive email from those ports , the only ports I had to add were the dns and the smtp port .... and voila it worked
                            i had to change my requirements : no portal anymore (as the open source endian does not have the portal included)  but hardcoded profiles ... those who can and those who cannot go to the internet ... and all are monitored
                            I use the proxy in transparent mode 
                            blocked the proxy for any access from those users who cannot
                            allowed the proxy for hard coded Ip addresses and for the dhcp addresses xxx.xxx.xxx.xxx/28 (16 addresses)

                            sorry if I came over harsh but I did receive also a very direct (!) response from your collegue ....

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.