Captive portal and allowed hosts problem

clevenp

Hello

I have been using pfsense 2.0 for some time now and occassionally i updated the software (about once per month).

recently I had to move my email servers to the google servers and I requested what to do and a new version came out with "allowed hosts: in the captive portal section
I was able to configure this for the users who had no login on the captive portal so the machines were updated able to go to some sites (e.g. the avast servers)

now I wanted to move all my clients to google mail and I needed to make sure the imap.googlemail.com/imap.gmail.com/pop.googlemail.com/pop.gmail.com/smtp.googlemail.com/smtp.gmail.com were accessable for all users (even those who are not allowed to use the internet).

I have users who don't need authorisation (the management), some need to use the captive portal authorisation and most don't have the authorisation.

I updated the pfsense software to the latest version. Additional packages I have installed are : squid, litesquid for reporting, cron

I moved one client to the new email servers

My client has problems as now sometimes the email client (both thunderbird as well as outlook) cannot connect to the imap server or to the smtp server or to both …
I made sure the (from all to is selected in the captive portal allowed hosts) for the sites mentioned above.

a quick test I did :
I configured the allowed hosts on captive portal :
www.google.com
www.google.co.th

those hosts should be accessible from all machines
BUT
when I type the address the login screen of captive portal is displayed instead of going to the website

see copied list of allowed hosts

Hostname Description

smtp.gmail.com smtp server gmail
smtp.googlemail.com smtp to alternative google mail
imap.gmail.com imap on gmail server
imap.googlemail.com imap to alternative google mail servers
any www.avast.com avast
any mail.colbree.com colbree mailserver
any smtp.totisp.net tot
any www.no-ip.com update for dyn ip
any www.whatismyip.com check external ip address
any www.watchmyip.com
any philiphome.dyndns.org
any www.dyndns.com
any auto.myip.is
www.no-ip.org
any mail.colbreeasia.com local webmail
any www.google.com google
pop.gmail.com pop mail for colbreeasia clients
pop.googlemail.com alternative pop for colbreeasia clients
any mail.google.com mail via web
any google.co.th th google

remark : the little symbol (little green arrow) did not copy correct

I postponed the transition of all mail clients from a host in the UK to google mail/apps as the users cannot access consistently their email on the google hosts.

I tested this and consistently got the same error across all machines in the factory...

version

2.0-RC3 (i386)
built on Thu Jul 7 22:58:43 EDT 2011

Name pfsense.colbreeasia
Version 2.0-RC3 (i386)
built on Thu Jul 7 22:58:43 EDT 2011

Update available. Click Here to view update.
Platform pfSense
CPU Type Intel(R) Pentium(R) D CPU 2.80GHz
Uptime
Current date/time
Sat Jul 9 11:02:57 ICT 2011
DNS server(s) 203.113.7.130
203.113.5.130
Last config change Sat Jul 9 10:38:46 ICT 2011
State table size
Show states
MBUF Usage 131 /1155

please help
a committed user of pfsense

cmb

Allowed hosts is not suitable for sites such as Google that frequently resolve to different IPs and return multiple IPs. What happens is the firewall will get one set of IPs when it does its lookup to allow those IPs, and then all the clients are likely to get completely different IPs when they do the same lookup. It was created specifically for scenarios where the hostname resolves to one IP and doesn't frequently change.

clevenp

When this is the case, services like google apps (including email (or microsoft 365 services) in combination with pfsense is not possible …
it kills the whole idea of using open source or free services with services in the cloud and the use of pfsense where users are sitting behind the portal (as they are restricted in accessing the internet)

another example is the update of avast ... avast also has a set of IP addresses that are rotating .... I think most large companies offering on line services are using rotating IP addresses (rotating, load balancing, traffic dependent routing etc)

not sure if this answer is satisfactory ...

Without the portal, it also means pfsense is not able to work correctly? as the name is resolved after it goes through the portal or is the portal trying to resolve the IP address? I am not sure but I do not understand the answer given as it does not make a lot of design sense

I thought (logically)
the portal checks the host name against a table and allows/disallows the passing through without authorisation,
then pfsense main engine will pick up the URL and resolve the address (after it passes through the portal)
for each user it is possible the same URL can give a different address (that is why I have the host name in squid white list so it is NOT cached)

if the solution of allowed hostnames is only resolving to 1 IP address then there is no need to use it as the allowed IP address tab is sufficient ... the pfsense admin just needs to look up the IP address of that particular site ....again the answer does not make any sense or the "allow hosts name" does not make any sense

if the allow host tab in captive portal only can be used for sites with 1 IP address then the solution becomes very difficult to maintain for pfsense admins who are now considering using google mail/apps of Microsoft 365 services for small companies as pfsense cannot support this (same applies for for updates of virus scanners such as avast)

Please confirm the answer is the only answer or consider the problem as a feature request/bug please

Philip Van Cleven

PS I was just considering to move school in Thailand to Google apps in conjunction with pfsense .... about 200 PC and about 1000 users (administration, teachers and students, each with different access rights)

PS 2
I requested this in February and I got this email from pfsense

ermal
Administrator
Hero Member

Offline

Posts: 2182

Re: gmail as domain email server and captive portal
« Reply #1 on: February 21, 2011, 07:44:46 am »
Quote
Now you can allow by dns name access in CP.
This is on 2.0 latest snapshots.

you can find this when you google

gmail as domain email server and captive portal

« on: February 19, 2011, 07:01:02 am »

cmb

It does what it was designed to do, no it doesn't meet every requirement but it wasn't designed to do so.

clevenp

:(
thanks for the response

it seems that the 2 admins are not talking the same language
it also seems clear from your response that the solution will not work and it is not considered as a bug ?
I worked many years in the software/telco industry and when I launched my request in Feb I explicitely asked if the system would support the setup as described (with google hosting the email services)
I lost 4 months and now I will loose a lot of goodwill at my clients when i will tell them the solution (I praised pfsense Open Source group for fast response on bugs and requests) is not going to work as the admins are giving 2 different answers and are now saying it will not work at all….

by the way ... as I said again

service providers such as AVAST and others (including Google) have load balancing around the globe and the IP addresses are changing when you initially contact them but to my knowledge no service will change the IP address within an open session. The argument given is a strange as the only thing the portal needs to do is to monitor the host name and to my knowledge it is the firewall engine that will do the DNS lookup (once it cleared the portal).
If I am wrong then maybe there is a design flaw as portal and firewall are 2 different beasts (one is doing authorisation and host list/Ip list verification from the lists whereas the firewall is doing all the rules once the requests passed the portal. The portal is not an essential part of the firewall but an addon with the functionality described as above ....
Maybe I am wrong as a software architect ...
if the portal is doing DNS lookups I still don't know why "www.google.com" as an URL will not work when it is in the whitelist as that is a simple enough URL or are you telling me (and the board) that www.google.com never can work in a pfsense environment with a captive portal even if the URL is in the whitelist (allowed host list)?
what about wikipedia (does not work either although it is in my whitelist)... In the school I want to give the students all access to wikipedia but only grade 6 has access to the internet ...

Please confirm it is a feature that is not in the system now (as described above) and will not be implemented in this release and there is no timeline for this feature? or it is a bug that will be solved ?

Philip

cmb

It's a limitation of the existing implementation design and is not a bug. We have no plans of adding support for such a scenario at this time. Most of the work here happens because someone is funding it for some purpose, and that purpose may not always suit every possible imagined desired usage. Usually we devote more time to ensure it's as widely usable as possible, to the extent we lose money or at best break even on most all open source development projects. But in this case it would have been 10 times the work, and we simply couldn't afford to dedicate that to it. It's no different from FQDNs in firewall aliases (and uses the same back end basically), aliases can't be reliably used with FQDNs that resolve to a different IP every time you query them. But they work great for the majority of usage cases for that functionality.

clevenp

Dear CMB

Thanks for your reply.

After a review of your answer, I will need to find a work around or an alternative for pfsense to make it work.

it looks pfsense cannot work reliably in an environment where the capitve portal service is used and where cloud services (using loadbalancing mechanisms across multiple sites (therefor they cannot use a virtual IP address)) are required from users who are not authorised to access the internet except for those sites that are defined.

It cannot resolve addresses that are used under global URL format such as mail.google.com / www.google.com / smtp.gmail.com etc

Thanks again for your very informative answer and it is great to hear how committed the pfsense team is.

Philip

Nachtfalke

@clevenp

Perhaps this is a possible solution for you:
You could try squid and squidguard and allow all sites with squidguard and deny all others which should not to be allowed.

I am using this on an extra interface/VLAN and just allow sites for updating windows, kaspersky, adobe, java.

clevenp

Thanks Nachtfalke

I will check later this week how to use Squidguard and try it out
I already have Squid and lightsquid running

Philip

clevenp

I found the solution

followed some advice from another board from Untangle : drop the portal and program the firewall to allow smtps, imaps, pop3s and dns for everyone.
Block in the proxy everything except those who are registered (with fixed IP)

downloaded the latest pfsense RC3 version and ran into trouble with stability and when it was stable it did not do what I expected : outgoing traffic was still blocked (IMAP.googlemail.com could not be resolved)

at the end I downloaded the community supported Endian firewall and guess what

the ports for googlemail etc work perfect … I got everyone blocked but mail is arriving and can be sent (just had to add port 465 in the allowed ssl ports). The settings are base settings for the firewall and some of them are already in place (good as example too!)
enabled the transparant proxy
created the rules for those IP who could not go to the internet (just deny of 80, 8080) --- worked perfect and they still enjoy their email
created the rules for the others to allow everything -- worked perfect (my dhcp users are transparant (so I can still log and make sure they cannot go to certain sites) and they can access the internet)

sorry developers of Pfsense ... I enjoyed the software for over 2 years but I was forced to look for a solution. I was shocked with the response I got when i posted my questions. The endian solution worked straight out of the box ...and it was pretty easy to learn how to make the rules ... I do have to admit that it takes a bit better hardware (a second generation P4 with 1 gig) but that is not a real issue to pick this up anywhere for less than 100$)

as I have multiple sites to support I will move all sites to the Endian firewall solution ...

Philip Van Cleven

sysadmin at various small factories in Thailand

eri--

Whatever makes you have a solution.

Be aware that in 3 posts you have expressed 3 different targets and what you have done with Endian can be done with pfSense.
For some things more work is needed in pfSense, especially third party packages, to configure though the end result is the same.

clevenp

a little reply

my objective was simple
everyone is on a domain email hosted by google
some users have access to the internet
some users have access to the internet using a portal
some users have no access
all users have email using the googlemail settings (ssl in and out)

environment
fw + squid proxy + proxylite + portal
proxy in transparent mode
old P4 (early model) with 1 gig of ram and 40 gig HD

I tried the suggestion to use the new version (as I was still on 1.2.3) as that has white pages for the portal
but that information was not complete (it was well intended)

with rc2, it kind of worked but my users were complaining that often the email gave an error (unresolved address)
then I found out that the white list in the portal was not really meant for what I wanted as google uses multiple IP (and not 1 virtual IP)

then had a heated discussion with no result

downloaded rc3 (last saterday) to try again

then I tried to just open the ports for outgoing traffic for mail (again ssl google definition) while blocking http traffic
but that did not work as I got an error about dns
I opened port 53 to resolve dns problems but problem still happened

whatever I did, email was not going out or in

at the same time RC3 was giving me me grief (machine hung at random times)

I tried many different combinations but all failed on the basic problem : email coming in for all users even when they are not allowed to use the internet

it was a desperate step to even look at other Firewalls
maybe I did not configure the fw correctly but I used the outgoing rules to open ports 53,465,993 and 995 (DNS,SMTP,POP and IMAP for google)
and this for any IP on the lan network with as destination anywhere

when I tried the same with endian … endian was already preconfigured to receive email from those ports , the only ports I had to add were the dns and the smtp port .... and voila it worked
i had to change my requirements : no portal anymore (as the open source endian does not have the portal included) but hardcoded profiles ... those who can and those who cannot go to the internet ... and all are monitored
I use the proxy in transparent mode
blocked the proxy for any access from those users who cannot
allowed the proxy for hard coded Ip addresses and for the dhcp addresses xxx.xxx.xxx.xxx/28 (16 addresses)

sorry if I came over harsh but I did receive also a very direct (!) response from your collegue ....