Dansguardian Bypass

bilbo

Bypass is working fine and with explicit proxy setup I can block ssl sites. The only problem with ssl sites is that the block/bypass page does not show up, just a blank page when the ssl site it blocked. No redirect to the blocked page. Is this typical when blocking ssl sites?

rjcrowder

@bilbo:

Bypass is working fine and with explicit proxy setup I can block ssl sites. The only problem with ssl sites is that the block/bypass page does not show up, just a blank page when the ssl site it blocked. No redirect to the blocked page. Is this typical when blocking ssl sites?

Honestly don't know… Since you cannot easily filter SSL sites (i.e. the content filtering doesn't work unless you do a "man in the middle" setup), I've opted to do a transparent redirect. I didn't want the hassle of having to setup an explicit proxy on all machines. I tried using a wpad file for a while, but I still ran into issues where some machines would not use it (i.e. they would need explicit proxy setting).

I accomplish basically the same thing by using dans only for http and then counting on the OpenDNS name servers to blacklist https (of course it also ads an additional layer of protection on http).

vernonspangler

rjcrowder & bilbo,

So to ensure that I am inline with what you accomplishing here. Please correct me if this feed is not that conversation.

I would like to regulate what my children are authorized to see by utilizing web filtering.

Web filtering is transparent so that all traffic from a the DHCP side is being filter while my other static IP Address is unfiltered. (Note that I can setup this with Dansguardian and Squid 2.7.9)
Within the filtering process the "Access is Denied" page would need a simple password text field/Submit button that visitors could get a temporary password that will allow them to view what they would like to see. (Your thoughts?)
What I am noticing is that network performances are dramatically being reduced when I setup everything up.

Please note that my current setup is as followed.

Modem –> PFSENSE (Firewall / Gateway / OpenVPN) --> Home Switch
PFSense is a two interface setup inline with the modem to switch (WAN/LAN)

Thanks for any assistance in advance.

rjcrowder

There are tons of options here… I can describe what I have setup.

If you happen to be interested in this config, I have a script that will take a vanilla pfSense install and create what I believe is the ultimate home filtering solution. It's fairly simple to setup and I have a readme that describes how to apply it to a base pfSense install.

Keep in mind that it is very focused on creating a home filtering solution - so... single LAN/WAN, Squid, Dansguardian, OpenDNS. The script will install in one of two "flavors". The first "flavor" leaves all of the pfSense menus and screens in place but pre-configures squid, DG, some IP alias groups, and firewall rules, DGLog2 for DG reporting, etc.. The second "flavor" takes it to another level by replacing some of the pfSense screens to simplify the static IP setup, adding simple screens for creating IP based time restrictions, and removing any of the pfSense menu items that are not necessary.

So... here is what I setup on my config.

1. Squid and DG in transparent mode. A firewall rule is used to redirect all HTTP traffic to DG. DG is configured to do both blacklist and phrase filtering and downloads the lists from shalla.
2. DG is configured for IP based authentication. At the moment, I don't do anything to put different IP ranges into different DG filtering groups, but it would be very easy to do.
3. OpenDNS is used as the DNS resolution service. This gives me blacklist based HTTPS filtering.
4. I've pre-created IP aliases for ranges to allow internet access rules within the firewall. MAC addresses can be assigned to specific IP's (or ranges if you use my modified screens) and the ranges are used in rules to do time based restriction.
5. IP alias ranges that are not filtered so that you can assign devices an unfiltered IP address (such as an XBox or ROKU).
6. I've created a DG bypass page that prompts for an ID/Password. The ID/Password is either a valid pfSense user ID/Password or an ID/Password combo from a text file. The text file can be edited on the DG logging screen. If a user enters a valid ID/Password, then their IP address is allowed to fully bypass the filter for approximately 15 minutes. The bypass is implemented by excluding the IP from the the firewall redirect rule.
7. Host overrides for Google to disable HTTPS search.
8. Layer 3 checks (using IPFW) to make sure that no one tries to hijack an unfiltered IP address (compares MAC to the static assignment).
9. DGLog2 for querying and reporting activity on the DG access logs. A menu item is added under pfSense but it is also accessible without logging into the DG web console.

I'll post some links to screenshots and the setup script in case you are interested...

rjcrowder

OK… I've attached a bunch of links.

The screenshots are all from what I call an "appliance" install. The appliance setup can be created by running the apply custom script in "appliance" mode (./apply_custom.sh -a). This is my attempt to create a pfSense install and user interface that is easy and usable as a home filtering solution but also nearly impossible to break. However, even with the stripped down web UI, all of the pfSense core and packages (including the original screens) is still installed and available - just hidden.

You can also run the script to create a "base" install (./apply_custom.sh -b). The base install sets up everything I described on the previous post but leaves almost all of the pfSense menus in place.

Install files (see readme.txt for instructions)…
https://dl.dropboxusercontent.com/u/55672566/install_files/readme.txt
https://dl.dropboxusercontent.com/u/55672566/install_files/apply_custom.sh
https://dl.dropboxusercontent.com/u/55672566/install_files/pfsense_custom.tar.gz
https://dl.dropboxusercontent.com/u/55672566/install_files/fetch_blacklist.sh

vernonspangler

rjcrowder

I tried both appliance and base on your install script. I did this on a solid state HDD and I am running into a few issues.
Actually 1 major issue. It installs with no issues but after installation I have no connectivity internal via pfsense web portal in order to make any adjustments, nor do i have access to the outside internet. I do however have the ability to ssh from my workstation to the pfsense.

WAN = DHCP from ISP
LAN = 192.168.20.1/24
DHCP 192.168.20.100 - 192.268.20.200

Please keep in mind I am doing this work on a demo system that is confirmed to work with pfsense in order to work out all the issues prior to applying this to my home network.

rjcrowder

OK… Hmmm... haven't seen that particular issue. Did you follow the readme exactly? You should do an install, set the IP address for the LAN, and then run the install... That's it.

The install script copies a config.xml over the /conf/config.xml that was originally created. It then replaces the IP address range with the range you had put in your original config.xml (in this case it sounds like you would have put 192.168.20.1/24. Your DHCP range will also be overwritten to be (in this case) 192.168.20.32 -> 192.168.20.95. The IP address of your pfsense server should (of course) be 192.168.20.1

I'd ssh to the box and check the IP addresses in /conf/config.xml. It should have created everything in the 20.x range. The other thing to check... do an "ipfw -x Dummy show" and make sure that all the ipfw rules are created for the 20.x range...

vernonspangler

@rjcrowder:

The install script copies a config.xml over the /conf/config.xml that was originally created. It then replaces the IP address range with the range you had put in your original config.xml (in this case it sounds like you would have put 192.168.20.1/24. Your DHCP range will also be overwritten to be (in this case) 192.168.20.32 -> 192.168.20.95. The IP address of your pfsense server should (of course) be 192.168.20.1

Okay so I am able to get it to work now. So I have a few more questions if you do not mind.

I am noticing that I have to authenticate twice in order to access a banned webpage. (I think this is just a browser issue)
I am getting a few errors that the unable to resolve name from browser (This might be me as I have this setup currently behind another FW/GW in order to stage a quick swap)
Why xxx.xxx.20.32 -> xxx.xxx.20.95 would it make a difference if I utilized xxx.xxx.2.254 for my pfsense?

As I review your setup, to understand what it is going to take for me if I so desire to change the pfsense ip address without locking me out again.

I would say that I would have to disable Dans and squid first.
Setup firewall rules and filters on the LAN to allow the new network scheme.
Adjust my DHCP server to new IP Scheme (ensure workstation has static IP of current IP scheme)
Change the LAN IP
Adjust my workstation to match new LAN IP scheme
Adjust Dans and Squid to new IP Scheme and start services back up.

(Ultimately it would be easier for me to setup with the IP scheme that I would to utilize as the pfsense LAN ID. Your thoughts?)

I am highly impressed with the filtering and scheduling of IP ranges that was going to be my next challenge after i figured out the web filtering.

Note:
You might want to update your readme.txt as the following.

1. Install pfsense 2.1 using normal USB memstick install (Currently being tested on HDD setup)

5. Copy the following files to the box
scp apply_custom.sh root@192.168.4.1:/root/.
scp pfsense_custom.tar.gz root@192.168.4.1:/root/.
scp pkg-install.php root@192.168.4.1:/root/.
scp fetch_blacklist.sh root@192.168.4.1:/root/.

Once again I would like express my high apprieciation for the work that you have compiled.

rjcrowder

First of all… you're welcome. Glad you got it working and I'm glad that someone else sees some value in it.

You can easily change the subnet that is used (say from 192.168.20.x to 192.168.2.x) by doing the following:
1.) Edit your current config.xml and change the "ipaddr" value under the lan interface.
2.) Re-run the "apply_custom.sh" script with the -i command line option

The script goes out and changes IP addresses that are kept in some other things I added. For example, addresses can be kept in the ipfw custom rules and they are also set in the DG bypass pages. However, the script only changes the subnet (not the last part of the IP address)... So, I modified it a little bit to also change the pfSense machine address. You can run the following script (save it as whatever.sh and make it executable). Be careful... 254 should work fine, but you don't want to change it to an address that is within one of the ranges that is being using for rules...

#!/bin/sh

#-----------------------------------------------------------------------
#-----------------------------------------------------------------------
update_ip() {
  config_file=/root/config.xml
  cp /cf/conf/config.xml $config_file

  # Prompt the user for the new lan ipv4 address and domain
  #
  echo 'Enter new LAN IP address of server (ex: 192.168.3.1):'
  read new_ip
  new_ip1=`echo $new_ip | cut -f1 -d"."`
  new_ip2=`echo $new_ip | cut -f2 -d"."`
  new_ip3=`echo $new_ip | cut -f3 -d"."`
  new_ip4=`echo $new_ip | cut -f4 -d"."`

  cfg_ip=`xmllint --xpath '/pfsense/interfaces/lan/ipaddr/text()' $config_file`
  cfg_ip1=`echo $cfg_ip | cut -f1 -d"."`
  cfg_ip2=`echo $cfg_ip | cut -f2 -d"."`
  cfg_ip3=`echo $cfg_ip | cut -f3 -d"."`
  cfg_ip4=`echo $cfg_ip | cut -f4 -d"."`

  cat $config_file | \
      sed -e "s/$cfg_ip1\.$cfg_ip2\.$cfg_ip3\.$cfg_ip4      sed -e "s/$cfg_ip1\.$cfg_ip2\.$cfg_ip3\./$new_ip1\.$new_ip2\.$new_ip3\./g" > \
      /cf/conf/config.xml

  pfs_ip=$cfg_ip
  pfs_ip1=`echo $pfs_ip | cut -f1 -d"."`
  pfs_ip2=`echo $pfs_ip | cut -f2 -d"."`
  pfs_ip3=`echo $pfs_ip | cut -f3 -d"."`
  pfs_ip4=`echo $pfs_ip | cut -f4 -d"."`

  # Update some other config files with the proper IP address
  #
  cfg_ip=`cat /usr/local/ipfw_custom_rules/checked_ranges.conf | grep -v "^#" | head -1 | cut -f1 -d'/'`
  update_lan_ip /usr/local/ipfw_custom_rules/checked_ranges.conf $cfg_ip $new_ip

  cfg_ip=`cat /usr/local/ipfw_custom_rules/macip_additions.conf | grep -v "^#" | head -1 | awk '{ print $2 }'`
  update_lan_ip /usr/local/ipfw_custom_rules/macip_additions.conf $cfg_ip $new_ip

  cfg_ip=`fgrep 'action="http' /usr/local/www/dgbypass/accessdenied.php | cut -f2 -d"=" | cut -f3 -d'/'`
  update_lan_ip /usr/local/www/dgbypass/accessdenied.php $cfg_ip $new_ip

  cfg_ip=`cat /usr/local/www/dgbypass/unfiltered | head -1 | cut -f1 -d'/'`
  update_lan_ip /usr/local/www/dgbypass/unfiltered $cfg_ip $new_ip

  cfg_ip=`cat /usr/local/dgbypass/gold_unfiltered | head -1 | cut -f1 -d'/'`
  update_lan_ip /usr/local/dgbypass/gold_unfiltered $cfg_ip $new_ip

  cfg_ip=`fgrep 'src="http' /usr/local/www/content_filter_logs.php | cut -f2 -d"=" | cut -f3 -d'/'`
  update_lan_ip /usr/local/www/content_filter_logs.php $cfg_ip $new_ip
}

#-----------------------------------------------------------------------
#-----------------------------------------------------------------------
update_lan_ip() {
  in_file=$1
  if [ "$2" != "" ]; then
    repl_ip1=`echo $2 | cut -f1 -d"."`
    repl_ip2=`echo $2 | cut -f2 -d"."`
    repl_ip3=`echo $2 | cut -f3 -d"."`
    new_ip1=`echo $3 | cut -f1 -d"."`
    new_ip2=`echo $3 | cut -f2 -d"."`
    new_ip3=`echo $3 | cut -f3 -d"."`
    new_ip4=`echo $3 | cut -f4 -d"."`
    mv $in_file $in_file.orig
    cat $in_file.orig | \
        sed -e "s/$pfs_ip1\.$pfs_ip2\.$pfs_ip3\.$pfs_ip4/$new_ip1\.$new_ip2\.$new_ip3\.$new_ip4/g" | \
        sed -e "s/$repl_ip1\.$repl_ip2\.$repl_ip3\./$new_ip1\.$new_ip2\.$new_ip3\./g" > \
        $in_file
  fi 
}

#-----------------------------------------------------------------------
# Main
#-----------------------------------------------------------------------

echo "Changing IP address and subnet of pfSense"

update_ip
/etc/rc.reboot

To your other question… there's no particular reason for the 32-95 range for dynamically assigned addresses other than the fact that it falls on maskable boundaries... You could change it to whatever you want as long as it doesn't conflict with ranges being used for other rules.

rjcrowder

OK… Updated the readme.txt file. Also changed the "-i" option of "apply_custom.sh" so that it will prompt you for the IP address to set the server to... it will also validate that the IP address is in a valid range (i.e. won't conflict with any of the pre-configured rules or the dynamically assigned range).

vernonspangler

This is great and helps out a ton. I was wondering if you have a script that setup for your post http://forum.pfsense.org/index.php/topic,68872.msg377435.html#msg377435

While I do appreciate this highly intense setup. Personally I do not need all the subnet rules.

My network setup is as followed and I am sure that others have something similar.

pfSense/GW/FW: LAN IP address = 192.168.x.x Subnet = 255.255.255.0
pfSense DHCP Range: 192.168.x.x - 192.168.x.x (whatever they feel that they want to issues out with the worries of subnets.)

The web filtering applies to the subnet range that applies to the interface you wish to filter… i.e. LAN, OPT.
From there a person could setup up there own aliases with a network range and apply FW rules and schedules.

Feel free to shot me a call anytime as I am off for the holidays and would enjoy speaking with you more on this...

;D ;D ;D ;D ;D ;D ;D ;D ;D

rjcrowder

At the moment, I don't have a script that only sets up the IP based dansguardian bypass. I could probably create one, but just haven't done it yet.

The script I have (in base/appliance mode) was created for several reasons. For the appliance setup, I was trying to come up with something that was a simple (hence getting the "alias groups", removal of DHCP page, replacement of the rules page, removal of most of the screens, menus, etc.) yet very solid filtering solution. My goal was to come up with something that was based solely on open source software but offered the ultimate in filtering for a home user who had a fairly standard config that was Modem <–-> pfSense <---> AccessPoint. My target was something similar to the functionality of this http://pandorashope.com/ commercial product but with a separate firewall/access point (and no yearly subscription).

The other mode of install - what I called "base" - is nothing other than what I use... It's accomplishes the same thing as the "appliance" install, but you have to know what you are doing...

There are a few limitations to what I'm adding on top of pfSense. Probably the biggest one is that I'm messing with the captive portal functionality by adding my own IPFW firewall rules (see the directory /usr/local/ipfw_custom_rules). In theory, the captive portal should still work if you change the value of "skip_captiveportal_rules" to be "false" (in my script), but I don't use the captive portal and therefore haven't tested it. The DHCP page has also been modified to add my custom rules every time you save - since the rules should be re-created every time you add a static address assignment.

I did one thing on the "appliance" setup that I would have preferred avoiding, but it made life easier. I added an element called "ipalias" into the xml for a static mapping. It just made it easier to track what alias group an IP was assigned to. If you happen to edit the static mapping with the default DHCP edit page, you will lose the ipalias value because it obviously doesn't save it...

Anyway... there are several other "features" that I added. If you want an explanation of how (or why) any of them are created the way they are - just let me know. I'd be happy to discuss.

Finally, I've thought about trying to distribute this somehow. I'm not really looking to make money, more as a ministry - I think it would be a great service to families if it could be made simple enough to setup. Certainly open to any ideas or help!

mrnadj

rjcrowder & all,

I could really use some help.
I have installed the script with the -b option, and it almost works.

My configuration is:
WAN = 192.168.1.100 /24 + ( GW: 192.168.1.1=adsl )
LAN = 192.168.1.2 /24 + (No Gateway) + (dhcpServer 192.168.1.34 - 192.168.1.64)

I had to create a new rule in NAT:
: -> *:80 redirect-> 192.168.1.2:8080
basically any traffic, redirect to DANS on port 8080

I have tested on a VirtualBox Client, and it gets a DHCP assigned address, with the 192.168.1.2 as gateway,
and it all seems to work. Its requests are even logged by Dans.

But when I test with a Actual Laptop, it gets the DHCP assigned address, with the 192.168.1.2 as gateway,
but no HTTP Traffic seems to work, and its requests are not logged by Dans.

I have tried apply_custom.sh -i, but it did not fix anything.
I have also tried adding PASS rules to firewall and disabling any Block rules, still no go.

As I understand it, this should only require like 1 NAT Rule, and maybe 1 LAN FW Rule, 1 WAN FW Rule?

rjcrowder

From what you describe, I can't figure what is wrong. Let me make a couple of statements (that might be helpful) and throw out one potential issue that you can check…

1.) The install script I created copies a config.xml in place. That config.xml sets the DHCP assigned range based on the fact that other ranges of addresses are used for specific devices. It sounds like you stuck with the address setup that I had configured so that's good. The config.xml had some pre-created rules that redirected to dans (port 8080) so you should not have needed to create your own rule. However, I don't know that I ever tried it with an address other than a ".1" for the gateway (think you did ".2").

2. One of the other modification I did was turning on the captive portal to enable the IPFW firewall. It then creates rules to skip the normal portal rules and check certain address ranges to make sure that no one is "hijacking" mac addresses. This is particularly important for the address ranges that are unfiltered - you don't want someone to get around the filter by manually setting their IP to one that is unfiltered. One of the things I've discovered is that there are situations where pfSense will see multiple MAC addresses for the same IP!!! The two things that I know can cause this to occur are a wireless access point configured as a wireless adapter (such as for an xbox) and a virtual machine running on a host with its own MAC address. I made a way to add additional "valid" MAC's for the same IP by adding them to a file that is used when the IPFW rules are created. The file is manually editable under /usr/local/ipfw_macip directory. In a newer version of my modifications (not posted on dropbox yet), I also added it to the DHCP screen.

You can easily check to see if traffic is hitting the box for a MAC and also see if it is being blocked in IPFW by looking at the IPFW rule listing. I believe the command is "ipfw -x Dummy list" (don't remember for sure... might be "show" instead of list).

Hope that's helpful...

series_rp

i found this problem too… thank for shareing...guy :D

mrnadj

rjcrowder ,

Thank you again for your guide, and assistance!

I thought this was impossible for me, after already spending about 5 days on it.

I gave it another go today, I was able to get Firefox, with manual HTTP Proxy to connect to Dans on 8080.

Once I had that working I set the Laptop to use PFS as Gateway.
With all the firewall rules deleted, it simply forwarded all traffic directed at the gateway.
I could see this on the console, option 10), pf Logs

I then added a Reject All traffic rule
Then I added a NAT rule for port 80.
Then a FW Allow rule for 443.

It now works well. Thank you rjcrowder. I am a programmer, and even I found this beyond challenging!

rjcrowder

Hmmm… sorry about that. I intended these scripts to be an easy way to setup a very specific configuration. I've never had any issue as long as I've stuck strictly to the intended use case. The downside of that approach is that I haven't tried a lot of variations (multiple gateway boxes, different gateway addresses, etc.) and I'm sure there are multiple ways it could be broken. However, if you can pin down issues with the install process or instructions (or give me enough info that I can find them) I'd love to know what they were so that I can try to fix them.

I'm a software guy by trade as well. What I've learned about networking has been purely by playing with stuff like this. Nice to see someone else branching out...