Policy routing squid3 tcp marked packets to VPN using firewall rules

[kesawi]

I currently use firewall rules on my LAN interface for policy routing traffic on my WAN and VPN interfaces for various internal clients and external destinations. This works well for normal traffic.

With the squid3 proxy on pfSense the traffic appears to originate from pfSense itself, rather than the LAN subnet, and therefore the LAN interface firewall rules don't apply. To work around this I've added tcp_outgoing_address configuration directive to the Custom ACLs (before auth) in my proxy configuration and specified the IP of my VPN interface.

The problem I'm encountering is that my VPN provider often changes the assigned IP address when I reconnect the OpenVPN client (seems to randomly switch between two addresses). Adding a second tcp_outgoing_address for the alternate IP doesn't work as squid only processes the first ACL it encounters, even if the VPN interface IP is doesn't existing.

Squid does have a tcp_outgoing_mark configuration directive which may possibly allow for policy based routing through the firewall rules, independent of the current VPN interface IP.

Does the packet marking applied by squid conform to the requirements of the packet matching in the firewall rules?

How would I need to configure the firewall rules to direct this marked traffic through the VPN interface as it is originating from the firewall itself rather than one of the private subnets? Would this be a outgoing WAN rule with packet matching and VPN gateway specified, or a floating rule of some kind?

[heper]

have you tried setting the outgoing address to 127.0.0.1 & using floating rules (quick/out)?

this might work, testing required

[kesawi]

If I set the outgoing address to 127.0.0.1 I'm assuming it will end up on the lo interface. How would I assign a floating rule as there's no localhost option in the interface when creating a rule?

[heper]

localhost will try to go out the default gateway.

if you can catch it before it goes out "default_wan" and force out the vpn_gateway, you might get away with it. (its a pain to get it working)

[kesawi]

How would I configure the firewall rule? I don't want to create a specific rule for each destination address, as I want to route squid based on the dstdomain acl type, and maintaining an alias list would be an administrative overhead. Would the source address be 127.0.0.1 or the WAN interface IP? If I use either of these addresses without a destination, will that have other unintended consequences (even if I just limit the destination ports to 80 and 443)?

[heper]

its been too long since i've bothered with squid & have no intention of going that way again.

-start by not specifying an interface or source, direction out, dest_port 80, turn on logging.
-see if you can get it todo what you want with policy routing
-trigger the rule with squid and see what the source address is & add it to the fw rule to avoid unintended behaviour.

it won't work the first time, it probably won't work the second time … if you keep going at it for long enough, there is a small chance of getting it right ;)

[kesawi]

I've come up with a concept of how I could possible get squid to direct traffic via the VPN with a dynamic IP but I'm not sure how to implement it in pfSense:

• Create an additional localhost interface (say lo1 with IP 127.0.0.2)

• Redirect traffic from the new lo1 out through my VPN interface ovpnc1

• Change the squid tcp_outgoing_address configuration directive to point to the new local host address 127.0.0.2

In theory this should resolve my issue with a dynamic IP address for the VPN interface as squid will only have the  static IP of the new localhost interface to contend with. How do I undertake steps 1 & 2?

[sjen5]

Did you get this to work.  I have been trying to do that same thing.

The following works fine;

• set tcp_outgoing_address to the dynamic IP assigned by your VPN provider using acls on certain domains

acl TEST dstdomain .ifconfig.co;
acl TEST dstdomain .whatismyipaddress.com;
tcp_outgoing_address 10.x.x.x   TEST;

The above works, however would stop working every time i get a new IP address from the VPN provider 10.x.x.x.  So I tried the following based on various posts around the internet, however I cannot get it to work.

• add a VIP to localhost of 127.0.0.2
• set tcp_outgoing_address to the VIP using acls on certain domains

acl TEST dstdomain .ifconfig.co;
acl TEST dstdomain .whatismyipaddress.com;
tcp_outgoing_address 127.0.0.2   TEST;

At this point in my configuration neither of the above domains works, I believe because there is no route to the internet for 127.0.0.2 to the internet.  As you can see there is a VIP for 127.0.0.2

[2.2.2-RELEASE][…..................]/home/…...............: ping 127.0.0.2
PING 127.0.0.2 (127.0.0.2): 56 data bytes
64 bytes from 127.0.0.2: icmp_seq=0 ttl=64 time=0.592 ms
64 bytes from 127.0.0.2: icmp_seq=1 ttl=64 time=0.442 ms

So I thought it may be because there is no outgoing nat rule in my case for 127.0.0.2 port 80.  So I add a nat rule like below, at this point routing just to WAN.

WAN 127.0.0.2/32 * * * WAN address * NO 127.0.0.2 to WAN

Now whatismyipaddress.com works fine, it's routed out of my WAN.  I want it to go out the VPN, so I modify the nat above to the following;

VPN_WAN  127.0.0.2/32 * * * VPN_WAN address * NO 127.0.0.2 to VPN

Again, page wont load, squid error (49) Can't assign requested address

So now I thought that perhaps its because the traffic isnt actually being routed to this gateway, so i created a floating firewall rule;  This should forward all TCP packets for source 127.0.0.2 out the VPN_WAN

IPv4 TCP 127.0.0.2 * * * VPN_WAN none 127.0.0.2

Unfortunately at this point rather than squid returning an error, the page load just times out, it seems that the packet forwarding inside pfsense is getting a bit lost :(  There are no obvious entries in the firewall log that would indicate an issue.  There is an entry however which has source 127.0.0.2 as expected and a destination of 104.71.204.14:80 (http://whatismyipaddress.com/) as expected, however the interface is still WAN.  It would appear that the floating rule is not forwarding out the VPN_WAN

any ideas?

simon

[kesawi]

I was able to create the additional loopback interface using the method described in /index.php?topic=63653.msg344553#msg344553 and then make it persistent at each boot with the Shellcmd package, but haven't been able to work out how to do the traffic redirection.

[kesawi]

I ended up solving my problem by creating a script which I use cron to run every 5 minutes. The script checks the current VPN interface address against the one provided in the squid.conf file and if it's different, replaces it and forces squid to reload the configuration file:

#!/bin/sh

# Variables
VPN_IFACE=ovpnc2
SQUID_CONFIG_FILE=/usr/local/etc/squid/squid.conf

# Get current IP address of VPN interface
VPN_IFACE_IP=$(ifconfig $VPN_IFACE | awk '{print $2}' | egrep -o '([0-9]+\.){3}[0-9]+')

# Check if VPN interface is up and exit if it isn't
if [ -z "$VPN_IFACE_IP" ]
then
        exit 0;
fi

# Check current IP for VPN interface in squid.conf file
VPN_CONFIG_IP=$(grep -m 1 "tcp_outgoing_address" $SQUID_CONFIG_FILE | awk '{print $2}' | egrep -o '([0-9]+\.){3}[0-9]+')

# Check if the config file matches the current VPN interface IP, and if so exit script
if [ "$VPN_IFACE_IP" == "$VPN_CONFIG_IP" ]
then
        exit 0;
fi

# Replace the previous IP address in the squid.conf file with the current VPN interface address
sed -ie 's/'"$VPN_CONFIG_IP"'/'"$VPN_IFACE_IP"'/' $SQUID_CONFIG_FILE

# Force reload of the new squid.conf file
/usr/local/sbin/squid -k reconfigure

[rob_kae]

Just wanted to give kesawi some kudos for this script, it really works and to me is the best way to control the outbound IP of SQUID when you want to use a VPN which can have a dynamic IP, and you dont want to set the default gateway on your PFsense box as a VPN connection (which for me caused problems)

The script doesnt change the IP address in the web interface (that probably needs the webinterface to be restarted) but if you download the CRON pfsense package, and save the script that Kesawi wrote to a file on the PFsense box (I called mine /usr/bin/IP.sh) the script really works.

If you continually run " grep -m 1 "tcp_outgoing_address" /usr/local/etc/squid/squid.conf | awk '{print $2}' | egrep -o '([0-9]+.){3}[0-9]+' " in a terminal session on the Pfsense box, you can see if the CRON job is changing the value.

Nice work!

Rob_kae

[kesawi]

I've since updated this script to handle failover to a second VPN where required.

#!/bin/sh

# Variables
# VPN_IFACE1 is the primary VPN interface, VPN_IFACE2 is the backup VPN interface
VPN_IFACE1=ovpnc1
VPN_IFACE2=ovpnc2
SQUID_CONFIG_FILE=/usr/local/etc/squid/squid.conf

# Check whether VPN interfaces are connected and assign connected interface to VPN_IFACE. Exit if both are down
VPN_IFACE1_STAUS=$(ifconfig $VPN_IFACE1 | awk '{print $2}' | egrep -o UP)
VPN_IFACE2_STAUS=$(ifconfig $VPN_IFACE2 | awk '{print $2}' | egrep -o UP)

if [ -z "VPN_IFACE1_STATUS" ]
then
	VPN_IFACE=$VPN_IFACE1
elif [ -z "VPN_IFACE2_STATUS" ]
then
	VPN_IFACE=$VPN_IFACE2
else
	echo "Both VPN interfaces down"
	exit 1;
fi

# Get current IP address of VPN interface
VPN_IFACE_IP=$(ifconfig $VPN_IFACE | awk '{print $2}' | egrep -o '([0-9]+\.){3}[0-9]+')

# Check current IP for VPN interface in squid.conf file
VPN_CONFIG_IP=$(grep -m 1 "tcp_outgoing_address" $SQUID_CONFIG_FILE | awk '{print $2}' | egrep -o '([0-9]+\.){3}[0-9]+')

# Check if the config file matches the current VPN interface IP, and if so exit script
if [ "$VPN_IFACE_IP" == "$VPN_CONFIG_IP" ]
then
        exit 0;
fi

# Replace the previous IP address in the squid.conf file with the current VPN interface address
sed -ie 's/'"$VPN_CONFIG_IP"'/'"$VPN_IFACE_IP"'/' $SQUID_CONFIG_FILE

# Force reload of the new squid.conf file
/usr/local/sbin/squid -k reconfigure