Lusca-cache r14756 transparent proxy broken on some installations - fix



  • Hello All,

    Short description:

    On some pfSense 1.2.3-RELEASE installation lusca-cache installs correctly and runs but transparent proxy does not work.  http browsing shows no activity in /var/squid/log/access.log. On other installations installing lusca runs out of the box without issue.  Following is an explanation of why and how to fix the problem.

    Long description:
    Lusca-cache has been installed on 2 seemingly identical pfsense installations. The first (the one that works) is a vmware appliance. The second one which doesnt work is an alix based appliance.

    The installation was done according to the following
    http://forum.pfsense.org/index.php/topic,16569.60.html
    using
    http://code.google.com/p/pfsense-cacheboy/wiki/Pfsense_Lusca

    proxy cache was configured and 3128 was entered into the proxy_port field under Services->Proxy server->General.

    On the vmware machine the proxy installed without error and once enabled all http traffic on the local LAN was redirected to the proxy server.  The log files show that caching is working well.

    Following the same installation procedure on the alix box shows that http traffic form the local LAN (in this case the WLAN or Opt 2) fails to be routed to the proxy-cache.

    The output from "pfctl -s nat | grep http" follows.

    ALIX
    rdr on ath0 inet6 proto tcp from any to ! (ath0) port = http -> fe80::280:48ff:fe67:f461 port 3128

    VMWARE
    rdr on em1 inet proto tcp from any to ! (em1) port = http -> 192.168.11.1 port 3128

    Note that the ALIX box is using an IPV6 address ?!?  while the VMWARE guest has an IPV4 address. Looking at /tmp/rules.debug we see
    rdr on em1 proto tcp from any to !(em1) port 80 -> em1 port 3128
    for the VMWARE client and
    rdr on ath0 proto tcp from any to !(ath0) port 80 -> ath0 port 312

    so… it seems that pfctl is the substituting IPV4 and IPV6 addresses even though IPV6 is disabled in System->Advanced->IPv6.

    ifconfig for both guests show
    VMWARE
    em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:0c:29:7c:c3:c0
    inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
    inet6 fe80::20c:29ff:fe7c:c3c0%em1 prefixlen 64 scopeid 0x2
    media: Ethernet autoselect
    status: no carrier

    ALIX

    ath0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    ether 00:80:48:67:f4:61
    inet6 fe80::280:48ff:fe67:f461%ath0 prefixlen 64 scopeid 0x4
    inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255
    media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap>status: associated
    ssid webXaccelerator channel 1 (2412 Mhz 11g) bssid 00:80:48:67:f4:61
    authmode WPA2/802.11i privacy MIXED deftxkey 3 TKIP 2:128-bit
    TKIP 3:128-bit txpower 31.5 scanvalid 60 bgscan bgscanintvl 300
    bgscanidle 250 roam:rssi11g 7 roam:rate11g 5 pureg protmode OFF burst
    -apbridge dtimperiod 1

    Note that Pv6 is listed ahead of the IPv4 address for the ALIX entry and the opposite for the VMWARE box.  So... it seems that pfctl expands the interace name to either an IPv4 or IPv6 addressed based on the order of the definitions in the interface.  Pretty strange!

    To further confuse things the entry created by squid looks like
    rdr on ath0 inet proto tcp from any to ! (ath0) port = http -> 127.0.0.1 port 80
    on the ALIX.

    Interesting that localhost is being used here instead of the IP address of the interface and the port (3128)

    Looking at squid.inc we find the following on line 1458
                          foreach ($ifaces as $iface) {
                            if ($squid_conf['proxy_port'] > 0) {
                                    $rules .= "rdr on $iface proto tcp from any to !($iface) port 80 -> $iface port $port\n";
                            } else {
                                    $rules .= "rdr on $iface proto tcp from any to !($iface) port 80 -> 127.0.0.1 port 80\n";
                            }
                            }

    if the proxy_port is set to 0 then the standard squid rule is used. Otherwise the lusca rule with the IP and port number are used.  Interesting…

    So... setting the default port number to 0 fixes the problem.  Setting the port number to 0 in the config causes 3128 to be used since the following snippet is found through out squid.inc
          $port = ($settings['proxy_port'] ? $settings['proxy_port'] : 3128);
    and also causes an pf rdr rule to be created that works.

    And indeed it does. Setting the port to 0 causes all http traffic to be routed to the proxy server.

    looking at the corresponding code snippet in the squid package squid.inc we find following on line 1339
                            foreach ($ifaces as $iface) {
                                    $rules .= "rdr on $iface proto tcp from any to !($iface) port 80 -> 127.0.0.1 port 80\n";
                            }

    Note that the squid package ** NEVER ** creates rdr rules based on IP address and port number.  Hm…  this is very strange.

    Also... when comparing the rules.debug file created by lusca and squid we find the following.

    diff rules.debug rules.debug.squid

    50c50
    < rdr on ath0 proto tcp from any to !(ath0) port 80 -> ath0 port 3128

    rdr on ath0 proto tcp from any to !(ath0) port 80 -> 127.0.0.1 port 80
    59,60c59,60
    < #pass in quick on ath0 proto tcp from any to !(ath0) port 80 flags S/SA keep state
    < #pass in quick on ath0 proto tcp from any to !(ath0) port 3128 flags S/SA keep state


    pass in quick on ath0 proto tcp from any to !(ath0) port 80 flags S/SA keep state
    pass in quick on ath0 proto tcp from any to !(ath0) port 3128 flags S/SA keep state

    The pf "pass in quick" rules for the lusca package are commented out. Why is that?!? and
    sure enough lusca's squid.inc shows this.
                            foreach ($ifaces as $iface) {
                                    $rules .= "# Setup squid pass rules for proxy\n";
                                    $rules .= "#pass in quick on $iface proto tcp from any to !($iface) port 80 flags S/SA keepstate\n";
                                    $rules .= "#pass in quick on $iface proto tcp from any to !($iface) port $port flags S/SA keep state\n";
                                    $rules .= "\n";
                            };

    while the squid.inc from the squid package has the following
                          foreach ($ifaces as $iface) {
                                    $rules .= "# Setup squid pass rules for proxy\n";
                                    $rules .= "pass in quick on $iface proto tcp from any to !($iface) port 80 flags S/SA keep state\n";
                                    $rules .= "pass in quick on $iface proto tcp from any to !($iface) port $port flags S/SA keep state\n";
                                    $rules .= "\n";
                            };

    Editing lines 1457 and 1478 in lusca's squid.inc file to make them the same as lines 1339 and 1356 in squids squid.inc file results in the creation of identical rules.debug pf input files.

    So two solutions are available if you run into this issue.
    a. set the proxy port to 0 (not possible if you want lusca to be on a port other than 3128)
    or
    b. modify the package after its been installed as described above.

    It would be interesting to hear form the lusca package builder why he felt the pf "rdr" rules should be constructed differently than those in squid.

    Take care.

    --luis

    --

    Luis Soltero, Ph.D., MCS
    Director of Software Development,
    CTO Global Marine Networks,
    LLC StarPilot,
    LLC Tel: 865-379-8723
    Fax: 865-681-5017
    E-Mail: lsoltero@globalmarinenet.net
    Web: http://www.globalmarinenet.net
    Web: http://www.starpilotllc.com</hostap></up,broadcast,running,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>


Log in to reply