Policy routing packets loss



  • Hello,

    I've just faced a really strange issue and have no ideas what might cause it. Sorry if the thread I chose is wrong.

    I'm setting up a transparent proxy forwarding for LAN users. The proxy itself (squid) is located on another server in the LAN. So, when users try to browse a site they routed by pfsense to the proxy server, which retrieves the site and return it to the user directly.

    Here is how I accomplished that:
    1. The gateway of the proxy server is defined in System:Routing
    2. There is a LAN firewall rule which routes all users tcp/80 traffic for non-local hosts to the proxy gateway
    3. The proxy server is a linux box which has appropriate iptables redirect rule

    Pretty simple.

    Everything works just fine, but from time to time users complain that browsing is not working: browser waits for connection to a web site. If a user takes patience the browser finally is able to connect and things go well again until next issue. They delay might be as several seconds as 10s seconds. In some cases it may freeze forever.

    Trying to figure out what's going on, I've discovered that policy routing does not work well all the time - some packets are not routed to the proxy gateway for some reason. Below are tcpdumps from the user's PC (172.26.10.1/16), the pfsense router and the proxy server (172.26.1.50/16) taken at some point in time (not the full session):

    User:

    
    12:53:50.565244 00:13:d3:64:4b:49 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 66: 172.26.10.1.1448 > 199.47.216.144.80: . ack 344 win 32764 <nop,nop,timestamp 37078="" 3780924096="">12:53:56.386479 00:13:d3:64:4b:49 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 259: 172.26.10.1.1448 > 199.47.216.144.80: P 194:387(193) ack 344 win 32764 <nop,nop,timestamp 37136="" 3780924096="">12:54:06.048685 00:13:d3:64:4b:49 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 259: 172.26.10.1.1448 > 199.47.216.144.80: P 194:387(193) ack 344 win 32764 <nop,nop,timestamp 37233="" 3780924096="">12:54:16.292892 00:17:31:4f:0e:3b > 00:13:d3:64:4b:49, ethertype IPv4 (0x0800), length 409: 199.47.216.144.80 > 172.26.10.1.1448: P 1:344(343) ack 194 win 243 <nop,nop,timestamp 36363="" 3780975351="">12:54:16.292914 00:13:d3:64:4b:49 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 66: 172.26.10.1.1448 > 199.47.216.144.80: . ack 344 win 32764 <nop,nop,timestamp 37335="" 3780924096="">12:54:25.378943 00:13:d3:64:4b:49 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 259: 172.26.10.1.1448 > 199.47.216.144.80: P 194:387(193) ack 344 win 327
    64 <nop,nop,timestamp 37426="" 3780924096=""></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp> 
    

    pfsense router's LAN interface:

    
    12:53:50.961214 00:13:d3:64:4b:49 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 66: 172.26.10.1.1448 > 199.47.216.144.80: Flags [.], ack 344, win 32764, options [nop,nop,TS val 37078 ecr 3780924096], length 0
    12:53:56.784260 00:13:d3:64:4b:49 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 259: 172.26.10.1.1448 > 199.47.216.144.80: Flags [P.], ack 344, win 32764, options [nop,nop,TS val 37136 ecr 3780924096], length 193
    12:54:06.449292 00:13:d3:64:4b:49 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 259: 172.26.10.1.1448 > 199.47.216.144.80: Flags [P.], ack 344, win 32764, options [nop,nop,TS val 37233 ecr 3780924096], length 193
    12:54:16.696550 00:13:d3:64:4b:49 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 66: 172.26.10.1.1448 > 199.47.216.144.80: Flags [.], ack 344, win 32764, options [nop,nop,TS val 37335 ecr 3780924096], length 0
    12:54:25.785307 00:13:d3:64:4b:49 > 00:00:5e:00:01:01, ethertype IPv4 (0x0800), length 259: 172.26.10.1.1448 > 199.47.216.144.80: Flags [P.], ack 344, win 32764, options [nop,nop,TS val 37426 ecr 3780924096], length 193
    
    

    proxy:

    
    12:54:16.673797 00:17:31:4f:0e:3b > 00:13:d3:64:4b:49, ethertype IPv4 (0x0800), length 409: 199.47.216.144.80 > 172.26.10.1.1448: Flags [P.], seq 0:343, ack 1, win 243, options [nop,nop,TS val 3780975351 ecr 36363], length 343
    
    

    As you can see, no packets were routed to the proxy at this time period, although there were 5 packets sent out of the user's PC.

    Some additional notes:
    1. Default gateway on the user's PC and the proxy server is the same and set up on the pfsense router's LAN interface as a CARP address (that's why it has the "strange" MAC address: 00:00:5e:00:01:01)
    2.```

    pfctl -sr | grep 172.26.10.1
    pass in quick on re0_vlan1 route-to (re0_vlan1 172.26.1.50) inet proto tcp from 172.26.10.1 to ! <companynets>port = http flags S/SA keep state label "USER_RULE: Transparent proxy forwarding"</companynets>

    3\. The proxy gateway is reachable from the pfsense router:
    

    ping 172.26.1.50
    PING 172.26.1.50 (172.26.1.50): 56 data bytes
    64 bytes from 172.26.1.50: icmp_seq=0 ttl=64 time=0.209 ms

    
    Let me know if you need anything else.
    
    I really need help! Thanks in advance.


  • Looks like I've manage to fix it by myself. The answer is to check "Any flags" for "TCP flags" in the LAN firewall rule which redirects traffic to the proxy server.
    So, now the rule looks like:

    
    pass in quick on re0_vlan1 route-to (re0_vlan1 172.26.1.50) inet proto tcp from 172.26.10.1 to ! <corporatenets>port = http flags any keep state label "USER_RULE: Transparent proxy forwarding"</corporatenets> 
    

    Hope, this will help someone.


Locked