Haproxy - Traffic Limiting

[treuss]

Hi,

we're currently setting up haproxy for traffic limiting. Our main motivation is to slow down bots/crawlers that stress our servers exceedingly.

Our haproxy looks like this:

frontend p2-web-80
  # Table definition  
  stick-table type ip size 512k expire 30s store conn_cur,conn_rate(3s),http_req_rate(5s)

  # Allow clean known IPs to bypass the filter
  #tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }

  # Shut the new connection as long as the client has already XX opened
  tcp-request connection reject if { src_conn_cur ge 10 }
  tcp-request connection track-sc1 src

  # Shut the new connection as long as the client has already 10 opened
  tcp-request connection reject if { src_conn_rate ge 10 }
  tcp-request connection track-sc1 src

  # Use General Purpose Couter (gpc) 0 in SC1 as a global abuse counter
  # Monitors the number of request sent by an IP over a period of 10 seconds
  tcp-request connection track-sc1 src
  tcp-request connection reject if { src_get_gpc0 gt 15 }

backend
  # If the source IP generated 10 or more http request over the defined period,
  # flag the IP as abuser on the frontend
  acl abuse src_http_err_rate(p2-web-80) ge 10
  acl flag_abuser src_inc_gpc0(p2-web-80)
  tcp-request content reject if abuse flag_abuser

We've tested these settings using siege, which worked pretty well, however, there are some open questions we haven't been able to find answers for:

• Although Firefox uses only 6 concurrent connections, some GET-requests on images will be blocked by haproxy, if Firefox is forced to reload the page via CTRL+SHIFT+R. Why is that?

• We need to maintain a whitelist (tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }) of corporate customers with clients hidden via NAT. Otherwise, obviously, their multiple requests will be blocked. Is it possible to maintain such a list on pfsense's GUI?

• In our backend, we would like to handle abusers, tracked by the frontend p2-web-80. Unfortunately, when I activate the backend rules, I'm getting this error:

  
      [WARNING] 050/104454 (87620) : parsing acl keyword 'src_inc_gpc0(p2-web-80)' :
      no pattern to match against were provided, so this ACL will never match.
      If this is what you intended, please add '–' to get rid of this warning.
      If you intended to match only for existence, please use '-m found'.
      If you wanted to force an int to match as a bool, please use '-m bool'. 
  
  

  Is there a way to fix this?

Best regards
Thomas

[treuss]

bump