EMERGENCY: pfsense not blocking ip



  • Hi , we currently under DDoS attack to our apache2 web servers. In front of those servers there is a pfsense firewall with haproxy that load balances traffic.

    We are adding a reject rule for the relevant ip but we observe that requests are still getting through!

    Could this be cause the connection might be already established ?
    Are firewall rules also applied on established connections?

    Thanks



  • I assume you added the reject source IP on your WAN and cleared all states pertaining to that IP?

    On a side note. If it's a volumetric attack, there is nothing you can do. Rejecting the traffic doesn't stop it from reaching you in the first place. If it's a resource attack, design your system better.


  • Netgate

    You will probably have to post your rules so someone can see what you did wrong.

    Must be a block rule
    must block the SOURCE address
    Must be above the rule that passes the normal traffic in the rule set



  • @Harvy66:

    I assume you added the reject source IP on your WAN and cleared all states pertaining to that IP?

    On a side note. If it's a volumetric attack, there is nothing you can do. Rejecting the traffic doesn't stop it from reaching you in the first place. If it's a resource attack, design your system better.

    clear all states pertraining to that IP? How do i do that?
    Regarding the side note, there isnt any issues with the bandwidth, the problem is that connection reach application servers and generate high load



  • @Derelict:

    You will probably have to post your rules so someone can see what you did wrong.

    Must be a block rule
    must block the SOURCE address
    Must be above the rule that passes the normal traffic in the rule set

    Thanks for the quick replay. I have included some screenshots of the rule.


  • Netgate

    Diagnostics > States
    Filter on the source IP address you are trying to block
    Kill them

    Probably not pertinent to your current issue but you really should upgrade to 2.3.4 at some point.



  • @Derelict:

    Diagnostics > States
    Filter on the source IP address you are trying to block
    Kill them

    Probably not pertinent to your current issue but you really should upgrade to 2.3.4 at some point.

    I filtered with the source ip 45.77.40.157 and it didnt return anything…strange.

    Could this has something to do with haproxy listening on port 80?



  • this for me is hard to diagnose as the only thing to work with is a screenshot of "part" of your rules configured in the GUI.

    Ideally need logs, and full cli diagnosis.

    Filtering on a software firewall helps if the attack is saturating resources behind the firewall such as e.g. available apache connections, it doesnt help if e.g. saturating the actual connection or PF states.

    As said above

    Make sure ip is not allowed by any other rules that might be processed ahead of reject rule.
    Clear all states.
    Make sure you covering the right protocol and connection flags.


  • Netgate

    As long as the block rule is above the rule that passes the traffic to HAproxy, no, it is just a connection on port 80.

    Perhaps you should describe exactly what you are seeing in this "DDoS".



  • Remember that if you make a rule during an "event" (connection attempt) as soon as you hit submit, clear the states ect..  any unsolicited attempts matching that firewall rule will be blocked/rejected..

    But solicited attempts will not be. Your server will still be answering the earlier attempts and will still connecting outbound, now the traffic is solicited and will be allowed.

    If I have an event which I want to stop then I tend to disable my servers outbound rule first, then the inbound, then clear the states.

    Do you still have an allow all rule for your servers outbound?  Maybe time to tighten that up a little..  You actually do not need any outbound rules for your server to answer a connection attempt from an inbound rule.



  • @chpalmer:

    Remember that if you make a rule during an "event" (connection attempt) as soon as you hit submit, clear the states ect..  any unsolicited attempts matching that firewall rule will be blocked/rejected..

    But solicited attempts will not be. Your server will still be answering the earlier attempts and will still connecting outbound, now the traffic is solicited and will be allowed.

    If I have an event which I want to stop then I tend to disable my servers outbound rule first, then the inbound, then clear the states.

    Do you still have an allow all rule for your servers outbound?  Maybe time to tighten that up a little..  You actually do not need any outbound rules for your server to answer a connection attempt from an inbound rule.

    @Derelict:

    As long as the block rule is above the rule that passes the traffic to HAproxy, no, it is just a connection on port 80.

    Perhaps you should describe exactly what you are seeing in this "DDoS".

    I Finally figured out what was happening, thankfully before i got mad! I still need your help though.

    This sneaky guy was crafting the "X-Forwarded-For" header and was using a fake ip address hence why when was blocking this ip, traffic was still shown in weblogs  (from this ip).

    We tried a different approach by counting established connections and we quickly started figuring out ips that caused the DDoS. Adding those ips in firewall rules did the trick.

    For clarification reasons, we did a "curl GET /" with an ip of "137.137.136.136" and indeed thats the ip (fake one) we show in our weblogs

    Now, in another cluster where we use nginx as reverse proxy, changing/crafting the X-Forwarded-For Header doesnt work. Meaning , even if you enter a random ip in the specific header nginx will still pass the correct ip to the backend webserver?

    Does this has to do in general with haproxy or is this a pfsense haproxy issue?
    Anyone had such an issue ?

    Once again, thanks for your help



  • Regarding X-Forwarded-For, this is a haproxy issue. In order to sort this out you must add the following parameter on haproxy frontend service in pfsense.

    reqidel ^X-Forwarded-For:.*


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy