Stop 10 Gbps of DDoS?



  • Hello Guys,

    a friend of mine asked me to help him with a very nasty issue.

    He has 2 x 10 Gbps Fiber connection to the local upsteam. This connection was already in place when he entered the building, as he use no more than 1 gbps of bandwidth.

    Recently he started experiencing DDoS in the range of 8-9 Gbps. It is not enough to fill his BW, but it is much more than needed to fry his firewalls. Attacks are mostly TCP SYN floods, but they often change. We once saw a massive 3 Gbps DoS from a single compromised machine.

    He is looking around for commercial solutions, but I hope I can convince him to go opensource, possibly pfsense. Eventually we could make up for the added complexity and less support of open source solutions by hiring a part time admin fully dedicated to this issue.

    The idea is to layer several x86 machines to gradually filter his traffic until only clean traffic reaches his network. Each machine use a different set of rules, so that we can reach hopefully an high enough PPS ratio,

    I was thinking about using this hardware:

    Sandy Bridge EP E5-2643 (less cores but more Mhz)
    64 (96 ?) Gb of ECC RAM
    4 x Intel 120 (300 ?) GB SSD
    Supermicro motherboard
    Intel 10Gb Fiber NIC, X520-DA2

    Or:

    Xeon E3-1270 V2
    32 GB ECC RAM
    2 x Intel 120 SSD
    Supermicro motherboard
    Intel 10gb Fiber NIC, X520-DA2

    (consider that 3 machines of the latter cost me more than one of the former)

    Do you think that by layering several machines we could reach enough throughput?

    What would be the best configuration?

    I thought about placing 2 machines directly attached to the upstream, each with 8 (4 ?) VM instances of pfsense, with load balancing and failover provided by CARP, with another machine in the back doing the last filtering of the traffic. Or maybe virtualization takes up too much overhead, and running bare on the second machine would give me a better throughput?

    Any other idea?

    Am I crazy/dreaming, or do you think that this is feasible?



  • You can't stop a 10 Gb DDoS with any firewall in existence. Attacks that big are extremely difficult to mitigate, and any firewall is definitely not the answer. I've mitigated numerous significantly smaller scale attacks and could offer some advice for those scenarios, but that's way beyond anything I've dealt with and any of the measures I've implemented elsewhere (which won't be possible at that scale). One possibility if you have a very fast router upstream of the firewalls may be to match on some specific attribute of the DDoS packets (sometimes they have something like window size or similar that makes it easy to differentiate and your router can drop that in hardware).

    Good luck is the best I can offer beyond that, but would definitely be interested in a follow up on what you end up doing to mitigate.



  • @cmb:

    You can't stop a 10 Gb DDoS with any firewall in existence. Attacks that big are extremely difficult to mitigate, and any firewall is definitely not the answer. I've mitigated numerous significantly smaller scale attacks and could offer some advice for those scenarios, but that's way beyond anything I've dealt with and any of the measures I've implemented elsewhere (which won't be possible at that scale). One possibility if you have a very fast router upstream of the firewalls may be to match on some specific attribute of the DDoS packets (sometimes they have something like window size or similar that makes it easy to differentiate and your router can drop that in hardware).

    Good luck is the best I can offer beyond that, but would definitely be interested in a follow up on what you end up doing to mitigate.

    our idea was to use load balancing via CARP, while layering several x86 machine.

    Let's say that we have 2 servers on the front row, each handling 5 gbps of traffic. Each server perform the simplest rules, and thus is capable of blocking 30% of the traffic.

    The second layer perform more complex rules, further reducing the traffic, and so on until only the clean traffic reach the network.

    I read some academic paper where this configuration was used for attack of similar dimension.

    Do you think this is unreasonable?

    which other strategies would you suggest, apart from contacting the upstream?



  • That sounds  very interesting.. I'd like to know if that actually works well against the TCP SYN DoS attacks



  • Even 1 Gbps of SYN flood is going to melt down every firewall that exists today, that pps and new connection per second rate is through the roof. CARP isn't the answer because of the way it works, we strictly have active/passive, and the load balancing functionality that exists with CARP in OpenBSD isn't suitable for such scenarios. It ultimately doesn't really matter how many rules you have or how complex they are (it does make a diff, but evaluating 1 rule is far too much under that big of a SYN flood). The first wouldn't just "block 30% of the traffic", it would melt down and take down everything. This kind of scale is something you need either your upstream to address, or consult with one of the big companies that specializes in huge scale DDoS mitigation. That kind of scale is very, very difficult to mitigate. All firewalls are the worst possible tool to address DDoS because their lowest performance limit and ability to scale is in number of new connections per second they can handle. There's a reason Cisco sells a DDoS mitigation appliance and not ASAs for that purpose.



  • I think it's possible to stop10 Gbps DDoS.
    If you can afford 10 Gbps of bandwidth then you can afford high end equipment like cisco switches or other fast network equipments.
    The 3700 series will load balance per src-ip the traffic to several different firewalls.
    They are also 10 Gbps capable. You put a couple of 10 Gbps X2 in it. 2 ISP at 10 Gbps each. 3 very high performance machine with 2 quad-nic. 16 GB of RAM should be enough.
    1 frond-end bonding of 4 Gbps and 4 Gbps bonding behind without any vlaning transformation done on the server side.
    It gives you 12 Gbps of pure bandwidth processing power.
    It costs a lot to own that. You should hire me. :)
    People who say it's impossible is because they don't understand how DDoS work and how to stop them. There are tricks to stop and slow them down. Some tricks are explain in this blog entry

    http://www.wedebugyou.com/2012/11/how-to-prevent-and-mitigate-ddos-part1/

    Jean



  • I had a quick look at your blog post, but I don't think that pfSense in its current form is the solution against a large DDoS, let alone a huge 10Gbps attack.

    Based on some posts I've seen, when FreeBSD 10 upcoming SMP-pf becomes available things might change.


  • Banned

    Why cant you scale on inbound IP-adress and block it, if packages reach a certain number per second?

    Is that possible?



  • For sure the new SMP-pf will be totally a big change. I achieve around 1.8 Gbps with a Dell 2850 and 2 quad nic. The secret rely in BGP, this is how you stop a DDoS. If 100 machines sends 1 Gbps, you have to tell your ISP through BGP to not send you the traffic of these 100 machines and it's free… as long as you have BGP. But, I think that pfsense could block many DDoS and easily scale horizontally with many pfsense machines. If they choke at 2 Gbps, then you put 5 machines at 2 gbps fully load-balance. You will then have 10 gbps around. I am sure they can achieve much more than that per machine.


Locked