What is the biggest attack in GBPS you stopped



  • hi,

    i am hit by ddos (upd flood mostly) and looking for solutions, hopefully opensource ones. I wanted to know what was the biggest multi gigabits attack you successfully stopped with your pfsense setup in the field ( so not with nullrouting at ISP level) and what the hardware used was.

    My actuel issue is on the 5 to 10 gbps DDOS udp flood attacks so i search to see if a 20gbps filtering firewall could work in the real world of April 2015 and help me mitigate 1-16gbps attacks. My problem is to filter myself not ask upstrream to help so i really speak of how i can filter this and if anyone here had setup playing at this level of gbps.

    regards,
    Ghislain.


  • Banned

    You dont stand a chance with pfSense.

    You need something that has been developed to mitigate the traffic.



  • that's bad news i hoped the  Chelsio cards and pci-e v3 and big xeon would have enough to rate limit upd if not more.

    Is this in your experience a limit with the harware or the Os (freebsd here) .

    I guess the answer is that the current software will not be fast enough in the current hardware but just for getting your insigth.

    Does not seems a lot of people tried it i see only on the forum people like me that wonder if this can work but not that i can found that successfully done multi gigabits protection a reality with amd64 type servers and open source firewalling.

    regards,
    Ghislain


  • Banned

    Its in the OS. Hardware can easily handle it if you got some muscle.

    I can take this site offline using a specific type of traffic that takes no more than 70-80Mbps bandwith.

    When that traffic hits pfSense, its dead. Goes offline instantly. No matter how powerful the hardware is.

    I run 8 Core, 16GB ram and SSD. Dead in a second if it hits.



  • Well that's scary.

    Any way to have information on how to do this because i would love to test this against the  pfsense and diverse others solution.

    I guess all the FreeBSD ones will suffer this issue. Perhaps others competing in this space like vyaos, routeros or netbsd or basic linux or even basic FreeBSD or modified pfsesnse if the latest kernel is patched against the issue could make it, depending on the issue used here. The issue is that the TCP/IP stack of FreeBSD is considered by the internet (and it is never wrong) to be the most reliable one and a lot of other's tcp/ip stack is borrowed from it so it is scary that it could be killed by 80mbps.


  • Banned

    I can tell you this much….

    Windows firewall doesnt get affected by any of these attacks. If you put the server out front and only have WF running and forwarding traffic to the server then it can handle it easily.

    It seems to only affect UNIX/Linux/BSD distros.



  • ok,

    seems nobody had any ddos success story yet so seems there is more thinking to be  done.

    best regards,



  • I can take this site offline using a specific type of traffic that takes no more than 70-80Mbps bandwith.

    Scary or?

    It seems to only affect UNIX/Linux/BSD distros.

    Really sad to hear about it!

    Pointed to the hardware it would not be the thing as I see it right.
    Lanner FW-8895
    test for pfSense are running, at the moment, but at this time it is not running related to the BIOS.
    Lanner Packet Processing card
    Would be awesome to see this card running under pfSense that can handle 40 GBit/s and comes with
    a bypass option.



  • @Supermule:

    I can tell you this much….

    Windows firewall doesnt get affected by any of these attacks. If you put the server out front and only have WF running and forwarding traffic to the server then it can handle it easily.

    It seems to only affect UNIX/Linux/BSD distros.

    Say what?  After all these years of Windows bashing by the nix and open source fans Windows firewall puts it to shame.  LOL


  • Banned

    Dunno really what's this debate about? You don't stop DDoS via packet filters. You null route it. It's damn useless to packet filter DDoS crap – even if the firewall does not crash, the packet flood still totally fills your pipeline and renders it useless.


  • Banned

    Depending on your pipe.

    I have 2x10G in the datacenter on WAN and it doesnt get filled at all. So no blocking of pipe in the relation.

    Its the handling of packets and yes you can packetfilter DDoS and not nullroute it.

    Thats exactly what they want to achieve and we need a FW solution that can handle it at wirespeed.

    No matter the cores and memory, pfSense still dies instantly.

    So tell me where the culprit is, because traffic hardly get to the servers at all. If I stick a windows firewall on WAN directly on the server, then everything is fine even under several gigabit/s traffic patterns.



  • @Supermule:

    Depending on your pipe.

    I have 2x10G in the datacenter on WAN and it doesnt get filled at all. So no blocking of pipe in the relation.

    Its the handling of packets and yes you can packetfilter DDoS and not nullroute it.

    Thats exactly what they want to achieve and we need a FW solution that can handle it at wirespeed.

    exactly, right now 6GBPS do not fill the pipe of the main access that peak at 40gbps , but yes it fill the pipe of the server that is 1G. The purpose is to prevent nullrouting for all attack inferior to half the pipe by putting a firewall between the main router and the rack ( 20gbps is half the datacenter pipe).


  • Banned

    And you cant do that with pfSense.

    It cant handle the DDoS traffic at all, especially SYN ACK and OVH scripts kills it instantly.



  • That is an unorthodox way of arguing, Supermule; proving an item's inadequacies by proving your inability to skillfully operate said <thing>(pfSense, Linux, Ford F150, shovels, pants, etc). To continually defend your stance, it is in your self-interest to not only stay ignorant, but perhaps even choose to learn exclusively the wrong ways of using <thing>. The worse your skills become with <thing>only validates your stand-point more and more. A strange way to prove a point… ;)

    ghislain26, you say you can get your device installed earlier in the HOPs? Is this data-center multi-homed?</thing></thing></thing>



  • @Nullity:

    ghislain26, you say you can get your device installed earlier in the HOPs? Is this data-center multi-homed?

    From what i have been told the 40gbps from multiple peerings arrives at 2 redundant cisco where i can connect in 10gbps ports ( 2 ports in bonding here) so i can filter there , normaly the main routers go in 20gbps to the rooms routers.
    So i would playing the role the room routers for my ip's with my filtering box and from there go to my rack probably in bridging mode. Of course the fact that i have basic knowledge of networking but not advanced ones limit my possibilities.



  • @ghislain26:

    @Nullity:

    ghislain26, you say you can get your device installed earlier in the HOPs? Is this data-center multi-homed?

    From what i have been told the 40gbps from multiple peerings arrives at 2 redundant cisco where i can connect in 10gbps ports ( 2 ports in bonding here) so i can filter there , normaly the main routers go in 20gbps to the rooms routers.
    So i would playing the role the room routers for my ip's with my filtering box and from there go to my rack probably in bridging mode. Of course the fact that i have basic knowledge of networking but not advanced ones limit my possibilities.

    Do you have plenty of logs during the DDoS so that you can figure out what the first hints will be that the attack is beginning? Did they use just 1 type of attack, or did they use multiple? Did you get multiple types of DDoS simultaneously, or did the attacks come in coordinated waves? Did a magority of the  IPs from a common area that you can set an alias for?



  • The attack we faced was from 2gbps to 6.5 gbps of smal udp packets flood on random ports (the destination ports changed every 10 minutes or so from what i gathered) . The ip were numerous and from india and china, probably some botnet or simply spoofed but i beleive it was a botnet.

    I do not have trace or evidence as i had no filtering device and when the attacks where "on" i was unable to reach the server until they nullrouted my ip at the data center so.. I poke around to see if i can build up some protection even if i know a flood bigger than 15gbps will probably make me nullrouted anyway i want to provide minimum resistance to the lesser flood one and not be killed by one person with 2 servers flooding a 1gbps port.



  • @ghislain26:

    The attack we faced was from 2gbps to 6.5 gbps of smal udp packets flood on random ports (the destination ports changed every 10 minutes or so from what i gathered) . The ip were numerous and from india and china, probably some botnet or simply spoofed but i beleive it was a botnet.

    I do not have trace or evidence as i had no filtering device and when the attacks where "on" i was unable to reach the server until they nullrouted my ip at the data center so.. I poke around to see if i can build up some protection even if i know a flood bigger than 15gbps will probably make me nullrouted anyway i want to provide minimum resistance to the lesser flood one and not be killed by one person with 2 servers flooding a 1gbps port.

    And there is nothing the datacenter where your rack is in can do for you? How do they nullroute
    the traffic for you? Why they can´t offering you something to protect you from that DDoS attacks?
    Would be interesting to know if the are not offering such a service as an option on top of their service!

    edit: I found something that would be placed between the both routers at the WAN interfaces and the
    firewalls later after them in the background, pending on what throughput we are talking about the right
    appliance must be chosen. here is a pdf from them about their hardware.

    Corero IPS 5500 ES-Series


  • Banned

    If you are so smart and "godlike" then pls. post a working config and an IP that I can target.

    Then I will prove my point…

    When logging, it can be both TCP and UDP. UDP floods tends to be bandwith consuming and TCP are specific protocols and maybe L7 traffic like VSE, RUDY or ARME Scripts...

    OVH takes down pfSense at once no matter the bandwith. We have seen as low as 40mbit to make it completely useless and servers are unreachable.

    @Nullity:

    That is an unorthodox way of arguing, Supermule; proving an item's inadequacies by proving your inability to skillfully operate said <thing>(pfSense, Linux, Ford F150, shovels, pants, etc). To continually defend your stance, it is in your self-interest to not only stay ignorant, but perhaps even choose to learn exclusively the wrong ways of using <thing>. The worse your skills become with <thing>only validates your stand-point more and more. A strange way to prove a point… ;)

    ghislain26, you say you can get your device installed earlier in the HOPs? Is this data-center multi-homed?</thing></thing></thing>



  • @Supermule:

    @Nullity:

    That is an unorthodox way of arguing, Supermule; proving an item's inadequacies by proving your inability to skillfully operate said <thing>(pfSense, Linux, Ford F150, shovels, pants, etc). To continually defend your stance, it is in your self-interest to not only stay ignorant, but perhaps even choose to learn exclusively the wrong ways of using <thing>. The worse your skills become with <thing>only validates your stand-point more and more. A strange way to prove a point… ;)

    ghislain26, you say you can get your device installed earlier in the HOPs? Is this data-center multi-homed?</thing></thing></thing>

    If you are so smart and "godlike" then pls. post a working config and an IP that I can target.

    Then I will prove my point…

    When logging, it can be both TCP and UDP. UDP floods tends to be bandwith consuming and TCP are specific protocols and maybe L7 traffic like VSE, RUDY or ARME Scripts...

    OVH takes down pfSense at once no matter the bandwith. We have seen as low as 40mbit to make it completely useless and servers are unreachable.

    Either become part of the solution to your problem, or move on. Perhaps create a thread dedicated to your problem and all the logical steps you have taken to isolate the bug. What led you to suspect that your bug affects all unix-based operating systems?
    What sort of things have you already eliminated as potential culprits? Have you already confirmed the cause of your bug?

    Being light on the details and heavy on the emotion says obvious things about your intent, though… I could be wrong. :)



  • BlueKobold: if i could make the DC make the move for me i would have trust me :p  They do not offer ddos mitigation other than nullrouting.the problem is not here to check if another one can do it for me i am sure it would be better to take it where the pipes are the biggest but i cannot. I openned this thread to see if there was some usecase similar to mine willing to share experience on this :)

    Supermule: if you agree to that i could send you an ip in private but before i will try to have written permission by the DC just to be sure and prevent any legal issue for anyone :p

    Anything that help the thing to move forward is great :) Look forward any other experiences i learn at the same time and educate myself.



  • They do not offer ddos mitigation other than nullrouting.

    Ah, ok this was not clear to me!


  • Banned

    You are.

    @Nullity:

    @Supermule:

    @Nullity:

    That is an unorthodox way of arguing, Supermule; proving an item's inadequacies by proving your inability to skillfully operate said <thing>(pfSense, Linux, Ford F150, shovels, pants, etc). To continually defend your stance, it is in your self-interest to not only stay ignorant, but perhaps even choose to learn exclusively the wrong ways of using <thing>. The worse your skills become with <thing>only validates your stand-point more and more. A strange way to prove a point… ;)

    ghislain26, you say you can get your device installed earlier in the HOPs? Is this data-center multi-homed?</thing></thing></thing>

    If you are so smart and "godlike" then pls. post a working config and an IP that I can target.

    Then I will prove my point…

    When logging, it can be both TCP and UDP. UDP floods tends to be bandwith consuming and TCP are specific protocols and maybe L7 traffic like VSE, RUDY or ARME Scripts...

    OVH takes down pfSense at once no matter the bandwith. We have seen as low as 40mbit to make it completely useless and servers are unreachable.

    Either become part of the solution to your problem, or move on. Perhaps create a thread dedicated to your problem and all the logical steps you have taken to isolate the bug. What led you to suspect that your bug affects all unix-based operating systems?
    What sort of things have you already eliminated as potential culprits? Have you already confirmed the cause of your bug?

    Being light on the details and heavy on the emotion says obvious things about your intent, though… I could be wrong. :)



  • I can't believe this thread is on to the second page…

    You cannot mitigate a DDoS attack with a single firewall/router.  If it was that easy, don't you think Sony, Microsoft and anyone running a cloud service would do it and DDoS would be a thing of the past?  If it was that easy, why are there services like CloudFlare that specialize in DDoS protection?  Only global traffic inspection & load-balancing will do it for you... if you're willing to pay.

    Give up on this notion of blocking DDoS with pfSense.


  • Banned

    Listen….pfSense replies to specific traffic in a non orderly fashion. It chokes itself...

    And yes you can. ASk yourself why people say it cant be done. Because they pay BIG bucks to get protected.....

    But in fact they dont. They just get null routed and then wait for services to come back online.



  • @Supermule:

    Listen….pfSense replies to specific traffic in a non orderly fashion. It chokes itself...

    And yes you can. ASk yourself why people say it cant be done. Because they pay BIG bucks to get protected.....

    But in fact they dont. They just get null routed and then wait for services to come back online.

    Can I see some proof, or must I trust you?



  • And yes you can. ASk yourself why people say it cant be done. Because they pay BIG bucks to get protected…..
    But in fact they dont. They just get null routed and then wait for services to come back online.

    I'm not sure who you're talking to here.

    Can I see some proof, or must I trust you?

    Heh, give him a public IP of one of your routers and perhaps you may see for yourself…



  • Null routing won't protect you against spoofed source IPs. It's the firewall's job to drop out of state packets, not die. I understand that the fast path is if the state already exists, I understand that running through the rules is not quite as fast as the fast path, but that's not the issue either. The issue is dropped packets are some how the most expensive path of all, to the point that the router dies with only a relatively trickle of them.

    Maybe this is more of a FreeBSD issue than PFSense, but it seems to be something misconfigured or a fundamental flaw.

    Step 1) See if packet is part of an existing flow, if so pass, else goto step 2
    Step 2) Check packet against rules, if passes, create new flow, else goto step 3
    Step 3) Drop packet then jump off a cliff

    Step 3 needs to be fixed to not be so emo.



  • I forgot that this was an issue that only occurred when lots of source IP+ports were used, not that dropping packet is expensive. PFSense handles a single source attempting to DOS, but the exact same attack as a DDOS, even with the same number of PPS, will result in PFSense crapping its pants.



  • @KOM:

    You cannot mitigate a DDoS attack with a single firewall/router.  If it was that easy, don't you think Sony, Microsoft and anyone running a cloud service would do it and DDoS would be a thing of the past?  If it was that easy, why are there services like CloudFlare that specialize in DDoS protection?  Only global traffic inspection & load-balancing will do it for you… if you're willing to pay.

    i dont think sony was hit by only 6gbps, if i cannot protect me against ALL ddos i should be able to mitigate the lesser one. It's like saying there is no point to lock your door because bank are robbed ?

    seems that everybody just say when a teenager get angry he pays 10 buck and boom goes your site, just null route it and wait a week or two and it will be okay or pay 9209318401841$ for a protection service with arbor ?

    All i see until now is that nobody seems to use pfsense successfully for somethign bigger than a single Gb or they stay silent :) For now all agree that it (and freeBSD) cannot handle this.

    Thanks for all the time you dedicated to answering  me, i will continue my journey for a solution (i will continue to monitor the forum about this just in case).

    best regards,
    Ghislain.



  • Stopping someone from entering your home; pfSense can do that.

    Stopping someone from picketing your house; pfSense cannot do that (from inside your house). You need to own the entire neighborhood.

    I would like to see some examples of someone stopping, or even slightly mitigating, a UDP-based DDoS while only controlling the final hop.



  • @Ghislain

    seems that everybody just say when a teenager get angry he pays 10 buck and boom goes your site, just null route it and wait a week or two and it will be okay

    He don´t must pay anything, he download a software as many others allso and then they are
    all attacking your site DoS is not DDoS!  8)

    or pay 9209318401841$ for a protection service with arbor ?

    Dealing with 10 Gbit/s single or multiple time is no child game, this is business like
    IT and this was never and will be never a game, journey or something for the "get it cheap"
    generation, as I see it right. And if your DC is not able to prevent you from those attacks
    will be showing up that this is not one option more to make money but more then spending
    much more money then the most customers want to pay for. Because this could be done
    it means also not that you are taking much money and solve it out for ever, because the
    next DDoS attack could be then hitting you with 65 GBit/s and more sufficient hardware or
    will services would be really urgent needed. With the Corero IPS 5500-2400ES you will be
    able to protect your server and this was in my eyes the core of your question. For sure those
    devices are not to shoot for some bucks at eBay and also not able to collect from the dump
    for paying nothing.  :D

    All i see until now is that nobody seems to use pfsense successfully for somethign bigger than a single Gb or they stay silent :) For now all agree that it (and freeBSD) cannot handle this.

    This is really sad in my eyes, only and because your wishes would not be able to solved out, we
    are all now the small players in this scene? But I was telling you and by setting up a link something
    how we protect our company by using Corero devices, and why we all are now using a single GBit/s
    WAN line??? For sure we must pay for that and in this scenario also twice.  ;)

    For now all agree that it (and freeBSD) cannot handle this.

    If Lanner gets the FW-8895 working for pfSense and the Tilera packet processing cards will
    be able to use, you get a fair change to work it out with OpenDPI, but as I read it between
    the lines, it could be then also again very expensive and this is nothing for you and your business
    as well you want it getting cheap. Because pfSense is OpenSource and free of cost, that is not
    meaning that pfSense is not willing to have a adequate hardware basis to run smooth for the job!

    Thanks for all the time you dedicated to answering  me, i will continue my journey for a solution (i will continue to monitor the forum about this just in case).

    Then you can start here in 2012, same question and with the same answers.
    Stop 10 Gbps of DDoS?  :o


  • Banned

    EXACTLY!

    And the funny shit is, that it dies also when changing SYNPROXY state to STATELESS!

    What would that tell you??

    Whats even funnier is that using OVH scripts and limiting the PPS pr. rule (even the block all rule) doesnt help. You can create an advanced ruleset with 100PPS and it still dies on specific scripts. Then the total bandwith will be very small, but pfSense dies…

    Where to look for an error like that? Its buried deep within BSD/Linux.

    I revived an old ISA Server 2006 and testet it out front and it wasnt affected when configured.

    @Harvy66:

    Null routing won't protect you against spoofed source IPs. It's the firewall's job to drop out of state packets, not die. I understand that the fast path is if the state already exists, I understand that running through the rules is not quite as fast as the fast path, but that's not the issue either. The issue is dropped packets are some how the most expensive path of all, to the point that the router dies with only a relatively trickle of them.

    Maybe this is more of a FreeBSD issue than PFSense, but it seems to be something misconfigured or a fundamental flaw.

    Step 1) See if packet is part of an existing flow, if so pass, else goto step 2
    Step 2) Check packet against rules, if passes, create new flow, else goto step 3
    Step 3) Drop packet then jump off a cliff

    Step 3 needs to be fixed to not be so emo.



  • @Nullity:

    I would like to see some examples of someone stopping, or even slightly mitigating, a UDP-based DDoS while only controlling the final hop.

    as long as you do not fill up the pipe i would have hoped a

    INCOMMING UDP to this IP => DROP ALL

    could let me have my web server continue working, this should not cost that much to a dual eight core xeon with multiple 10gbps chelsio T5 cards and plenty of ddr4 ram.

    I understand i have only a theorical 2000ft view of it but the numbers seems to indicate that this level of hardware is theoricaly capable of handling the flow, now the cost of the operating system and tcp stack is a big part of unknow here but i was naively thinking it could do this.

    I am not trying to do this on the cheap, what i am trying is to keep control of it, opensource is a way to keep control of what is done and beter than a blackbox imho. Also more important i am trying to see if anyone has such setup. All the answers here indicate that this is not the case and not feasible that i should look for an upstream protection. If this is the experience of people on the field i understand, i keep looking anyway.



  • opensource is a way to keep control of what is done and beter than a blackbox imho.

    OpenSource or Closed Source, Black Box or Self made Box, is all either for me. If I have a problem
    and find one who is also able to solve it out, that is my dealer!

    See where they are placing their solution, between the routers and the firewalls.
    And you try to find out a way to solve the problems out at only one point, the firewall.

    Picture source: Corero.com

    ![Corero IPS 5500.jpg](/public/imported_attachments/1/Corero IPS 5500.jpg)
    ![Corero IPS 5500.jpg_thumb](/public/imported_attachments/1/Corero IPS 5500.jpg_thumb)



  • It's like saying there is no point to lock your door because bank are robbed ?

    No, I'm saying that the strongest door you can find will happily collapse when it's being pounded on by a tank.


  • Banned

    Try to add this manually in the system -> tunables

    kern.ipc.somaxconn = 32768

    And test again. We have seen some improvement using that setting



  • @Supermule:

    Try to add this manually in the system -> tunables

    kern.ipc.somaxconn = 32768

    And test again. We have seen some improvement using that setting

    Was the improvement seen during a TCP or UDP DDoS?


  • Banned

    Test using TCP scripts.



  • @KOM:

    It's like saying there is no point to lock your door because bank are robbed ?

    No, I'm saying that the strongest door you can find will happily collapse when it's being pounded on by a tank.

    If what Supermule is saying is correct, 70-80 Mbps is no tank.  It's like a spit wad pea shooter.

    If pfSense really can be taken down by that, that is a huge serious issue.

    @Supermule:

    Its in the OS. Hardware can easily handle it if you got some muscle.

    I can take this site offline using a specific type of traffic that takes no more than 70-80Mbps bandwith.

    When that traffic hits pfSense, its dead. Goes offline instantly. No matter how powerful the hardware is.

    I run 8 Core, 16GB ram and SSD. Dead in a second if it hits.


Locked