Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    What is the biggest attack in GBPS you stopped

    Scheduled Pinned Locked Moved General pfSense Questions
    737 Posts 33 Posters 600.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      firewalluser
      last edited by

      @Supermule:

      No….but maybe some updates to what they find or not find??

      Maybe hints to what could be done to minimize impact by adding things to system -> tunables??

      Have you considered that CMB is now under contract and cant disclose? This was something disclosed by Snowden, some individuals were forced/required to form a legal entity under guidance of the NSA.

      http://www.tomsguide.com/us/nsa-tech-coercion,news-17517.html

      Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

      Asch Conformity, mainly the blind leading the blind.

      1 Reply Last reply Reply Quote 0
      • S
        Supermule Banned
        last edited by

        HAHAHAHAHAHAHAHA :D

        If thats the case, then pfSense is dead as of THIS moment :D

        1 Reply Last reply Reply Quote 0
        • G
          gadnet
          last edited by

          @firewalluser:

          Some DDOS attacks can be nullified by simply changing the ip address(es) at the dns level.
          Where a DNS lookup is taking place, you need to identify the rogue who is doing the dns lookup and send them off to 23.37.28.215 or 195.99.147.120 if you have a sense of humour which contrary to popular belief also includes these guys 77.87.229.22.  ;D

          the issue is on a webserver with XX+ domains the udp attack do not show which one is targetted and also some domains are handled by the main branch of the customer of our customer's office in another country with days of business paperwork nonsense to finaly react and change the dns :)

          this is why i started to look at beeffy machines with pfsense to help but first i try to gather information about people using it for this and it seems no one, that answer here, use pfsense in multi gigabit setup or has experienced multi gigabit attacks on a pfsense box. I am happy thet they do not get attacked but i would have loved they had been to have some feedback ;p Supermule is providing feedback on "small scale" attack that would take down a firewall like this so i am not closer to any solution right now  (and still fight on DC side to get a POC setup) :)

          1 Reply Last reply Reply Quote 0
          • S
            Supermule Banned
            last edited by

            Lowprofile and I both have 10bgit setups in the production scenario and they are affected the same way.

            We dont need to run high bandwith attacks like DNS, NTP, SSDP or anything like that when the interesting stuff takes place when using small bandwith scripts that takes the firewall offline.

            When using little bandwith, then the attacker multiplies in numbers since they dont need 1gbit or more to take you offline. They only need a 50mbit pipe to do so.

            THATS the scary part!

            1 Reply Last reply Reply Quote 0
            • F
              firewalluser
              last edited by

              @Supermule:

              Could be. And yes its not supported any more.

              But we were testing…..

              Its possible to make the Windows core hang in some circumstances from the net even desktops behind fw's, but havent tested Win8 or later. Seen it on ubuntu 14.04 as well.

              @ghislain26:

              @firewalluser:

              Some DDOS attacks can be nullified by simply changing the ip address(es) at the dns level.
              Where a DNS lookup is taking place, you need to identify the rogue who is doing the dns lookup and send them off to 23.37.28.215 or 195.99.147.120 if you have a sense of humour which contrary to popular belief also includes these guys 77.87.229.22.  ;D

              the issue is on a webserver with XX+ domains the udp attack do not show which one is targetted and also some domains are handled by the main branch of the customer of our customer's office in another country with days of business paperwork nonsense to finaly react and change the dns :)

              this is why i started to look at beeffy machines with pfsense to help but first i try to gather information about people using it for this and it seems no one, that answer here, use pfsense in multi gigabit setup or has experienced multi gigabit attacks on a pfsense box. I am happy thet they do not get attacked but i would have loved they had been to have some feedback ;p Supermule is providing feedback on "small scale" attack that would take down a firewall like this so i am not closer to any solution right now  (and still fight on DC side to get a POC setup) :)

              I wonder if the L2 cache is causing a problem. Can this exploit be tried on a non AMD64 instruction set cpu if such a chip/device exists which can run pfsense and handle the bandwith? Its not something I can test on my RPi's sadly.  ;)

              http://www.lshift.net/blog/2013/10/08/cpu-cache-collisions-in-the-context-of-performance/

              Edit. A quick search suggests its not possible to switch off the L2 or any other cache now adays in the bios, but one way around this might be to limit it to a single core instead which will be doable on VM's like ESXI.

              When I suggest a non AMD64 cpu, please include the x86 32bit cpu's as well.

              I sadly can not participate in this little experiment due to only having 5MB download, but would be interested to see the data generated by the scripts none the less, so Supermule if you dont mind sharing the script via pm, I'd be curious to see what it generates to see what patterns are observed over a closed network between a couple of machines.

              FWIW.

              Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

              Asch Conformity, mainly the blind leading the blind.

              1 Reply Last reply Reply Quote 0
              • S
                Supermule Banned
                last edited by

                I run this in the test bench

                http://ark.intel.com/products/33927/Intel-Xeon-Processor-E5420-12M-Cache-2_50-GHz-1333-MHz-FSB

                12M L2 cache. I actually dont know how big the L1 cache is.

                1 Reply Last reply Reply Quote 0
                • S
                  Supermule Banned
                  last edited by

                  This is what I run in the datacenters pfsense clusters

                  http://ark.intel.com/products/47920/Intel-Xeon-Processor-X5670-12M-Cache-2_93-GHz-6_40-GTs-Intel-QPI

                  1 Reply Last reply Reply Quote 0
                  • K
                    kroberts
                    last edited by

                    Full disclosure:  I'm a hobbyist here, no multi-gigabit access.

                    You said this works on Linux too.  Is there common networking code between the two?

                    I would surely like to see a patch before this goes public, but IMO full disclosure to official channels might be a good option.  For example, https://www.freebsd.org/security/reporting.html if it's freebsd itself, or https://www.kb.cert.org/vuls/html/report-a-vulnerability/ if it's cross-platform.  Both offer encryption keys to send confidential information.

                    1 Reply Last reply Reply Quote 0
                    • H
                      Harvy66
                      last edited by

                      Anyone know what "kernel" is? When I was getting DDOS'd by SuperMule, that process seemed to be the offender. Prior to his attack, I have never seen that process. Just based off of this, kernel seems to be doing a lot of work that it probably doesn't need to be doing or is doing in a slow way.

                      1 Reply Last reply Reply Quote 0
                      • KOMK
                        KOM
                        last edited by

                        Probably a NIC driver hook into the kernel or something like that.  This may be helpful if anyone hasn't already seen it:

                        https://forums.freebsd.org/threads/high-cpu-interrupts-on-the-router-igb-driver-how-to-fix.28219/

                        1 Reply Last reply Reply Quote 0
                        • S
                          Supermule Banned
                          last edited by

                          First one is idle…Second one is during DoS

                          pfinfo_idle.PNG
                          pfinfo_idle.PNG_thumb
                          pfinfo_DoS.PNG
                          pfinfo_DoS.PNG_thumb

                          1 Reply Last reply Reply Quote 0
                          • H
                            Harvy66
                            last edited by

                            @KOM:

                            Probably a NIC driver hook into the kernel or something like that.  This may be helpful if anyone hasn't already seen it:

                            https://forums.freebsd.org/threads/high-cpu-interrupts-on-the-router-igb-driver-how-to-fix.28219/

                            The funny thing is my interrupts were low, it was "kernel" that was high. On average, my NIC interrupts consume about 130x more CPU than kernel, but during the DDOS, kernel was suddenly doing a lot of stuff. Even when load testing PFSense via WAN-LAN+NAT, I never see kernel. Normally interrupts are the number one cause of load on the firewall, which makes sense because it's just a ton of network IO.

                            I wonder what kernel is doing that it suddenly decides to do 10,000x more work than it normally does.

                            Poorly scaling algorithm?

                            1 Reply Last reply Reply Quote 0
                            • F
                              firewalluser
                              last edited by

                              @KOM:

                              Probably a NIC driver hook into the kernel or something like that.  This may be helpful if anyone hasn't already seen it:

                              https://forums.freebsd.org/threads/high-cpu-interrupts-on-the-router-igb-driver-how-to-fix.28219/

                              Possibly, but that doesnt explain how windows core & linux hangs when sat behind firewalls like pfsense which processes each packet before sending it inwards to the lan. Put another way it would seem pfsense can hang windows and linux machines sat behind them.

                              Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                              Asch Conformity, mainly the blind leading the blind.

                              1 Reply Last reply Reply Quote 0
                              • C
                                cmb
                                last edited by

                                @Supermule:

                                Lowprofile is also in this test scenario.

                                And he ended up with a config that stands up to said attacks. Edit: though further testing saw other problems.

                                I have packet captures of the traffic that's generated by the tools from when lowprofile ran it against one of our systems. It's nothing special that I saw, and we went through multiple types. If someone would like to provide additional pcaps, I'll definitely check them out.

                                @firewalluser:

                                Have you considered that CMB is now under contract and cant disclose? This was something disclosed by Snowden, some individuals were forced/required to form a legal entity under guidance of the NSA.

                                And now we're into conspiracy theories. No, that's not the case.

                                @Harvy66:

                                The funny thing is my interrupts were low, it was "kernel" that was high.

                                It's the queues of the NIC, not just kernel. That's where a good chunk of pf's processing will show. Where you're hitting the packet filter hard, that's what you will see.

                                @Supermule:

                                First one is idle…Second one is during DoS

                                You have polling enabled, which probably isn't a good idea (in theory it might help such circumstances, in practice especially in a VM it's probably not good). You're also running snort, and logging blocked DDoS traffic from the looks of it. All not good for DoS resiliency.

                                @firewalluser:

                                Put another way it would seem pfsense can hang windows and linux machines sat behind them.

                                You're passing enough attack traffic through to hang Windows and Linux in that case.

                                1 Reply Last reply Reply Quote 0
                                • L
                                  lowprofile
                                  last edited by

                                  My last test with 2.2.2 was a big failure. Right now i am struggling to get pfsense stable (2.2.2)
                                  But i did a test and it was not impressive. A standard configuration. After adjusting it was still not good enough at all. So now i will go trough all tweak and tuning again in 2.2.2…. re-google and start all over. the syn proxy feature should easily had solved this issue, but i will make further test and return. I tried with a general SYN proxy rule. I locked my self out from gui... no response, then tried again since it was replying ICMP request, but no difference. The SYN proxy feature should had handled this issue, but it is not working as it should. Same behaviour.

                                  Read this as well as i think some settings here would help. http://people.freebsd.org/~jlemon/papers/syncache.pdf
                                  I will return after some test.

                                  It would be nice if you could set a treshold for SYN proxy in general. e.x 100 half open connection pr. ip would trigger SYNproxy to be enabled....

                                  1 Reply Last reply Reply Quote 0
                                  • C
                                    cmb
                                    last edited by

                                    @lowprofile:

                                    My last test with 2.2.2 was a big failure. Right now i am struggling to get pfsense stable (2.2.2)

                                    Which test was that you were running? I have the traffic saved with the name of the test if it was one of the same named ones.

                                    I keep asking for pcaps and have gotten only what lowprofile helped me get. Supermule, anyone else, save the attack traffic to a pcap and put it somewhere I can download.

                                    Run something like the following:

                                    tcpdump -i em0 -s 0 -w ddos1.pcap 
                                    

                                    Where em0 is the source interface of the traffic. Replicate what you're seeing, then ctrl-c to break out of the tcpdump. SCP the file off and get it to me. Try not to make it too absurdly large, but a few GB is fine.

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      Supermule Banned
                                      last edited by

                                      Packet capture crashed and I have filed a crash report on the subject.

                                      EDIT: Managed getting it going. Get the capture here:

                                      http://bruksparken.com/log/files/packetcapture.zip

                                      1 Reply Last reply Reply Quote 0
                                      • F
                                        firewalluser
                                        last edited by

                                        @cmb:

                                        @firewalluser:

                                        Have you considered that CMB is now under contract and cant disclose? This was something disclosed by Snowden, some individuals were forced/required to form a legal entity under guidance of the NSA.

                                        And now we're into conspiracy theories. No, that's not the case.

                                        @firewalluser:

                                        Put another way it would seem pfsense can hang windows and linux machines sat behind them.

                                        You're passing enough attack traffic through to hang Windows and Linux in that case.

                                        I'm sure you are familiar with the wide ranging contracts that exist in the world today beit Non Disclosure Agreements, or even the more common non compete agreements as exampled here: http://pando.com/2014/03/22/revealed-apple-and-googles-wage-fixing-cartel-involved-dozens-more-companies-over-one-million-employees/
                                        http://www.businessinsider.com/emails-eric-schmidt-sergey-brin-hiring-apple-2014-3?IR=T
                                        http://venturebeat.com/2014/05/23/4-tech-companies-are-paying-a-325m-fine-for-their-illegal-non-compete-pact/

                                        Put simply, you are not in a position to prove your innocence, because

                                        1. To adhere to the terms of any NDA contract you may have been forced/coerced to sign would mean any disclosure would render you in breach of your contract and liable to whatever penalties may have been included in any agreement and who in their "right" mind would put themselves at a disadvantage?

                                        2. Even if you have not signed any NDA contract you still cant prove your innocence, ergo the spooks & govt(s) still win, its classic divide and conquer techniques, which then begs the question why trust military & govt(s) or banks who carry out activities in secret?

                                        What I can say is trust can take ages to build up, but can be destroyed in seconds.

                                        On the point of passing enough traffic through pfsense, this has happened with less than 1mbits of traffic, a simple web page loading can trigger the OS cores to hang. Volume is irrelevant in the example I mention, but in relation to this thread and amounts of data, I wondered two things, exploiting the CPU designs namely cache and/or something network related as also mentioned by Kom here
                                        https://forum.pfsense.org/index.php?topic=91856.msg517296#msg517296

                                        I'm inclined at this stage to err towards something nic related but I will examine the zip posted by supermule to see if I can see anything untoward, but this could be a variation on the heap spraying exploit http://en.wikipedia.org/wiki/Heap_spraying

                                        I wonder if those affected are running snort and if so do the problems still exist, assuming snort is already aware of the problem much like AV software need to have found a virus before it can protect against it?

                                        All of the above is said with the best of intentions and for it to be educational to those who might not be aware of the deceit and duplicity in the world today.

                                        Edit.

                                        Has anyone tried an earlier version of pfsense like a 1.x version by any chance? ;)

                                        Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                                        Asch Conformity, mainly the blind leading the blind.

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          Supermule Banned
                                          last edited by

                                          M0n0wall suffers the same.

                                          I am running Snort.

                                          I can easily test without it. 2mins.

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            Supermule Banned
                                            last edited by

                                            Download it here.

                                            No Snort running, but same behaviour.

                                            http://bruksparken.com/log/files/packetcapture_no_snort.zip

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.