Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    What is the biggest attack in GBPS you stopped

    Scheduled Pinned Locked Moved
    General pfSense Questions
    33
    737
    507.6k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      Supermule Banned
      last edited by

      Done it at the console at no useful output was generated for people to see.

      I stopped Snort running and here is the output from the DoS.

      First 2 is idle and next 2 is under DoS.

      PS_idle.PNG
      PS_idle.PNG_thumb
      netstat_idle.PNG
      netstat_idle.PNG_thumb
      PS_DoS.PNG
      PS_DoS.PNG_thumb
      netstat_DoS.PNG
      netstat_DoS.PNG_thumb

      1 Reply Last reply Reply Quote 0
      • S
        Supermule Banned
        last edited by

        Done some more testing this morning.

        2-3mbps is all it takes. Has downscaled the Mbufs and state max a little.

        http://youtu.be/NPtDnM8ixXs

        Dennypage. Thanks for the info. Want to help diagnose then contact me on PM.

        1 Reply Last reply Reply Quote 0
        • T
          tim.mcmanus
          last edited by

          This link is probably important to note the differences between versions:  https://doc.pfsense.org/index.php/Does_pfSense_support_SMP_(multi-processor_and/or_core)_systems

          2.1 was single-threaded and 2.2 is multi-threaded.  That's why you're seeing an impact/performance difference between the two; it's not hard to extrapolate how and why.

          I think what you're trying to determine, and this is based on my review of the thread, is which part of pf is choking.  In order to determine this you need to debug each component in the chain from the NIC to the CPU and back out as well as the code.  I'm not entirely sure you know programmatically where and which networking event triggers the issue inside pf, only that a large volume of data of a specific type starts the event.

          You've moved beyond evaluating pf from a networking perspective and more into evaluating the codebase.  This requires a different kind of data collection and troubleshooting.  It also take an excruciatingly long time to identify and resolve these kinds of issues.  It's a lot more than just tweaking a setting in some cases.

          Best of luck in determining the root cause and solution to this issue.

          1 Reply Last reply Reply Quote 0
          • S
            Supermule Banned
            last edited by

            Thanks Tim.

            You are 100% correct.

            1 Reply Last reply Reply Quote 0
            • K
              kejianshi
              last edited by

              Silly question - Is it possible to set a max cpu % that may be used by the packet filter?  Keep some in reserve for other processes?

              1 Reply Last reply Reply Quote 0
              • H
                Harvy66
                last edited by

                Someone needs to use DTrace and make a flame-graph of what methods are being called in the kernel.

                1 Reply Last reply Reply Quote 0
                • S
                  Supermule Banned
                  last edited by

                  Shall we test again Harvy??

                  1 Reply Last reply Reply Quote 0
                  • H
                    Harvy66
                    last edited by

                    I don't know how to do flame graphs, I've only seen them in Netflix presentations talking about optimizing FreeBSD.

                    1 Reply Last reply Reply Quote 0
                    • S
                      Supermule Banned
                      last edited by

                      Tested Fortigate Virtual Server and after enabling Flood Protection, it ran perfectly during all tests.

                      1CPU and 1GB RAM in a VmWare VM.

                      Had an email conversation with a guy named Dave Huffman and he was able to replicate the scenario but only using DoS and not DDoS. Not that important, but it seems pfSense is not able to handle legitimate traffic vs. offending IP's.

                      It chokes somewhere in the stack.

                      1 Reply Last reply Reply Quote 0
                      • N
                        Nullity
                        last edited by

                        Do you need help figuring out how to enable profiling or some other debugging software?

                        Please correct any obvious misinformation in my posts.
                        -Not a professional; an arrogant ignoramous.

                        1 Reply Last reply Reply Quote 0
                        • S
                          Supermule Banned
                          last edited by

                          Yes because I need to get to the bottom of this.

                          1 Reply Last reply Reply Quote 0
                          • T
                            tim.mcmanus
                            last edited by

                            I would start here.  http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug.html

                            You can do remote kernel debugging as an option.  It's not for the faint of heart.  Debugging never is.

                            1 Reply Last reply Reply Quote 0
                            • F
                              firewalluser
                              last edited by

                              @Supermule:

                              Yes because I need to get to the bottom of this.

                              Try a USB nic on the wan, see how the data is handled differently.

                              You'll find these useful as well.
                              https://software.intel.com/sites/default/files/profiling_debugging_freebsd_kernel_321772.pdf
                              https://2008.asiabsdcon.org/papers/P8B-paper.pdf

                              Mutexs can catch some people out, but they are just locks to ensure the code doesnt deadlock in a multicore environment.

                              IMO profiling is better than retrospectively debugging crash dumps as you can make the crash dumps misleading in some situations masking the real root cause of the problem. I've found bugs in programming languages that have existed for over 15 years, thats how difficult some of these bugs are to find, even though it took me less than a week to find, theres a lot of exposed software out there.

                              Main difficultly are multi cores when it comes to debugging, you could be looking at the code running on one core whilst a bug in code running on another core creates the problem which crashes the code running on the core you are looking at. You can mask some problems by running on a single core but you will still have the problem, just less often as these things are just inglorious clockwork turkmachines.

                              Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                              Asch Conformity, mainly the blind leading the blind.

                              1 Reply Last reply Reply Quote 0
                              • S
                                Supermule Banned
                                last edited by

                                Yes but doesnt crash as in crash….

                                It just goes to a standstill and is unresponsive. You dont see anything on the console and in the logs besides excessive traffic.

                                I dont see any queueing on the NIC's as well so its pretty odd.

                                1 Reply Last reply Reply Quote 0
                                • F
                                  firewalluser
                                  last edited by

                                  Maybe something is getting out of order, perhaps a queue/buffer is being processed LIFO when it should be FIFO which would generally only show up under load considering the speeds of todays CPU's.

                                  Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                                  Asch Conformity, mainly the blind leading the blind.

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    Supermule Banned
                                    last edited by

                                    Could be.

                                    The fact is that it cannot sort traffic in legitimate and non-legit traffic.

                                    Its pretty weird that a lot of excess ressources is not used on the webserver to keep the GUI alive at least.

                                    When the flood control setting is activated in the Fortinet VM appliance then it can handle everything we threw at it.

                                    It can sort traffic at wirespeed, but pfsense cannot.

                                    It sortof feels like pfsense is responding to packets that shouldnt get a response and holds everything else…

                                    I dont know where to start.

                                    A guy called Dave Huffmann contacted and has written a script that can replicate most of what we see. I dont know if he has come up with an answer yet on whats causing it to slow down.

                                    I am yet to receive news from him.

                                    1 Reply Last reply Reply Quote 0
                                    • H
                                      Harvy66
                                      last edited by

                                      Lookup flame graphs and Netflix. They're very useful for looking for offending code paths.

                                      1 Reply Last reply Reply Quote 0
                                      • F
                                        firewalluser
                                        last edited by

                                        http://techblog.netflix.com/2014/11/nodejs-in-flames.html

                                        Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                                        Asch Conformity, mainly the blind leading the blind.

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          Supermule Banned
                                          last edited by

                                          Looks really interesting and is a good read.

                                          How to impelement it in pfsense??

                                          Can this be implemented somehow in a package?

                                          http://www.brendangregg.com/blog/2015-03-10/freebsd-flame-graphs.html

                                          There is code in there…

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            Supermule Banned
                                            last edited by

                                            This is how my logs look like…
                                            May 14 14:27:51 php-fpm[1097]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                                            May 14 14:27:41 php-fpm[1097]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                                            May 14 14:27:29 php-fpm[71742]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                                            May 14 14:27:19 php-fpm[71742]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                                            May 14 14:27:07 php-fpm[71510]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                                            May 14 14:26:56 php-fpm[71510]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                                            May 14 14:26:45 php-fpm[53920]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                                            May 14 14:26:35 php-fpm[53920]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                                            May 14 14:26:23 php-fpm[1097]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                                            May 14 14:26:13 php-fpm[1097]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                                            May 14 14:26:01 php-fpm[71742]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                                            May 14 14:25:51 php-fpm[71742]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                                            May 14 14:25:39 php-fpm[71510]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                                            May 14 14:25:29 php-fpm[71510]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                                            May 14 14:25:27 check_reload_status: Reloading filter
                                            May 14 14:25:27 check_reload_status: Restarting OpenVPN tunnels/interfaces
                                            May 14 14:25:27 check_reload_status: Restarting ipsec tunnels
                                            May 14 14:25:27 check_reload_status: updating dyndns Yousee
                                            May 14 14:25:23 check_reload_status: Reloading filter
                                            May 14 14:25:23 check_reload_status: Restarting OpenVPN tunnels/interfaces
                                            May 14 14:25:23 check_reload_status: Restarting ipsec tunnels
                                            May 14 14:25:23 check_reload_status: updating dyndns Yousee
                                            May 14 14:25:17 php-fpm[53920]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                                            May 14 14:25:13 check_reload_status: Reloading filter
                                            May 14 14:25:13 check_reload_status: Restarting OpenVPN tunnels/interfaces
                                            May 14 14:25:13 check_reload_status: Restarting ipsec tunnels
                                            May 14 14:25:13 check_reload_status: updating dyndns Yousee
                                            May 14 14:25:11 check_reload_status: Reloading filter
                                            May 14 14:25:11 check_reload_status: Restarting OpenVPN tunnels/interfaces
                                            May 14 14:25:11 check_reload_status: Restarting ipsec tunnels
                                            May 14 14:25:11 check_reload_status: updating dyndns Yousee
                                            May 14 14:25:03 php-fpm[53920]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                                            May 14 14:24:55 check_reload_status: Reloading filter
                                            May 14 14:24:55 check_reload_status: Restarting OpenVPN tunnels/interfaces
                                            May 14 14:24:55 check_reload_status: Restarting ipsec tunnels
                                            May 14 14:24:55 check_reload_status: updating dyndns Yousee

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post