Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    What is the biggest attack in GBPS you stopped

    Scheduled Pinned Locked Moved General pfSense Questions
    737 Posts 33 Posters 603.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      tim.mcmanus
      last edited by

      I would start here.  http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug.html

      You can do remote kernel debugging as an option.  It's not for the faint of heart.  Debugging never is.

      1 Reply Last reply Reply Quote 0
      • F
        firewalluser
        last edited by

        @Supermule:

        Yes because I need to get to the bottom of this.

        Try a USB nic on the wan, see how the data is handled differently.

        You'll find these useful as well.
        https://software.intel.com/sites/default/files/profiling_debugging_freebsd_kernel_321772.pdf
        https://2008.asiabsdcon.org/papers/P8B-paper.pdf

        Mutexs can catch some people out, but they are just locks to ensure the code doesnt deadlock in a multicore environment.

        IMO profiling is better than retrospectively debugging crash dumps as you can make the crash dumps misleading in some situations masking the real root cause of the problem. I've found bugs in programming languages that have existed for over 15 years, thats how difficult some of these bugs are to find, even though it took me less than a week to find, theres a lot of exposed software out there.

        Main difficultly are multi cores when it comes to debugging, you could be looking at the code running on one core whilst a bug in code running on another core creates the problem which crashes the code running on the core you are looking at. You can mask some problems by running on a single core but you will still have the problem, just less often as these things are just inglorious clockwork turkmachines.

        Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

        Asch Conformity, mainly the blind leading the blind.

        1 Reply Last reply Reply Quote 0
        • S
          Supermule Banned
          last edited by

          Yes but doesnt crash as in crash….

          It just goes to a standstill and is unresponsive. You dont see anything on the console and in the logs besides excessive traffic.

          I dont see any queueing on the NIC's as well so its pretty odd.

          1 Reply Last reply Reply Quote 0
          • F
            firewalluser
            last edited by

            Maybe something is getting out of order, perhaps a queue/buffer is being processed LIFO when it should be FIFO which would generally only show up under load considering the speeds of todays CPU's.

            Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

            Asch Conformity, mainly the blind leading the blind.

            1 Reply Last reply Reply Quote 0
            • S
              Supermule Banned
              last edited by

              Could be.

              The fact is that it cannot sort traffic in legitimate and non-legit traffic.

              Its pretty weird that a lot of excess ressources is not used on the webserver to keep the GUI alive at least.

              When the flood control setting is activated in the Fortinet VM appliance then it can handle everything we threw at it.

              It can sort traffic at wirespeed, but pfsense cannot.

              It sortof feels like pfsense is responding to packets that shouldnt get a response and holds everything else…

              I dont know where to start.

              A guy called Dave Huffmann contacted and has written a script that can replicate most of what we see. I dont know if he has come up with an answer yet on whats causing it to slow down.

              I am yet to receive news from him.

              1 Reply Last reply Reply Quote 0
              • H
                Harvy66
                last edited by

                Lookup flame graphs and Netflix. They're very useful for looking for offending code paths.

                1 Reply Last reply Reply Quote 0
                • F
                  firewalluser
                  last edited by

                  http://techblog.netflix.com/2014/11/nodejs-in-flames.html

                  Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                  Asch Conformity, mainly the blind leading the blind.

                  1 Reply Last reply Reply Quote 0
                  • S
                    Supermule Banned
                    last edited by

                    Looks really interesting and is a good read.

                    How to impelement it in pfsense??

                    Can this be implemented somehow in a package?

                    http://www.brendangregg.com/blog/2015-03-10/freebsd-flame-graphs.html

                    There is code in there…

                    1 Reply Last reply Reply Quote 0
                    • S
                      Supermule Banned
                      last edited by

                      This is how my logs look like…
                      May 14 14:27:51 php-fpm[1097]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                      May 14 14:27:41 php-fpm[1097]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                      May 14 14:27:29 php-fpm[71742]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                      May 14 14:27:19 php-fpm[71742]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                      May 14 14:27:07 php-fpm[71510]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                      May 14 14:26:56 php-fpm[71510]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                      May 14 14:26:45 php-fpm[53920]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                      May 14 14:26:35 php-fpm[53920]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                      May 14 14:26:23 php-fpm[1097]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                      May 14 14:26:13 php-fpm[1097]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                      May 14 14:26:01 php-fpm[71742]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                      May 14 14:25:51 php-fpm[71742]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                      May 14 14:25:39 php-fpm[71510]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                      May 14 14:25:29 php-fpm[71510]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                      May 14 14:25:27 check_reload_status: Reloading filter
                      May 14 14:25:27 check_reload_status: Restarting OpenVPN tunnels/interfaces
                      May 14 14:25:27 check_reload_status: Restarting ipsec tunnels
                      May 14 14:25:27 check_reload_status: updating dyndns Yousee
                      May 14 14:25:23 check_reload_status: Reloading filter
                      May 14 14:25:23 check_reload_status: Restarting OpenVPN tunnels/interfaces
                      May 14 14:25:23 check_reload_status: Restarting ipsec tunnels
                      May 14 14:25:23 check_reload_status: updating dyndns Yousee
                      May 14 14:25:17 php-fpm[53920]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
                      May 14 14:25:13 check_reload_status: Reloading filter
                      May 14 14:25:13 check_reload_status: Restarting OpenVPN tunnels/interfaces
                      May 14 14:25:13 check_reload_status: Restarting ipsec tunnels
                      May 14 14:25:13 check_reload_status: updating dyndns Yousee
                      May 14 14:25:11 check_reload_status: Reloading filter
                      May 14 14:25:11 check_reload_status: Restarting OpenVPN tunnels/interfaces
                      May 14 14:25:11 check_reload_status: Restarting ipsec tunnels
                      May 14 14:25:11 check_reload_status: updating dyndns Yousee
                      May 14 14:25:03 php-fpm[53920]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
                      May 14 14:24:55 check_reload_status: Reloading filter
                      May 14 14:24:55 check_reload_status: Restarting OpenVPN tunnels/interfaces
                      May 14 14:24:55 check_reload_status: Restarting ipsec tunnels
                      May 14 14:24:55 check_reload_status: updating dyndns Yousee

                      1 Reply Last reply Reply Quote 0
                      • T
                        tim.mcmanus
                        last edited by

                        @Supermule:

                        Looks really interesting and is a good read.

                        How to impelement it in pfsense??

                        Can this be implemented somehow in a package?

                        http://www.brendangregg.com/blog/2015-03-10/freebsd-flame-graphs.html

                        There is code in there…

                        IMHO, I wouldn't put this into a package or integrate it into pfSense.  It's a development debugging tool that ideally should be installed on dev or test machines.

                        1 Reply Last reply Reply Quote 0
                        • S
                          Supermule Banned
                          last edited by

                          Hmmm.

                          EDIT:

                          Thinking it would work as a great tool in debugging issues related to pfsense and give the IT-admins a better insight of whats going on in their environments.

                          1 Reply Last reply Reply Quote 0
                          • L
                            lowprofile
                            last edited by

                            Hi guys

                            Just want to share some info.

                            I am going away from pfsense. PFsense is a great firewall but all these buggy versions out there have cost me $$$$ - 2.2* has been so unstable and i've lost the trust to pfsense.

                            Regarding DDoS, pfsense cannot handle a simple flood SYN. In 2.2* it got more worse. You can spend many weeks in tuning, tweaking etc, but then you will then have a system which is unreliable at the end. Too much core changing.

                            I managed to get it somehow 80% resistent to SYN floods in 2.1.5, but it had its sideaffects. I now experienced unstability generel.

                            I've tried fortigate VM appliance with 1gb ram and 1core (trial) - i was surprised how stable it was with same hardware (virtual) You have a special option to block SYN/ICMP/FIN etc floods. Very simple option. See screenshot.

                            I used 10min to install it and further 10min to set it up. Activated the ddos policy. and bingo i had a stable setup. I know fortigate cost much more, but most of the appliances are built on linux or freebsd. I have concluded the pfsense does lack this crucial "feature" and protection.

                            No more packet drop, even the attack was on +100mbit SYN flood. Very stable, not a single drop in ping.
                            Sad to say, but this has proven the source to the issue = PFsense.

                            I am now investing in a proper firewall. VM or box, doesnt matter, it just wont be PFsense. I liked pfsense untill i got these issues and some serious stability issues in newer versions. Time to move on.  ;)

                            ddosfortigate.PNG
                            ddosfortigate.PNG_thumb

                            1 Reply Last reply Reply Quote 0
                            • N
                              Nullity
                              last edited by

                              @lowprofile:

                              Hi guys

                              Just want to share some info.

                              I am going away from pfsense. PFsense is a great firewall but all these buggy versions out there have cost me $$$$ - 2.2* has been so unstable and i've lost the trust to pfsense.

                              Regarding DDoS, pfsense cannot handle a simple flood SYN. In 2.2* it got more worse. You can spend many weeks in tuning, tweaking etc, but then you will then have a system which is unreliable at the end. Too much core changing.

                              I managed to get it somehow 80% resistent to SYN floods in 2.1.5, but it had its sideaffects. I now experienced unstability generel.

                              I've tried fortigate VM appliance with 1gb ram and 1core (trial) - i was surprised how stable it was with same hardware (virtual) You have a special option to block SYN/ICMP/FIN etc floods. Very simple option. See screenshot.

                              I used 10min to install it and further 10min to set it up. Activated the ddos policy. and bingo i had a stable setup. I know fortigate cost much more, but most of the appliances are built on linux or freebsd. I have concluded the pfsense does lack this crucial "feature" and protection.

                              No more packet drop, even the attack was on +100mbit SYN flood. Very stable, not a single drop in ping.
                              Sad to say, but this has proven the source to the issue = PFsense.

                              I am now investing in a proper firewall. VM or box, doesnt matter, it just wont be PFsense. I liked pfsense untill i got these issues and some serious stability issues in newer versions. Time to move on.  ;)

                              I love you to bro.

                              Thanks for trying to spread some negativity and get a pfSense vs Fortigate fight going on your way out. You will be missed.

                              Please correct any obvious misinformation in my posts.
                              -Not a professional; an arrogant ignoramous.

                              1 Reply Last reply Reply Quote 0
                              • N
                                NOYB
                                last edited by

                                @lowprofile:

                                Hi guys

                                Just want to share some info.

                                I am going away from pfsense. PFsense is a great firewall but all these buggy versions out there have cost me $$$$ - 2.2* has been so unstable and i've lost the trust to pfsense.

                                Regarding DDoS, pfsense cannot handle a simple flood SYN. In 2.2* it got more worse. You can spend many weeks in tuning, tweaking etc, but then you will then have a system which is unreliable at the end. Too much core changing.

                                I managed to get it somehow 80% resistent to SYN floods in 2.1.5, but it had its sideaffects. I now experienced unstability generel.

                                I've tried fortigate VM appliance with 1gb ram and 1core (trial) - i was surprised how stable it was with same hardware (virtual) You have a special option to block SYN/ICMP/FIN etc floods. Very simple option. See screenshot.

                                I used 10min to install it and further 10min to set it up. Activated the ddos policy. and bingo i had a stable setup. I know fortigate cost much more, but most of the appliances are built on linux or freebsd. I have concluded the pfsense does lack this crucial "feature" and protection.

                                No more packet drop, even the attack was on +100mbit SYN flood. Very stable, not a single drop in ping.
                                Sad to say, but this has proven the source to the issue = PFsense.

                                I am now investing in a proper firewall. VM or box, doesnt matter, it just wont be PFsense. I liked pfsense untill i got these issues and some serious stability issues in newer versions. Time to move on.  ;)

                                If I were using pfSense in a business environment I'd be right behind you.

                                That a disgruntled employee, dissatisfied customer, or unscrupulous competitor, could take a business behind pfSense offline with such a small amount of traffic would be a really scary and unacceptable risk.  And that the pfSense team doesn't really seem to be very engaged in figuring it out so it can be fixed doesn't instill any confidence that it will be fixed anytime soon.  In fact it indicates that they either have no idea what's causing the issue, or that they do know and know there is no timely fix on the horizon.  So down play it.  Just multiplies the sentiment.

                                1 Reply Last reply Reply Quote 0
                                • H
                                  Harvy66
                                  last edited by

                                  You may want to re-evaluate PFSense in the future. Big performance changes in 3.0, hopefully this stuff will get fixed and it will be done with.

                                  1 Reply Last reply Reply Quote 0
                                  • N
                                    Nullity
                                    last edited by

                                    @NOYB:

                                    If I were using pfSense in a business environment I'd be right behind you.

                                    That a disgruntled employee, dissatisfied customer, or unscrupulous competitor, could take a business behind pfSense offline with such a small amount of traffic would be a really scary and unacceptable risk.  And that the pfSense team doesn't really seem to be very engaged in figuring it out so it can be fixed doesn't instill any confidence that it will be fixed anytime soon.  In fact it indicates that they either have no idea what's causing the issue, or that they do know and know there is no timely fix on the horizon.  So down play it.  Just multiplies the sentiment.

                                    I have been assuming that this is not actually a problem, and that FreeBSD/pfSense is fully capable of withstanding this attack if configured properly.

                                    Honestly, when I first found pfSense I bought into the "omg, SuperNetAdmin must use this!", but after a few months, the GUI's limitations were obvious even to a networking newbie like me.

                                    I like this community though. I hope it doesn't crumble…

                                    Please correct any obvious misinformation in my posts.
                                    -Not a professional; an arrogant ignoramous.

                                    1 Reply Last reply Reply Quote 0
                                    • N
                                      NOYB
                                      last edited by

                                      I wouldn't run a business network on assumptions.

                                      1 Reply Last reply Reply Quote 0
                                      • H
                                        Harvy66
                                        last edited by

                                        FreeBSD is a good platform. Even PFSense moves forward slowly, as long as it keeps moving forward. It works good enough for me. If I ever stopped using PFSense, I'd probably just switch to FreeBSD/PCBSD and learn how to configure things directly or use packages if they exist.

                                        1 Reply Last reply Reply Quote 0
                                        • T
                                          tim.mcmanus
                                          last edited by

                                          I run pfSense on multiple WANs and LANs on the same box for my business.  I'm also hosting multiple web servers and mail servers.  Never once taken down by any attacks.

                                          What are you doing to subject yourself to these kinds of attacks, and why hasn't your ISP done anything to mitigate them?

                                          I used to work for an MSP that resold and supported FortiGates and thought they sucked.  I tore most of them out and replaced them with pfSense.

                                          I just received some additional equipment to build a Security Onion appliance that I'm going to mirror two WAN ports into.  I could easily integrate Snort on pfSense to use the barnyard database on the SO appliance to most likely mitigate the whole SYN attack.  After I'm done building it, it would be interesting to test.  Plus I'd be able to capture and inspect every packet coming in.  I'm also interested to see if it takes out the MikroTik switch in front of pfSense first.

                                          1 Reply Last reply Reply Quote 0
                                          • N
                                            NOYB
                                            last edited by

                                            @tim.mcmanus:

                                            What are you doing to subject yourself to these kinds of attacks, and why hasn't your ISP done anything to mitigate them?

                                            Wow!  Yeah it must be the attackie's fault.  After all certainly no one would do such a thing without provocation.

                                            • Certainly if a business has an employee that becomes disgruntled it is without a doubt the businesses fault.
                                            • Certainly if a business has a customer that becomes dissatisfied it is without a doubt the businesses fault.
                                            • Certainly if a business has an unscrupulous competitor it is without a doubt the businesses fault.

                                            Really?  You expect a business to rely on ISP to protect against a low bandwidth attack such as this.  A business could be down for days before being able to get an ISP to take meaningful action.  Sure hope that it is not pfSense position that an ISP should protect a business from such a low bandwidth attack so their product doesn't have to.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.