Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    What is the biggest attack in GBPS you stopped

    Scheduled Pinned Locked Moved General pfSense Questions
    737 Posts 33 Posters 602.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      Supermule Banned
      last edited by

      1: Yes.

      2: Picture attached.

      3: From ISO directly on to the ESXi

      4: Homeserver (Windows 2008 R2)

      pfsense_advanced_networking.PNG
      pfsense_advanced_networking.PNG_thumb

      1 Reply Last reply Reply Quote 0
      • S
        Supermule Banned
        last edited by

        Disabling Apinger so the interface doesnt get restarted all the time during an attack.

        IN the traffic graph you are able to see the drop in traffic after the ifconfig em0 -promisc reload in cron.

        That makes the firewall come alive and start routing packets again.

        I have attached the screenshot before and after the reload as running top -HSP

        traffic_drop.PNG
        traffic_drop.PNG_thumb
        ![top -HSP before reload of -promisc.PNG](/public/imported_attachments/1/top -HSP before reload of -promisc.PNG)
        ![top -HSP before reload of -promisc.PNG_thumb](/public/imported_attachments/1/top -HSP before reload of -promisc.PNG_thumb)
        ![top -HSP after reload of -promisc.PNG](/public/imported_attachments/1/top -HSP after reload of -promisc.PNG)
        ![top -HSP after reload of -promisc.PNG_thumb](/public/imported_attachments/1/top -HSP after reload of -promisc.PNG_thumb)

        1 Reply Last reply Reply Quote 0
        • F
          firewalluser
          last edited by

          In your pic, the first ticked checkbox is normally unticked for default.

          Have you been toggling these?
          If so notice any difference?

          Have you been through the pfsense 2 on VMware ESXI 5.5 pfsense docs to check settings?

          When you run the DDOS is homeserver running as well? I know thats your aim ultimately, but does pfsense perform better without it running?

          Do you have the Sata driver installed? Check out airvpn.org/topic/11847-pfsense-performance-configs-on-esxi-vmware/
          that might be a lead?

          Hows your management channel setup?

          Apologies if you have posted your VM settings, I dont recall seeing them, but if you havent can you post them as its a case of trying to see if thats been setup properly and not causing the problem which is making pfsense fail under the ddos. We cant rule the ESXi VM guest settings out just yet imo.

          Got to go out for a couple hours now, but its definately worth going back over all the settings.

          Have you even setup a basic pfsense with minimal settings, no packages, no config changes other than ip address changes for nics to see how that copes with the DDOS?

          I think this is a back to basics moment like others have suggested, although I know baremetal isnt an option, but making sure the guest is configured right and then installing a basic pfsense installation would be my next move. If that handles the DDOS, I'd pull the XML backups and compare differences as its easy to miss something when toggling various settings in situations like this.

          Good luck!  :)

          @Supermule:

          Disabling Apinger so the interface doesnt get restarted all the time during an attack.

          IN the traffic graph you are able to see the drop in traffic after the ifconfig em0 -promisc reload in cron.

          That makes the firewall come alive and start routing packets again.

          I have attached the screenshot before and after the reload as running top -HSP

          Although I could only get 2.42Mbps, apinger was still getting out for me as its only got to ping some ip addresses a couple hops away unlike your ddos traffic which is coming from all around the world and thus further away. The network infrastructure would let ip addresses closer to me get through as the bottle necks would be further away.

          Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

          Asch Conformity, mainly the blind leading the blind.

          1 Reply Last reply Reply Quote 0
          • S
            Supermule Banned
            last edited by

            @firewalluser:

            In your pic, the first ticked checkbox is normally unticked for default.

            Have you been toggling these?
            If so notice any difference?

            Not seen any difference at all.

            Have you been through the pfsense 2 on VMware ESXI 5.5 pfsense docs to check settings?

            Yes

            When you run the DDOS is homeserver running as well?

            Yes.

            I know thats your aim ultimately, but does pfsense perform better without it running?

            No difference.

            Do you have the Sata driver installed?

            No running on a scsi controller.

            Hows your management channel setup?

            Not understood. Using vsphere client, console and Putty.

            Apologies if you have posted your VM settings, I dont recall seeing them, but if you havent can you post them as its a case of trying to see if thats been setup properly and not causing the problem which is making pfsense fail under the ddos. We cant rule the ESXi VM guest settings out just yet imo.

            Youtube Video

            Got to go out for a couple hours now, but its definately worth going back over all the settings.

            Enjoy.

            Have you even setup a basic pfsense with minimal settings, no packages, no config changes other than ip address changes for nics to see how that copes with the DDOS?

            Yes. It didnt do very well.

            I think this is a back to basics moment like others have suggested, although I know baremetal isnt an option, but making sure the guest is configured right and then installing a basic pfsense installation would be my next move. If that handles the DDOS, I'd pull the XML backups and compare differences as its easy to miss something when toggling various settings in situations like this.

            I haves asked Tim and Almabes if they have any available. Apparently it does make any difference.

            Good luck!  :)

            Thanks :)

            1 Reply Last reply Reply Quote 0
            • S
              Supermule Banned
              last edited by

              Any way to monitor cron jobs in real time??

              Tried crontab -l but it says it cannot find any for the user root…

              Tried changing it to crontab - admin -l but that doesnt work either.

              I want to have a console running so I can see specifically when cron is run since it doesnt say anything in the system logs.

              This is a real bitch to trouble shoot internally.................................

              1 Reply Last reply Reply Quote 0
              • M
                mer
                last edited by

                cat /etc/crontab
                log for cron is typically /var/log/cron

                Looks like there is also a few things run by "minicron", look at /etc/rc:

                rc:cd /tmp && /usr/sbin/cron -s 2>/dev/null
                rc:/usr/local/bin/minicron 240 $varrunpath/ping_hosts.pid /usr/local/bin/ping_hosts.sh
                rc:/usr/local/bin/minicron 3600 $varrunpath/expire_accounts.pid '/usr/local/sbin/fcgicli -f /etc/rc.expireaccounts'
                rc:/usr/local/bin/minicron 86400 $varrunpath/update_alias_url_data.pid '/usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data'
                rc:    /usr/local/bin/minicron 60 /var/run/gmirror_status_check.pid /usr/local/sbin/gmirror_status_check.php

                1 Reply Last reply Reply Quote 0
                • F
                  firewalluser
                  last edited by

                  Hows your management channel setup?

                  Not understood. Using vsphere client, console and Putty."

                  Either in the pfsense docs or the AirVPN link it mentions having the management channel setup a particular way, might be worth checking out.

                  Are you still getting the massive waits?

                  www.yellow-bricks.com/2012/07/17/why-is-wait-so-high/

                  Your %wait figure might be a red herring as we say.  :)

                  Basically the link suggests the %wait includes idle time and %vmwait might be a better figure.

                  %wait is like MS including CPU idle time in the CPU processor load.  ::)

                  Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                  Asch Conformity, mainly the blind leading the blind.

                  1 Reply Last reply Reply Quote 0
                  • T
                    tim.mcmanus
                    last edited by

                    @firewalluser:

                    Hows your management channel setup?

                    Not understood. Using vsphere client, console and Putty."

                    Either in the pfsense docs or the AirVPN link it mentions having the management channel setup a particular way, might be worth checking out.

                    Are you still getting the massive waits?

                    www.yellow-bricks.com/2012/07/17/why-is-wait-so-high/

                    Your %wait figure might be a red herring as we say.  :)

                    Basically the link suggests the %wait includes idle time and %vmwait might be a better figure.

                    %wait is like MS including CPU idle time in the CPU processor load.  ::)

                    %wait is very important.

                    Network->NIC->hypervisor Kernel->VM->VM NIC->VM kernel (and then back down the stack to move a packet)

                    If the hypervisor kernel is consuming resources, the VM isn't going to get any.  So your VM wait time will be high, but the VM kernel activity will be low.  You could also be dropping packets prior to getting to the VM because of high wait times.  This is why enterprises regularly go through right-sizing activities.  You'll see low CPU utilization on your VMs but very high wait times, and your performance will mysteriously suck.

                    1 Reply Last reply Reply Quote 0
                    • F
                      firewalluser
                      last edited by

                      Have you tried earlier versions of ESXi and/or pfsense?

                      Are you still on pfsense 2.1 or did you try 2.2?

                      It might be a build conflict somewhere, maybe even at the bios level. I've seen bios make some hw drag.

                      Can you try same setup on different hw maybe with a different provider?

                      Might be worth spinning something up on the amazon cloud to test although thats got its own custom build of pfsense, so maybe have a look at those settings to see what differences there are. You might get some clues from that as to what settings are important.

                      Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                      Asch Conformity, mainly the blind leading the blind.

                      1 Reply Last reply Reply Quote 0
                      • S
                        Supermule Banned
                        last edited by

                        Yes…. but wait is not the problem.

                        I can easily move it away and on to local storage on the host if you want me to and test again.

                        Youtube Video

                        Notice the sudden drop in traffic and the firewall comes alive. WHY??

                        1 Reply Last reply Reply Quote 0
                        • F
                          firewalluser
                          last edited by

                          Rather than more it to local hw, how about a different online provider, it might be your existing online provider who has the issue, never assume everyone knows what they are talking about.  ;D

                          Always question everything.  ;)

                          Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                          Asch Conformity, mainly the blind leading the blind.

                          1 Reply Last reply Reply Quote 0
                          • S
                            Supermule Banned
                            last edited by

                            @firewalluser:

                            Have you tried earlier versions of ESXi and/or pfsense?

                            Yes. All the way back to 1.2.3 and all failed. ESXi from 4.1 to 6.0. ESXi 6.0 actually becomes unstable when testing. M0n0wall has been tested as well as OPNsense. OPNsense did better. Mikrotik wasnt even touched by the attack.

                            Are you still on pfsense 2.1 or did you try 2.2?

                            Production is on 2.1.5 and this testing is also done on 2.2.2 REL.

                            It might be a build conflict somewhere, maybe even at the bios level. I've seen bios make some hw drag.

                            Can you try same setup on different hw maybe with a different provider?

                            Its not a provider issue since I see the same patterne on 10gbit, 1gbit and 100mbit connections.

                            Might be worth spinning something up on the amazon cloud to test although thats got its own custom build of pfsense, so maybe have a look at those settings to see what differences there are. You might get some clues from that as to what settings are important.

                            Your choice but then we dont have a clue of whats in front and total control over the test environment.

                            1 Reply Last reply Reply Quote 0
                            • A
                              almabes
                              last edited by

                              My base FreeBSD 10.1 box will be back up and running again by Saturday.  I had to repurpose it's already repurposed hardware, and I'll have more time to assist with data collection.

                              This whole thing of having to work for a living can really get in the way of the fun with packets.

                              1 Reply Last reply Reply Quote 0
                              • S
                                Supermule Banned
                                last edited by

                                Wait as stated is not the issue.

                                Moved the VM to a 6 disk/RAID10 local setup on a IBM X3650 from a 8 disk RAID10 NAS storage device.

                                Youtube Video

                                Very little difference in %WAIT

                                But the drop in traffic and the FW coming alive is the same. SOMETHING is making it handle differently….

                                1 Reply Last reply Reply Quote 0
                                • T
                                  tim.mcmanus
                                  last edited by

                                  @Supermule:

                                  Wait as stated is not the issue.

                                  Moved the VM to a 6 disk/RAID10 local setup on a IBM X3650 from a 8 disk RAID10 NAS storage device.

                                  http://youtu.be/tD5A-kElWw8

                                  Very little difference in %WAIT

                                  Wait state isn't the issue, correct.  But it will mask the underlying kernel problem.  You'll never see the IRQ interrupt storm on a VM because it's the hypervisor kernel that's managing the hardware, not the VM kernel.

                                  1 Reply Last reply Reply Quote 0
                                  • F
                                    firewalluser
                                    last edited by

                                    What happens if you slow the nic speed right down to its slowest setting so the nic acts as a throttle?
                                    I dont know if it will be worth even trying half duplex at this stage.

                                    Try the slowest speed with System:Advanced:Networking, Network Interfaces tick boxes 2,3 & 4 unticked so the nic handles more of the packet processing.
                                    2 = disable hw checksum offload
                                    3= disable hw tcp segmentation offload
                                    4= disable hw large receive offload.

                                    I dont even know if these check boxes will have any effect [edit]running as a VM [/edit]in ESXi either, so it might be worth setting the nic speed in ESXi as another test.

                                    This is just a WAG though.

                                    Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                                    Asch Conformity, mainly the blind leading the blind.

                                    1 Reply Last reply Reply Quote 0
                                    • dennypageD
                                      dennypage
                                      last edited by

                                      It's not about the problem being better on bare metal. It's about reducing the number of variables in the test. A basic principle in problem isolation is to eliminate as many variables as possible. You identify the simplest configuration that demonstrates the problem, and then work with that.

                                      VM infrastructure is a massive variable when you are trying to diagnose an under-load kernel issue.

                                      @lowprofile:

                                      @dennypage:

                                      I couldn't agree more.

                                      @tim.mcmanus:

                                      Please, please, please stop wasting your time testing this issue on a hypervisor.  Put pfSense on bare metal and test it there.

                                      IT ISN'T BETTER ON BAREMETAL. Problem still exist. I tried several times on my bare metal supermicro.
                                      Read the thread and the other threads again. You will see the history.

                                      • i am though not using pfsense anymore. So i can no more test
                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        Supermule Banned
                                        last edited by

                                        Nothing so far despite setting everything on ESXi, Switch and pfsense…

                                        Playing around with settings on disabling offloading didnt yield anything either.

                                        Not a blip difference.

                                        @firewalluser:

                                        What happens if you slow the nic speed right down to its slowest setting so the nic acts as a throttle?
                                        I dont know if it will be worth even trying half duplex at this stage.

                                        Try the slowest speed with System:Advanced:Networking, Network Interfaces tick boxes 2,3 & 4 unticked so the nic handles more of the packet processing.
                                        2 = disable hw checksum offload
                                        3= disable hw tcp segmentation offload
                                        4= disable hw large receive offload.

                                        I dont even know if these check boxes will have any effect [edit]running as a VM [/edit]in ESXi either, so it might be worth setting the nic speed in ESXi as another test.

                                        This is just a WAG though.

                                        1 Reply Last reply Reply Quote 0
                                        • N
                                          NOYB
                                          last edited by

                                          @dennypage:

                                          It's not about the problem being better on bare metal. It's about reducing the number of variables in the test. A basic principle in problem isolation is to eliminate as many variables as possible. You identify the simplest configuration that demonstrates the problem, and then work with that.

                                          VM infrastructure is a massive variable when you are trying to diagnose an under-load kernel issue.

                                          @lowprofile:

                                          @dennypage:

                                          I couldn't agree more.

                                          @tim.mcmanus:

                                          Please, please, please stop wasting your time testing this issue on a hypervisor.  Put pfSense on bare metal and test it there.

                                          IT ISN'T BETTER ON BAREMETAL. Problem still exist. I tried several times on my bare metal supermicro.
                                          Read the thread and the other threads again. You will see the history.

                                          • i am though not using pfsense anymore. So i can no more test

                                          Ditto, ditto, ditto…

                                          Please don't make me resurrect the KISS philosophy subject again.  It doesn't seem to do well for my karma.  ;)

                                          1 Reply Last reply Reply Quote 0
                                          • F
                                            firewalluser
                                            last edited by

                                            I think the inevitable use of Dtrace is upon us.  :)

                                            I dont know the status of FreeBSD 11, but if its at least in a Release Candidate status, is it worth trying to port pfsense onto a FreeBSD 11 build?

                                            Although even if we could, if the problems still showed up, we'd still need Dtrace….

                                            Is there really no debugging facility built into pfsense?

                                            Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                                            Asch Conformity, mainly the blind leading the blind.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.