What is the biggest attack in GBPS you stopped

[Harvy66]

@firewalluser:

@Harvy66:

@firewalluser:

@Harvy66:

System Activity or "ps" will tell you total CPU time consumed. Just remember, a quad core can consume 4 CPU seconds per second.

Not always, you need to understand how the L2 cache works, ie its shared between cores on Intel, but AMD tend to have a cache amount per core, ie AMD would be less prone to cache collisions unlike Intel cpu's.

Cache misses counts as CPU time. If it takes an extra 250 cycles because of a cache miss, well, that's counting against you. CPU time is the amount of time a process has been scheduled. What it does during that time is irrelevant from the schedulers's standpoint.

@Harvy66:

System Activity or "ps" will tell you total CPU time consumed. Just remember, a quad core can consume 4 CPU seconds per second.

Yes

Yes & No

If no cache collisions occur then yes your "4 CPU seconds per second" would be right but when a cache collision occurs then its a matter of debate whether the cpu is giving you any cpu time useful to the task being asked of it by said software because a cache collision by definition is a failure of the cpu/core depending on where the cache collision occurs ie L1,2,3 which means no cpu processing useful to the task being asked of it as it backs out and resolves the cache collision.

To then make it a little more complicated or simpler depending on perspective, if the cache collision occurs on cache shared across all the cores then no you dont get your 4 cpu seconds per second as the CPU backs out and resolves the cache collision which holds up one or more other cores.

If the cache collision occurs on cache available only to a single core like L1 and some L2 (L2 on some chips is shared and on others its a small % of the total L2 but unique to each core), then you could consider it in your 4 cpu seconds per second statement but then there is still the matter of whether the CPU is giving you any "useful" processing time whilst it resolves the collision. Technically the time spent/clock cycles filling the cache having a collision and then resolving the collision is time wasted but it could still show as 100% core or CPU activity depending on the cache affected. So Yes when you see CPU activity at 100%, that would be correct but its not the whole picture as its hiding the cock ups of the CPU cache and the bus waits that are occuring.

Now even if we dont have any cache collisions, on a multi core cpu, time is then further spent wasted as the individual cores spend time waiting to access ram or the disk depending on bus architecture.

I've got software here which I have written which can run mulithreaded and multi cored, but its also capable of running on a single thread on a single core or x threads on a single core or x threads on x cores.

Guess which one runs the fastest?

The single threaded single core version.

Why is this?

Its because there is no time wasted handshaking between threads at the OS level and cores at the HW to access the ram and disk. Disk activity shows this up the most as disk/permanent storage is an order of magnitude slower to access even SSD's when compared to ram.

In some respects even though Arm chips are RISC ie dont have as many common tasks normally carried out by OS software functions which have made it into the cpu architecture unlike say Intels AES-NI to pick a relevant example http://en.wikipedia.org/wiki/AES_instruction_set
of where some common software functions have made it into the cpu architecture, they generally but not always tend to speed up the software but all of this ultimately depends on how the software is written and to a lessor extent the language and compiler used as optimising compilers like cache can work for you and against you as well depending on the chip used to run the software.

This is why I suggested right back at the beginning to try a 1.x version of pfsense. Considering the new features and improvements to functionality made to OS's over time, not only can code be compared easily, it will be possible to workout by elimination and some observations where the problem lies. I suspect knowing how HW drivers used to be for printers especially HP printers in the Win3.1,W95,W98, NT3.5, NT4 days that the drivers have not been updated enough to keep pace with OS developments, hence why I agree with KOM and suspect its a NIC hook issue in the OS, but it will also be compounded by the multi core's seen in cpu's today which is why I also suggested for those running it virtualised like on ESXI, to restrict the core's available to 1.

Apologies if this making you suck eggs, but due to limited data ie not knowing you or your past I dont know how much you know or dont know, hence the explaination above.  :)

You are correct, but I was not incorrect either. All I was saying was that you can see CPU time spent. All CPU time is the amount of time spent in a given context. yes, AMD's new arch has a much greater chance of cache line collisions, especially given the size of their L2 caches and the limited n-way associativity, but that reduces the amount of work done per unit of time, not the amount of time spent. I do agree that AMD can take more time to get the same amount of work done, but "cpu time" is still wall-clock time spent in a context.

Nice to know other people share my affection for understanding computers  :-)

[Harvy66]

What is "KERN.IPC.NMBUF"? I can't find anything about it?

[Supermule]

Kernel buffers.

https://www.google.dk/search?q=KERN.IPC.NMBUF&ie=UTF-8

[Supermule]

It goes down so fast you dont see the utilization…

@Harvy66:

@Supermule:

8 cores

http://youtu.be/-xTtzLEQx08

Not as good as hoped but not running 100% CPU like all the others. It seems that the response on the WAN graph are related to the PING on WAN.

It seems that the 2 CORE setup is the one that performs best in beginning until around 35 seconds into the attack. Then crash. 4 and 8 cores keep the GUI online.

You may be at 100% cpu, but according to the dashboard, you're running at 311mhz even when at 100%.

[Supermule]

4mbps attack and 40% packetloss.

Netstat -L doesnt see any exhaustion of queues.

Anybody know how to change the backlog to 1024??

Just to see if it matters.

[Supermule]

Here is the output of vmstat -z

Anybody find something unusual in this?


[dennypage]

Guys, you need to be much more rigorous in collecting data. You are trying to diagnose a network packet processing problem. Using the web interface to execute shell commands will not produce a consistent and reliable result. Not only is the web interface heavy weight, it is lower priority than kernel packet processing. And most importantly, your diagnostic data collection is dependent upon the behavior of the system you are trying to diagnose.

Let's assume you don't want to build a custom kernel…

You need to shed as many variables as possible and get as close to real data as you can. Turn Snort off for crying out loud. And anything else optional that might interfere with metrics. If you want to use command line tools, execute them outside of network processing. This means using the console, not ssh. Create a shell script that collects information on a periodic basis. Elevate the priority of the script to ensure timely execution. And save the output for every run.

Here is a sample script:

#!/bin/sh
ps -axuwww
While true
do
    /bin/date
    /usr/bin/netstat -m
    sleep 2
done

Here is a sample execution:

/usr/bin/nice -n -19 myscript

[Supermule]

Done it at the console at no useful output was generated for people to see.

I stopped Snort running and here is the output from the DoS.

First 2 is idle and next 2 is under DoS.

[Supermule]

Done some more testing this morning.

2-3mbps is all it takes. Has downscaled the Mbufs and state max a little.

http://youtu.be/NPtDnM8ixXs

Dennypage. Thanks for the info. Want to help diagnose then contact me on PM.

[tim.mcmanus]

This link is probably important to note the differences between versions:  https://doc.pfsense.org/index.php/Does_pfSense_support_SMP_(multi-processor_and/or_core)_systems

2.1 was single-threaded and 2.2 is multi-threaded.  That's why you're seeing an impact/performance difference between the two; it's not hard to extrapolate how and why.

I think what you're trying to determine, and this is based on my review of the thread, is which part of pf is choking.  In order to determine this you need to debug each component in the chain from the NIC to the CPU and back out as well as the code.  I'm not entirely sure you know programmatically where and which networking event triggers the issue inside pf, only that a large volume of data of a specific type starts the event.

You've moved beyond evaluating pf from a networking perspective and more into evaluating the codebase.  This requires a different kind of data collection and troubleshooting.  It also take an excruciatingly long time to identify and resolve these kinds of issues.  It's a lot more than just tweaking a setting in some cases.

Best of luck in determining the root cause and solution to this issue.

[Supermule]

Thanks Tim.

You are 100% correct.

[kejianshi]

Silly question - Is it possible to set a max cpu % that may be used by the packet filter?  Keep some in reserve for other processes?

[Harvy66]

Someone needs to use DTrace and make a flame-graph of what methods are being called in the kernel.

[Supermule]

Shall we test again Harvy??

[Harvy66]

I don't know how to do flame graphs, I've only seen them in Netflix presentations talking about optimizing FreeBSD.

[Supermule]

Tested Fortigate Virtual Server and after enabling Flood Protection, it ran perfectly during all tests.

1CPU and 1GB RAM in a VmWare VM.

Had an email conversation with a guy named Dave Huffman and he was able to replicate the scenario but only using DoS and not DDoS. Not that important, but it seems pfSense is not able to handle legitimate traffic vs. offending IP's.

It chokes somewhere in the stack.

[Nullity]

Do you need help figuring out how to enable profiling or some other debugging software?

[Supermule]

Yes because I need to get to the bottom of this.

[tim.mcmanus]

I would start here.  http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug.html

You can do remote kernel debugging as an option.  It's not for the faint of heart.  Debugging never is.

[firewalluser]

@Supermule:

Yes because I need to get to the bottom of this.

Try a USB nic on the wan, see how the data is handled differently.

You'll find these useful as well.
https://software.intel.com/sites/default/files/profiling_debugging_freebsd_kernel_321772.pdf
https://2008.asiabsdcon.org/papers/P8B-paper.pdf

Mutexs can catch some people out, but they are just locks to ensure the code doesnt deadlock in a multicore environment.

IMO profiling is better than retrospectively debugging crash dumps as you can make the crash dumps misleading in some situations masking the real root cause of the problem. I've found bugs in programming languages that have existed for over 15 years, thats how difficult some of these bugs are to find, even though it took me less than a week to find, theres a lot of exposed software out there.

Main difficultly are multi cores when it comes to debugging, you could be looking at the code running on one core whilst a bug in code running on another core creates the problem which crashes the code running on the core you are looking at. You can mask some problems by running on a single core but you will still have the problem, just less often as these things are just inglorious clockwork turkmachines.