What is the biggest attack in GBPS you stopped
-
But CPU load is not the issue??
Unless it consumes the cache and cannot process the IO cue.
But it should handle 3Gbyte/S no problem. So I dont understand why it should be CPU related at all.
-
I am more into the way PF handles the packets and what it does with it internally.
pf reads copies of packets and inspects them. Maybe its in that regard something (buffer) could be to small.
So when we reach x number of PPS, then it falls apart and cant keep up.
-
In the case of that one attack kernel-que was using nearly 100% of my CPU. If it was less thread friendly, it probably would have been closer to 25% cpu. The old PF could not consume 100% of a multi-core CPU, the new PF potentially could.
-
How do you inspect the kernel queue in pf??
I could then compare the 2.1.5 vs 2.2.2 and see if there are any difference?
In the case of that one attack kernel-que was using nearly 100% of my CPU. If it was less thread friendly, it probably would have been closer to 25% cpu. The old PF could not consume 100% of a multi-core CPU, the new PF potentially could.
-
As FreeBSD grew from 4.x to 5 and up to 10 I believe they've been pushing finer and finer grained locking into the kernel (like Solaris). This means that there can be more kernel preemption happening. I'd have to dig into code to verify but I'm guessing the "kernel igb0 que" thread is the one sitting between the bottom half of the interrupt handler and the next layer up (basically holding packets). If PF is looking at packets, modifying (or not) and putting them back, there may be a lot of queue manipulation going on (grab lock, deque/enque packet, release lock) so that may cause the que threads to start sucking CPU.
This is speculation, generalities; I have not really looked at this portion of the FBSD kernel so I could be totally off base.
-
Is there any way to inspect time used by process or cpu cycles used in pfSense???
-
System Activity or "ps" will tell you total CPU time consumed. Just remember, a quad core can consume 4 CPU seconds per second.
-
I'm sure you are familiar with the wide ranging contracts that exist in the world today beit Non Disclosure Agreements, or even the more common non compete agreements as exampled here: http://pando.com/2014/03/22/revealed-apple-and-googles-wage-fixing-cartel-involved-dozens-more-companies-over-one-million-employees/
http://www.businessinsider.com/emails-eric-schmidt-sergey-brin-hiring-apple-2014-3?IR=T
http://venturebeat.com/2014/05/23/4-tech-companies-are-paying-a-325m-fine-for-their-illegal-non-compete-pact/Put simply, you are not in a position to prove your innocence, because
-
To adhere to the terms of any NDA contract you may have been forced/coerced to sign would mean any disclosure would render you in breach of your contract and liable to whatever penalties may have been included in any agreement and who in their "right" mind would put themselves at a disadvantage?
-
Even if you have not signed any NDA contract you still cant prove your innocence, ergo the spooks & govt(s) still win, its classic divide and conquer techniques, which then begs the question why trust military & govt(s) or banks who carry out activities in secret?
What I can say is trust can take ages to build up, but can be destroyed in seconds.
On the point of passing enough traffic through pfsense, this has happened with less than 1mbits of traffic, a simple web page loading can trigger the OS cores to hang. Volume is irrelevant in the example I mention, but in relation to this thread and amounts of data, I wondered two things, exploiting the CPU designs namely cache and/or something network related as also mentioned by Kom here
https://forum.pfsense.org/index.php?topic=91856.msg517296#msg517296I'm inclined at this stage to err towards something nic related but I will examine the zip posted by supermule to see if I can see anything untoward, but this could be a variation on the heap spraying exploit http://en.wikipedia.org/wiki/Heap_spraying
I wonder if those affected are running snort and if so do the problems still exist, assuming snort is already aware of the problem much like AV software need to have found a virus before it can protect against it?
All of the above is said with the best of intentions and for it to be educational to those who might not be aware of the deceit and duplicity in the world today.
Edit.
Has anyone tried an earlier version of pfsense like a 1.x version by any chance? ;)
And likewise by your logic you can't prove you haven't signed a contract with some organization bent on sowing distrust in pfSense in general, or cmb specifically. Your post here looks exactly like what I would expect such an attack to look like. Don't bother denying it, you're obviously under and NDA or non-compete.
Conspiracy theories are great entertainment, but don't get carried away by it.
I can but it will only come out in court if it ever gets to court. The Rabbit Warren runs deeper than you may want to believe!
-
-
System Activity or "ps" will tell you total CPU time consumed. Just remember, a quad core can consume 4 CPU seconds per second.
Not always, you need to understand how the L2 cache works, ie its shared between cores on Intel, but AMD tend to have a cache amount per core, ie AMD would be less prone to cache collisions unlike Intel cpu's.
-
@mer:
As FreeBSD grew from 4.x to 5 and up to 10 I believe they've been pushing finer and finer grained locking into the kernel (like Solaris). This means that there can be more kernel preemption happening. I'd have to dig into code to verify but I'm guessing the "kernel igb0 que" thread is the one sitting between the bottom half of the interrupt handler and the next layer up (basically holding packets). If PF is looking at packets, modifying (or not) and putting them back, there may be a lot of queue manipulation going on (grab lock, deque/enque packet, release lock) so that may cause the que threads to start sucking CPU.
This is speculation, generalities; I have not really looked at this portion of the FBSD kernel so I could be totally off base.
I suspect its like KOM suggested with the driver nic integrating into the OS, thats why I have suggested to others where possible to go back to a 1.x pf version to test. The earlier OS's wont be so bogged down with "new" features", but your idea of locking is on the ball in terms of how OS's handle multi threading as against preemptive threading.
-
1st one is idle and 2nd is DoS.
Load is approx. 20mbit and maybe 50k PPS
-
Out of curiosity, has any posted about this over in FreeBSD mailing lists?
-
System Activity or "ps" will tell you total CPU time consumed. Just remember, a quad core can consume 4 CPU seconds per second.
Not always, you need to understand how the L2 cache works, ie its shared between cores on Intel, but AMD tend to have a cache amount per core, ie AMD would be less prone to cache collisions unlike Intel cpu's.
Cache misses counts as CPU time. If it takes an extra 250 cycles because of a cache miss, well, that's counting against you. CPU time is the amount of time a process has been scheduled. What it does during that time is irrelevant from the schedulers's standpoint.
-
250?
[Cache misses counts as CPU time. If it takes an extra 250 cycles because of a cache miss, well, that's counting against you.
[/quote] -
I'm not sure I understand the question. 250 cycles is a number I just pulled from thin air for going to main memory. I was talking about reading CPU time from "ps" or "top" and I said on a quad core, every second of real time is 4 seconds of CPU time. firewalluser then mentioned "not always" and said some stuff about L2 cache. I was responding by saying that cache does not affect CPU time.
-
Sorry, I didn't understand the thin air part. :)
I'm not sure I understand the question. 250 cycles is a number I just pulled from thin air for going to main memory.
-
I'm sure you are familiar with the wide ranging contracts that exist in the world today beit Non Disclosure Agreements, or even the more common non compete agreements as exampled here: http://pando.com/2014/03/22/revealed-apple-and-googles-wage-fixing-cartel-involved-dozens-more-companies-over-one-million-employees/
http://www.businessinsider.com/emails-eric-schmidt-sergey-brin-hiring-apple-2014-3?IR=T
http://venturebeat.com/2014/05/23/4-tech-companies-are-paying-a-325m-fine-for-their-illegal-non-compete-pact/Put simply, you are not in a position to prove your innocence, because
-
To adhere to the terms of any NDA contract you may have been forced/coerced to sign would mean any disclosure would render you in breach of your contract and liable to whatever penalties may have been included in any agreement and who in their "right" mind would put themselves at a disadvantage?
-
Even if you have not signed any NDA contract you still cant prove your innocence, ergo the spooks & govt(s) still win, its classic divide and conquer techniques, which then begs the question why trust military & govt(s) or banks who carry out activities in secret?
What I can say is trust can take ages to build up, but can be destroyed in seconds.
On the point of passing enough traffic through pfsense, this has happened with less than 1mbits of traffic, a simple web page loading can trigger the OS cores to hang. Volume is irrelevant in the example I mention, but in relation to this thread and amounts of data, I wondered two things, exploiting the CPU designs namely cache and/or something network related as also mentioned by Kom here
https://forum.pfsense.org/index.php?topic=91856.msg517296#msg517296I'm inclined at this stage to err towards something nic related but I will examine the zip posted by supermule to see if I can see anything untoward, but this could be a variation on the heap spraying exploit http://en.wikipedia.org/wiki/Heap_spraying
I wonder if those affected are running snort and if so do the problems still exist, assuming snort is already aware of the problem much like AV software need to have found a virus before it can protect against it?
All of the above is said with the best of intentions and for it to be educational to those who might not be aware of the deceit and duplicity in the world today.
Edit.
Has anyone tried an earlier version of pfsense like a 1.x version by any chance? ;)
And likewise by your logic you can't prove you haven't signed a contract with some organization bent on sowing distrust in pfSense in general, or cmb specifically. Your post here looks exactly like what I would expect such an attack to look like. Don't bother denying it, you're obviously under and NDA or non-compete.
Conspiracy theories are great entertainment, but don't get carried away by it.
I can but it will only come out in court if it ever gets to court. The Rabbit Warren runs deeper than you may want to believe!
Come on. Conspiracy theory is self proved and self destructive by its very nature. My sole point was that babbling about it here is in no way helpful to the original topic. Please stop.
-
-
Come on. Conspiracy theory is self proved and self destructive by its very nature. My sole point was that babbling about it here is in no way helpful to the original topic. Please stop.
+1. This topic and the way of "disclosing" the issue is already shitty enough – even without this conspiracy junk.
-
System Activity or "ps" will tell you total CPU time consumed. Just remember, a quad core can consume 4 CPU seconds per second.
Not always, you need to understand how the L2 cache works, ie its shared between cores on Intel, but AMD tend to have a cache amount per core, ie AMD would be less prone to cache collisions unlike Intel cpu's.
Cache misses counts as CPU time. If it takes an extra 250 cycles because of a cache miss, well, that's counting against you. CPU time is the amount of time a process has been scheduled. What it does during that time is irrelevant from the schedulers's standpoint.
System Activity or "ps" will tell you total CPU time consumed. Just remember, a quad core can consume 4 CPU seconds per second.
Yes
Yes & No
If no cache collisions occur then yes your "4 CPU seconds per second" would be right but when a cache collision occurs then its a matter of debate whether the cpu is giving you any cpu time useful to the task being asked of it by said software because a cache collision by definition is a failure of the cpu/core depending on where the cache collision occurs ie L1,2,3 which means no cpu processing useful to the task being asked of it as it backs out and resolves the cache collision.
To then make it a little more complicated or simpler depending on perspective, if the cache collision occurs on cache shared across all the cores then no you dont get your 4 cpu seconds per second as the CPU backs out and resolves the cache collision which holds up one or more other cores.
If the cache collision occurs on cache available only to a single core like L1 and some L2 (L2 on some chips is shared and on others its a small % of the total L2 but unique to each core), then you could consider it in your 4 cpu seconds per second statement but then there is still the matter of whether the CPU is giving you any "useful" processing time whilst it resolves the collision. Technically the time spent/clock cycles filling the cache having a collision and then resolving the collision is time wasted but it could still show as 100% core or CPU activity depending on the cache affected. So Yes when you see CPU activity at 100%, that would be correct but its not the whole picture as its hiding the cock ups of the CPU cache and the bus waits that are occuring.
Now even if we dont have any cache collisions, on a multi core cpu, time is then further spent wasted as the individual cores spend time waiting to access ram or the disk depending on bus architecture.
I've got software here which I have written which can run mulithreaded and multi cored, but its also capable of running on a single thread on a single core or x threads on a single core or x threads on x cores.
Guess which one runs the fastest?
The single threaded single core version.
Why is this?
Its because there is no time wasted handshaking between threads at the OS level and cores at the HW to access the ram and disk. Disk activity shows this up the most as disk/permanent storage is an order of magnitude slower to access even SSD's when compared to ram.
In some respects even though Arm chips are RISC ie dont have as many common tasks normally carried out by OS software functions which have made it into the cpu architecture unlike say Intels AES-NI to pick a relevant example http://en.wikipedia.org/wiki/AES_instruction_set
of where some common software functions have made it into the cpu architecture, they generally but not always tend to speed up the software but all of this ultimately depends on how the software is written and to a lessor extent the language and compiler used as optimising compilers like cache can work for you and against you as well depending on the chip used to run the software.This is why I suggested right back at the beginning to try a 1.x version of pfsense. Considering the new features and improvements to functionality made to OS's over time, not only can code be compared easily, it will be possible to workout by elimination and some observations where the problem lies. I suspect knowing how HW drivers used to be for printers especially HP printers in the Win3.1,W95,W98, NT3.5, NT4 days that the drivers have not been updated enough to keep pace with OS developments, hence why I agree with KOM and suspect its a NIC hook issue in the OS, but it will also be compounded by the multi core's seen in cpu's today which is why I also suggested for those running it virtualised like on ESXI, to restrict the core's available to 1.
Apologies if this making you suck eggs, but due to limited data ie not knowing you or your past I dont know how much you know or dont know, hence the explaination above. :)
-
Its not that I don't believe hardware is intercepted and modified for some people or that state level agencies don't hack and compromise target systems. Its just that unless there is some reason I doubt seriously they are doing it to you.
Are you someone worth targeting?
