What is the biggest attack in GBPS you stopped
-
So have you identified the code thats running on the core that maxes out when this happens?
I'm almost certain the issue is with the network driver in FreeBSD and it's also being contributed to by PF.
When my state table is low (394K) the attack cripples the entire box with the exception of the console. PF alerts that it hit its state table max in the console. I'm not sure why a full state table creates a more significant impact on the box, but it does.
What makes you say that?
Difficult to tell really without dtrace running wouldnt you say?
-
Yes. Could be great with better logger tools built in pfSense.
We are fighting a weird battle right now.
Sometimes it handles the attacks fine, then the same config crashes instantly on the same attack seconds later.
Only difference on my system is the the number 4 core hits 100%. When that happens then it goes down and packetloss occurs.
When it doesnt, then it can handle it. I have 8 cores and I cant see what uses that specific core.
-
I havent got a clue of where to begin and where to look.
I cant see whats using the core…
What do you make of this? Core nr. 4 says its idle at 100%
Was going to say Dtrace might be a good start but then I saw this. https://forum.pfsense.org/index.php?topic=94260.0
You can load up FreeBSD modules included with FreeBSD dist. I just tried to get it working but some trouble with the DTrace providers hindered me. After removing the /usr/lib/dtrace dir I got these not-so-clear results.
[2.2.2-RELEASE][admin@pfsense]/usr/lib: /usr/share/dtrace/toolkit/hotkernel Sampling... Hit Ctrl-C to end. dtrace: buffer size lowered to 2m dtrace: aggregation size lowered to 2m ^C FUNCTION COUNT PCNT 0xffffffff8035fbe2 1 0.0% 0xffffffff80dd0860 1 0.0% 0xffffffff80abad44 1 0.0% 0xffffffff8035fb5c 1 0.0% 0xffffffff8035fab5 1 0.0% 0xffffffff8035fbd7 1 0.0% 0xffffffff80f46fb6 1 0.0% 0xffffffff8035d4b2 1 0.0% 0xffffffff8035fbf9 1 0.0% 0xffffffff8097f4c0 1 0.0% 0xffffffff80d06b71 1 0.0% 0xffffffff8035fb8b 1 0.0% 0xffffffff80d06c28 1 0.0% 0xffffffff8035fac5 1 0.0% 0xffffffff8035fb67 1 0.0% 0xffffffff80f3275e 1 0.0% 0xffffffff8035faa0 1 0.0% 0xffffffff80f3712d 3 0.0% 0xffffffff80dd48bd 3 0.0% 0xffffffff80d069ea 8 0.0% 0xffffffff80f3726b 105 0.6% 0xffffffff80f2d8e6 17886 99.2%Seems promising.
Edit: May be related: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=185290
-
Last Supermule said, the problem only occurs when port forwarding is enabled in NAT. My guess is NAT unless we can get a confirmation that what I read was incorrect.
-
So have you identified the code thats running on the core that maxes out when this happens?
I'm almost certain the issue is with the network driver in FreeBSD and it's also being contributed to by PF.
When my state table is low (394K) the attack cripples the entire box with the exception of the console. PF alerts that it hit its state table max in the console. I'm not sure why a full state table creates a more significant impact on the box, but it does.
What makes you say that?
Difficult to tell really without dtrace running wouldnt you say?
Yes and no. There's enough empirical evidence to support this based on the architecture of PF.
Simply stated, PF does stateful packet inspection, therefore it must pass every packet to the kernel and then up to PF. A DDOS is a capacity-maxing attack. Doesn't matter where that capacity is–bandwidth, CPU, RAM, states, IRQ interrupts, etc.--but it is designed to impair service by maxing out capacity. This particular attack will take down any device that has to do stateful packet inspection. It took down a FreeBSD 10.1/Apache 2.4 box and a CentOS 6.4/Apache 2.4. It also affected the hypervisor each of those systems were running on. Nothing that does stateful packet inspection is immune to this kind of attack.
As a comparison, on the same hypervisor as the FreeBSD and CentOS box, I was running a Windows 8.1 VM and wireshark. The ingress and egress ports being attacked were mirrored to the Windows 8.1 box. So that interface was experiencing the same thing as the pfSense box. Since the Windows 8.1 box was not doing stateful packet inspections, and it was capturing every packet that came into the interface, that box was unaffected by the attack. Again, same hypervisor, different results.
Also, to clarify another point, this isn't a security issue per se (though I did discuss the tenants of security--CIA: confidentiality, integrity, and availability with my security practice today...) when you take into consideration that the pfSense device, as a firewall and stateful packet inspection engine, maintains the confidentiality and integrity of everything behind it. It is not suited, by design, to withstand a DDOS like this. You would need another stateless packet device to mitigate this kind of attack. If you attempt to make pfSense a stateless device, many of the other features would not be possible because they require stateful packet inspection to perform their tasks.
So in summary, there is no solution to this issue because of what pfSense does and how it does it. Not only is this true for pfSense, it's true for any stateful packet inspection device.
-
Not only is this true for pfSense, it's true for any stateful packet inspection device.
As I stated earlier in this thread, the attack took down my Cisco whateverthehellitis cough "Business Class" router installed by my ISP. It wasn't even the device being attacked! It just had to hand the packets over to an unfortunate instance of pfSense 2.2.2 running in a VM with a windows box running RDP behind it.
I'm wondering if someone has thought of having the firewall do something similar to greylisting if a state table or TCP connection attempt rate starts to climb.
That still wouldn't address the issue of a malicious party sucking up all your bandwidth, though, but might give the firewall an extra layer of defense from a SYN flood type attack. I just don't know how expensive that would be, processing wise.
-
Being susceptible to DDOS is not inherent to stateful firewalls, it's about not having a slow path that kills the machine. The fast path is existing states. If the slow path really has to be as bad as it is, like 1000 times slower, then have it give up when it spends too much time. Drop the packets for the non-existent states, don't allow existing states to be punished by blocked states.
nutshell: The slow path is a corner case that is pathological and can be trigger on demand, make it lower priority so it doesn't blow stuff up. Existing state should not be affected.
edit: a lot of what I do involves Big-O scaling, edge and corner cases, and making sure the worst case allows the system to function in a well defined limp-mode. Rule of thumb, modern computers have way more CPU and memory than internet bandwidth. If your network breaks before running out of bandwidth, something is incorrectly designed.
-
Super, maybe this weekend we can test me turning off port forwarding and testing again, this time I'll have my console up.
-
Being susceptible to DDOS is not inherent to stateful firewalls, it's about not having a slow path that kills the machine. The fast path is existing states. If the slow path really has to be as bad as it is, like 1000 times slower, then have it give up when it spends too much time. Drop the packets for the non-existent states, don't allow existing states to be punished by blocked states.
nutshell: The slow path is a corner case that is pathological and can be trigger on demand, make it lower priority so it doesn't blow stuff up. Existing state should not be affected.
edit: a lot of what I do involves Big-O scaling, edge and corner cases, and making sure the worst case allows the system to function in a well defined limp-mode. Rule of thumb, modern computers have way more CPU and memory than internet bandwidth. If your network breaks before running out of bandwidth, something is incorrectly designed.
Not entire true.
The first test, and the screen shots are on this thread, filled the state table. First capacity limit hit, interface goes down. Second test, screen shot again posted, created an IRQ interrupt storm. That is a hardware issue (probably a driver issue, but I'll explain more). IRQ interrupt capacity hit, interface goes down.
An IRQ interrupt storm can be generated by any piece of hardware. Google it for some interesting examples. When SSDs fail in some cases it generates IRQ interrupt storm, and it affects the machine in a similar fashion.
When I increased the state limit in pfSense, I hit a system limitation where the incoming data could not be consumed fast enough by the hardware and software resources. I could probably tweak this setting, but there will always be an upper limit. Set high enough and the DDOS would consume all of my bandwidth, essentially achieving the same thing: encumbering the interface.
Some good reading if you want to tweak the performance of FreeBSD: https://calomel.org/freebsd_network_tuning.html
https://forums.freebsd.org/threads/igb-interrupt-storm-detected.9271/
http://www.keil.com/forum/21608/
From this link: http://conferences.sigcomm.org/imc/2010/papers/p206.pdf
"A packet’s journey through the capturing system begins at the network interface card (NIC). Modern cards copy the packets into the operating systems kernel memory using Di- rect Memory Access (DMA), which reduces the work the driver and thus the CPU has to perform in order to transfer the data into memory. The driver is responsible for allocat- ing and assigning memory pages to the card that can be used for DMA transfer. After the card has copied the captured packets into memory, the driver has to be informed about the new packets through an hardware interrupt. Raising an interrupt for each incoming packet will result in packet loss, as the system gets busy handling the interrupts (also known as an interrupt storm). This well-known issue has lead to the development of techniques like interrupt mod- eration or device polling, which have been proposed several years ago [7, 10, 11]. However, even today hardware inter- rupts can be a problem because some drivers are not able to use the hardware features or do not use polling—actually, when we used the igb driver in FreeBSD 8.0, which was re- leased in late 2009, we experienced bad performance due to interrupt storms. Hence, bad capturing performance can be explained by bad drivers; therefore, users should check the number of generated interrupts if high packet loss rates are observed.
The driver’s hardware interrupt handler is called imme- diately upon the reception of an interrupt, which interrupts the normal operation of the system. An interrupt handler is supposed to fulfill its tasks as fast as possible. It therefore usually doesn’t pass on the captured packets to the operating systems capturing stack by himself, because this operation would take to long. Instead, the packet handling is deferred by the interrupt handler. In order to do this, a kernel thread is scheduled to perform the packet handling in a later point in time. The system scheduler chooses a kernel thread to perform the further processing of the captured packets ac- cording to the system scheduling rules. Packet processing is deferred until there is a free thread that can continue the packet handling.
As soon as the chosen kernel thread is running, it passes the received packets into the network stack of the operat- ing system. From there on, packets need to be passed to the monitoring application that wants to perform some kind of analysis. The standard Linux capturing path leads to a subsystem called PF PACKET; the corresponding system in FreeBSD is called BPF (Berkeley Packet Filter). Improve- ments for both subsystems have been proposed."
-
You are correct.
Last Supermule said, the problem only occurs when port forwarding is enabled in NAT. My guess is NAT unless we can get a confirmation that what I read was incorrect.
-
Not only is this true for pfSense, it's true for any stateful packet inspection device.
As I stated earlier in this thread, the attack took down my Cisco whateverthehellitis cough "Business Class" router installed by my ISP. It wasn't even the device being attacked! It just had to hand the packets over to an unfortunate instance of pfSense 2.2.2 running in a VM with a windows box running RDP behind it.
I'm wondering if someone has thought of having the firewall do something similar to greylisting if a state table or TCP connection attempt rate starts to climb.
That still wouldn't address the issue of a malicious party sucking up all your bandwidth, though, but might give the firewall an extra layer of defense from a SYN flood type attack. I just don't know how expensive that would be, processing wise.
PF (at least in OpenBSD versions, probably in FreeBSD port) has the concept of max rate qualifiers on rules. If you have a copy of Hansteen's "Book of PF", pg 69. They call them "state tracking options". I think it would give you functionality similar to greylisting. I'm guessing it's implemented in PF, not sure how. Rules that have these rate limits (or state tracking options) could dump the src address into a "block these address" table, then a simple block quick from early in the rule set would jump out quicker, perhaps keeping things alive.
-
I tested like mad yesterday.
This is pfsense running stateless:
This is pfsense running SYN proxy
This is pfsense running keep state
This is pfsense running 3 attacks in a row. As you see it handles them differently. Look at top. When core nr.4 hits 100%, packet loss is there. When it doesnt, then it survives and still routes the traffic to webserver behind.
To test some other things, I changed MBUFS to see if it fared batter. Sometimes it did, sometimes it didnt.
To chime in on whats happening here, is I want to know what runs on specific cores and I havent found out yet. I cant find a tool that does that.
The attached picture is of the 2nd and 3rd attack in 3attacks video. In the video you see packetloss and GW offline because Core4 hits 100%. It doesnt in the 2nd attack in the video.
I have top -P running and you dont see any interrupt storm and overall load is not very high. You dont even see the core that runs 100% and I have no idea why you dont. Thats why I specifically wants to target core nr. 4 to see any process running of that core.
That would be the bottleneck in pfsense and something we can deal with.
My hypervisor (ESXi 4.1U3) doesnt get unstable or unresponsive in any way. The targeted server is handling traffic fine and doesnt become unstable.
I would really much like some of the dev's to lend a hand for this and get a instance of Dtrace going on the box or any other box we can test.
I know that the box can handle it if the bottleneck is dealt with.
About the stateful vs. stateless discussion:
he biggest difference between simple IP filtering and stateful IP filtering is that simple IP filters have no recollection of packets that have already passed through the filter. Every packet is handled on an individual basis. Previously forwarded packets belonging to a connection have no bearing on the filter's decision to forward or drop the packet.
Stateful firewall (any firewall that performs stateful packet inspection or stateful inspection) is a firewall that keeps track of the state of network connections (such as TCP streams) travelling across it. The firewall is programmed to distinguish legitimate packets for different types of connections. Only packets matching a known connection state will be allowed by the firewall; others will be rejected.
stateless firewall is a firewall that treats each network frame (or packet) in isolation. Such a firewall has no way of knowing if any given packet is part of an existing connection, is trying to establish a new connection, or is just a rogue packet.
It doesnt matter to pfsense if its one or the other (look at the video). Packet loss still occurs and it goes offline.
Snort as tested works in stateless mode as well. If it was PF itself that was the culprit, I believe it should fare better running stateless than with stateful inspection. It didnt.
 -
Gonna try to get DTrace running?
-
I'll submit a little pfsense box running on a 5Mbit ADSL residential (not business) broadband line for testing later on today if theres enough bandwidth to get it to fall over?
Who should I PM the ip address to?
-
Yes but we need a developer to get it going me thinks.
Its above my head to do that.
So if anybody with deep knowledge of freebsd/pfsense want to chime in then it would be great!
Gonna try to get DTrace running?
-
Me.
IP and port number is needed.
My skype is : kontaktnetsupply if you need to chat. :)
Pls. make sure it answers to ping as well since I can monitor the result from here.
I'll submit a little pfsense box running on a 5Mbit ADSL residential (not business) broadband line for testing later on today if theres enough bandwidth to get it to fall over?
Who should I PM the ip address to?
-
But we have come a lot closer to finding out what could be done.
By tweaking system tunables the box is way tougher than out of the box. I will come back to that later.
The 4 types of states all tested sequentially and the video shows how the box responds.
Picture is how the load is on the VM.
In order it is:
Keep state
Sloppy State
Syn Proxy state
StatelessAS you can see the overall load is smallest on the SYN Proxy setting. As it should be.
In addition to this, I would like to add that the sweetspot for pfsense is 4cores on 1 socket and 4GB ram on this specific system.
I have 2 Quads in 2 sockets and it could easily be the answer to that. (Intel(R) Xeon(R) CPU E5420 @ 2.50GHz )
Why I dont know.
-
Yes but we need a developer to get it going me thinks.
Its above my head to do that.
So if anybody with deep knowledge of freebsd/pfsense want to chime in then it would be great!
Gonna try to get DTrace running?
I had it 90% functioning yesterday on 2.2.2. Maybe try the development build/snapshot?
You need like a half-dozen modules from FreeBSD, and "kldload dtraceall" loads without error.
-
Keep going at let us knowq when you have it going 100% and can trace the actual cpu's
Then we would be very close to getting this upstream.
-
Keep going at let us knowq when you have it going 100% and can trace the actual cpu's
Then we would be very close to getting this upstream.
A couple of posts ago, you had "3 attacks in a row, fine unless core 4 hit 100%", yes? In between, the box was not rebooted or was it rebooted? If not rebooted, that becomes interesting, perhaps a process migrating across cores and in the bad case maybe it winds up doing a lot of "interprocessor" locking.