What is the biggest attack in GBPS you stopped
-
Turned of Apinger and turned on Cron.
Much more responsive GUI and the traffic graphs didnt die on me completely this time.
Recovered instantly after the attack was stopped.
-
When enabling NTP, the box responsiveness became worse.
Not updating the graphs as quickly and a little worse in recovery time as seen in the little bend in the right hand corner of the last CPU graph from VmWare. It was a 15-20 second longer recovery.
-
Enabled Snort to see if it made things worse.
The answer to that is yes and no… initial phase was really good, since it took out the initial spike in CPU and made the load more even (slower boost). CPU had a shorter MAX load interval than on previous attacks.
It activated a more even CPU usage on the 8 cores and recovery was really good.
Traffic graphs didnt fare as well as with Cron only running.
-
When Snort AND Cron were enabled the graphs looked like this…
Total load was higher, but recovery was fine but initial CPU offloading was not there anymore.
-
Enabling Apinger did not do any good to the box as expected.
More CPU on top and a minute longer to recover. For what its worth, the impact didnt seem as big as the first test but still a trend (big).
Traffic graphs very unresponsive as the rest of the GUI. It dies on the first CPU spike in the GUI.
-
So maybe some services are more resource hungry and/or less refined in the scheme of things.
I wonder if there would be any benefit in having two firewalls in series, where the forward facing/1st was stripped of all unnecessary tasks/services, then the 2nd inline firewall had the unnecessary services/tasks running on it?
It certainly seems like there might not be any one property, service/task which is at fault, but maybe a combination of things which can affect how well the system stays up.
Have you seen the links I pm'ed almabes regarding setting up pfsense to reduce/avoid syn floods? If so did you give them ago and how did they perform?
-
I did not see the links.
States is about 1% on the box and it has limiters to how many states can be created pr. rule.
Running SYn Proxy state with allowance of 50 new connections pr. sec.
That allows the state table to have some "air" but it doesnt help much.
-
I have come a BIG step closer to locating the culprit.
Look at the graphs when NTPD is enabled.
It destroys the GUI completely and takes the interfaces offline in the GUI. No response from them. The graphs is a 3 minute attack and only maybe 10 seconds are showing.
Whats really interesting is the VmWare graph. When it spikes for the last time, the GUI comes back and the CPU graph in the GUI starts working again.
Wonder if NTPD and Apinger together could make something?
-
Deleted the Vmware Tools package and tested again.
Did a little better this time with NTPD and Apinger running.
Little spike before the last one is a reboot. Recovery took about a minute longer than usual.
-
After deleting the Vmware Tools then I disabled Apinger and NTPD.
The graphs on ESXi looked the same as "normal". No jitter from Apinger and NTPD afterwards.
Recovery was instant. Traffic graphs didnt respond well.
-
Ping from LAN -> WAN during the flood.
Next will be disabling traffic limitations in the SynProxy settings.
-
Running stateless with the box not having any limits to states pr sec. and other things pr. rule settings…
Box ran fine. Responsive. A little fallout on the traffic graphs but not so bad as seen before.
CPU load is a LOT less and only 5 dropped packets to Google via ping.
Instead of beeing crippled to a halt, it actually routed traffic to the server behind.
Whats a little odd, is that the traffic doubled in bandwith from around 4-5mbit/s to around 8-10mbit/s running stateless compared to SynProxy state.
-
APINGER running and the box is useless….
This is the difference running stateless and apinger vs no apinger.
Spike in CPu is 20% or more on ESXi and recovery takes about a minute longer...
-
Last one for today…
Enabling NTPD was also something that crippled the box.
Whats really interesting in the graphs in VmWare. The last 3 is the following:
1: Stateless NOT running Apinger and NTPD. No CPU hits 100% and the box is responsive and routes traffic.
2: Stateless Runing Apinger but NOT NTPD. 1 CPU (nr. 4) is 100% and the box stops routing and loses packets.
3: Stateless NOT running Apinger but running NTPD. 1 CPU is 100% (nr. 3) and the box stops responding and loses packets.1st graph doesnt have the small "bump" at the end of the attack and is responsive all along. When enabling Apinger OR NTPD OR both, then the box dies and recovery time is long (minutes). Recovery time is longer when running Apinger than with NTPD running.
When running SynProxy state the same pattern can be seen when attacked. Some CPU runs 100% and the box is dead.
Last image is a better view of the cpu usage.
1st one maxes out and packet loss occurs. 2nd does not and routes everything fine. 3rd is initially fine, but as soon as 1 cpu hits 100%, then the box is gone. (about halfway into the attack).
-
What I noticed was that pfSense would not let go of states for several minutes (5+). So when I was hit with 4.8M states, I'd still see 2.9M several minutes later. It wasn't until I rebooted the box several hours later that it returned to 3,500 states. IMHO boxes that are unresponsive after attacks still haven't released their states. Mine for whatever reason, recovered almost immediately after the attacks ceased.
Here is my hardware:
Intel Core i3-2100 Sandy Bridge dual core
Intel BOXDQ77MK LGA 1155 Intel Q77
4GB RAM
320 GB 7200RM HD
2 x Intel EXPI9301CTBLK 10/ 100/ 1000Mbps PCI-Express Network AdapterAlso note, that the initial SYN flood significantly burdened the UI but not the console, and when I increased states, the box was fine. The interface that was being attacked was disabled, but the box and the other three interfaces were working perfectly.
-
My setup:
Cisco Comcast business CPE -> Cisco SG300 switch -> WAN side of two firewalls, one HW one VM
VM firewall is running on a Dell precision 7500 with dual Xeon 5650 processors and 48 GB RAM
ESXi 6.0
Official ESF pfSense 2.2.2 OVA (2GB ram and 2 cores)
WAN goes to a broadcom add in NIC
LAN is coming out the onboard NIC and plugs into a catalyst 2948 switchThe Hardware firewall is a VK40-TE. It runs the nanoBSD version of pfSense 2.2.2
Wan side plugs into the SG300
Lan into the 2948I have 5 IPs, so the two firewalls have different WAN interface IP addresses. Additionally, on the VM firewall, I 1:1 natted a windows box and opened RDP.
I set up two laptops on the LAN, one wireless, because I wanted to sit on my couch. The other was plugged into the 2948. The wired laptop was configured to use the VM as it's gateway. I fired of ping -t www.google.com, ping -t <other firewall="" ip="">, and ping -t <comcast public="" cpe="" ip="">.
The results were unexpected.
During the attack, I could ping the WAN interface of the "opposite" firewall and get a 2ms response, as if nothing was happening. After a few seconds, www.google.com failed to reply, or came back with 2100+ ms replies, through BOTH firewalls. Even the un attacked one. The Comcastic gateway device went from sub 10 ms pings to 300-400ms.
Even after the attack stopped, the Comcastic gateway failed to route traffic. I had to power cycle it to get back on the grid.</comcast></other>
-
Did one more test, because I noticed a configuration error in my test setup.
After I reconfigured, I RDPd over to the test Win2k12 box's public IP, fired up wireshark and had supermule attack it.
I set up a ping to www.google.com through the un atacked firewall. It immediately started timing out.So again, my comcast gateway quickly crapped on itself, but pfSense didn't break a sweat. I was able to watch the packet capture over the RDP connection through the instance of pfSense under attack.
So, does the Cisco DPC3939B suck rocks, or was it "protecting" me by taking the brunt of the attack?
My states never got above 40k.
CPU hit about 30%, once.
RDP session never blinked.As soon as the attack stopped, www.google.com was pingable again.
-
Its your GW Cisco box that crapped itself and took the heat of pfsense.
-
Its your GW Cisco box that crapped itself and took the heat of pfsense.
I agree. My next step is to hit up a friend of mine that is the owner and chief packet plumber of an ISP. I'll haul the VM box over there and we'll test.
-
After deleting the Vmware Tools then I disabled Apinger and NTPD.
The graphs on ESXi looked the same as "normal". No jitter from Apinger and NTPD afterwards.
Recovery was instant. Traffic graphs didnt respond well.
Might be relevant for tuning apinger.
https://forum.pfsense.org/index.php?topic=37740.msg194857#msg194857WRT the gui going unresponsive during an attack, you guys wont be logged into the gui in real life anyway, it will probably occur when you are at home cooking on the bbq, so the fact the gui goes unresponsive is perhaps best looked at as a side show distraction not really relevant in the scheme of things.
How are those affected handling the log data? Are you just writing to pfsense's own log or are you syslogging it out to another machine?
WRT to NTPD, I noticed back over Xmas some of the NTP pools were being dominated by certain countries. I didnt like this so I moved away from the default NTP pools used by pfsense and chose some of my own to put in pfsense, although building our own NTP server might be an option http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html but I should also point out, it might be possible to hijack/affect the GPS signals based on a hack I read about a while back (which has quite major significance when you think about it) but cant find the link to it atm.