What is the biggest attack in GBPS you stopped

almabes

Status update:

I am at a point where I need help. I am not a BSD or UNIX guru. I need help getting pf configured with a NAT behind it. I also have DTrace working, but need guidance with what to run to get it to capture useful data during an attack.

We tested base FreeBSD 10.1-RELEASE. No pf, no NAT, just open to the wild internet. Supermule's attacks would overwhelm my available bandwidth and the box would be unreachable. The instant they stopped, the box was back.

Then I tried to configure pf and screwed myself. I don't have console access to the bare metal FreeBSD box, while sitting on my couch. I'll fix it when I'm in the office today.

After screwing myself, I had yard work to get done before the rains that saturated Texas attempted to do the same to Georgia and deliver a repaired laptop to the owner of a barbecue restaurant.

Harvy66

@almabes:

Supermule's attacks would overwhelm my available bandwidth and the box would be unreachable.

Orerwhelming bandwidth is the expected situation, we're concerned about overwhelming the CPU for a fairly high end CPU and a small amount of traffic.

almabes

From what I could see in top, the attack did not get anywhere near overwhelming a CPU on the bare metal box. Interrupts were fairly low, as well. Unfortunately, the circuit is only 20Mb, but the CPE is a Catalyst 3500 series L3 switch, unlike my home connection.

Then I screwed myself trying to activate pf.

The box is not exactly high end, but it's not a steaming pile of doggie doo with an OS either. It only has the one Intel em interface, not an not an Intel igb. I'm going to scrounge up another GigE NIC today to stick in it, among things I have to do to pay bills.

I'm trying to test methodically, one layer at a time.

Base FreeBSD 10.1-RELEASE. No pf, no NAT
Base FreeBSD 10.1-RELEASE. pf, no NAT
Base FreeBSD 10.1-RELEASE. pf, NAT no ports forwarded
Base FreeBSD 10.1-RELEASE. pf, NAT one port forwarded to box on LAN

and so on…

Supermule

The bandwith comsumed depends on the handling of the firewall…

It sounds odd, but its not a bandwith attack and the reply vs. no reply from the FW is the difference between 30mbit and 7-8mbit traffic.

Its hard to explain but when I see the traffic graphs its very different when the FW loses packets and when it survives.

firewalluser

@Harvy66:

@almabes:

Supermule's attacks would overwhelm my available bandwidth and the box would be unreachable.

Orerwhelming bandwidth is the expected situation, we're concerned about overwhelming the CPU for a fairly high end CPU and a small amount of traffic.

Exactly, its quite possible there might be too code expected to run when considering the workload Snort has to do when SM tested it with snort running.

Nothing can be done by a fw when the bandwith is tested to the max provided the fw can handle the bandwith in the first place which is going to be different due to different hw in use including differences in cpu's, bus speeds of the mb, nic's used etc etc. Thats a lot of variables to check.

Add in additional packages and services that its expected to run and before long you end up with something like a MS Small Business Server which is expected to handle quite alot of work on one box and then users wonder why networks grind to a halt when everyone logs in at the same time in the morning.

Supermule

Snort is not the issue since it didnt matter if it was there or not.

I wanted Snort on the testbox because its nice and I wanted to test it to see if it did somthing bad/good in regards to the overall load on the box.

When one core hits 100% its gone….

firewalluser

Well lots of possibilities but imo until we can get Dtrace working on a pfsense machine its going to be guess work for now.

Supermule

EDIT: Rant over…

firewalluser

Any reason why?

Perhaps they feel Dtrace is not the best tool for the job?
Maybe too busy in the short term?
Some other reason?

It still doesnt stop us from having a go though does it?

tim.mcmanus

@Supermule:

Yes and the dev's dont want to help at all.

Devs have helped.

almabes

@tim.mcmanus:

@Supermule:

Yes and the dev's dont want to help at all.

Devs have helped.

I agree with Tim, here. I started a conversation with the Devs last week. Not going into it completely on this thread though.
Long and the short of it was:

A) We have a mismatch of problem and solution (DoS attack and Stateful packet filtering device)
B) We need to recreate the problem on FreeBSD 10.1-RELEASE. (Something I am working on with Supermule)
C) DTrace will help locate the issue, and the quickest way to get it going is on a stock FreeBSD box.

The message from devs was also educational as to the architecture of BSD in general, and FreeBSD specifically.

When we isolate the issue, I will release documentation. Keep your pants on. If you can help with tracing or logging, or configuring pf then help me. It'll help the community get the desired results (pfSense more resistant to nefarious scripts) much quicker.

Supermule

https://forum.pfsense.org/index.php?topic=24337.0

Very interesting thread!

Do we have any that have something else than em driver nics on the boxes?

tim.mcmanus

@Supermule:

https://forum.pfsense.org/index.php?topic=24337.0

Very interesting thread!

Do we have any that have something else than em driver nics on the boxes?

So it's a driver and an OS issue, you don't say? ;D

Supermule

We dont know yet since we have tuned the damn thing hence a lot of numbers dont add up in the default settings in pfsense.

So we are working on it :D

tim.mcmanus

I do think the last link you posted was excellent with a lot of good discussion, and it might be worth testing non-em boxes. All of my interfaces are em's, so I won't be able to help.

firewalluser

I've got a usb2 10/100 plugable.com usb nic currently on my wan interface which I can swap with a gigabit usb3 plugable.com usb nic if need be as well as a sole onboard intel nic built into my little Intel D847 dual core celeron with 8Gb of ram and 30Gb msata ssd, so I can test all 3 formats to see how the different network drivers work on the same system by reassigning the wan interface. Avg CPU load is between 30-40% since installing the latest version of snort on it the other day so it should fall over quite quickly without any system tuning if this thread is anything to go by.

I can do it tomorrow later on in the morning as have appointments first thing as I've now found about about dangling states which was holding me up.

tim.mcmanus

@firewalluser:

I've got a usb2 10/100 plugable.com usb nic currently on my wan interface which I can swap with a gigabit usb3 plugable.com usb nic if need be as well as a sole onboard intel nic built into my little Intel D847 dual core celeron with 8Gb of ram and 30Gb msata ssd, so I can test all 3 formats to see how the different network drivers work on the same system by reassigning the wan interface. Avg CPU load is between 30-40% since installing the latest version of snort on it the other day so it should fall over quite quickly without any system tuning if this thread is anything to go by.

I can do it tomorrow later on in the morning as have appointments first thing as I've now found about about dangling states which was holding me up.

Make sure you know exactly which drivers are loading up to support the interface. Some commands to run in FreeBSD to help:


pciconf -lv

That will list the interfaces and classify them for you. Also go to /var/log/dmesg.boot to see your last boot log. You'll see a section in there that will look something like this:


em0: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xf060-0xf07f mem 0xf7c00000-0xf7c1ffff,0xf7c38000-0xf7c38fff irq 20 at device 25.0 on pci0
em0: Using an MSI interrupt
ehci0: <intel panther="" point="" usb="" 2.0="" controller="">mem 0xf7c37000-0xf7c373ff irq 16 at device 26.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
hdac0: <intel panther="" point="" hda="" controller="">mem 0xf7c30000-0xf7c33fff irq 22 at device 27.0 on pci0
pcib1: <acpi pci-pci="" bridge="">irq 16 at device 28.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
em1: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xe000-0xe01f mem 0xf7bc0000-0xf7bdffff,0xf7b00000-0xf7b7ffff,0xf7be0000-0xf7be3fff irq 16 at device 0.0 on pci1
em1: Using MSIX interrupts with 3 vectors
pcib2: <acpi pci-pci="" bridge="">irq 16 at device 28.4 on pci0
pci2: <acpi pci="" bus="">on pcib2
em2: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xd000-0xd01f mem 0xf7ac0000-0xf7adffff,0xf7a00000-0xf7a7ffff,0xf7ae0000-0xf7ae3fff irq 16 at device 0.0 on pci2
em2: Using MSIX interrupts with 3 vectors
pcib3: <acpi pci-pci="" bridge="">irq 18 at device 28.6 on pci0
pci3: <acpi pci="" bus="">on pcib3
em3: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xc000-0xc01f mem 0xf7900000-0xf791ffff,0xf7920000-0xf7923fff irq 18 at device 0.0 on pci3
em3: Using MSIX interrupts with 3 vectors</intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel></intel></intel(r)>

And of course the ever popular ifconfig.

FreeBSD loads NIC drivers (I think) from the kernel, so it might take some hunting to determine which ones are being used.

fsansfil

Who can hit me with the attack? There are few scenarios I would like to see how they respond. Can someone PM or email with available timezone so we could try a few of them ?

F.

almabes

@fsansfil:

Who can hit me with the attack? There are few scenarios I would like to see how they respond. Can someone PM or email with available timezone so we could try a few of them ?

F.

Supermule can execute the attack. He's UTC + 1.

Supermule

My skype name is: kontaktnetsupply

Its a lot easier that way :)