What is the biggest attack in GBPS you stopped
-
I noticed the first test had about 30Mb in the first test and only 9Mb on the second, after you made the change for adaptive scaling. What this a DDOS syn attack or just a normal one?
-
DDoS.
-
Apparently not if you migrate to OPNsense…
Excellent. So, now this thread finally can be closed…
-
Apparently not if you migrate to OPNsense…
Excellent. So, now this thread finally can be closed…
It doesnt help existing users of pfsense who dont want to waste time jumping ship to yet another product though.
I personally would like to get to the bottom of it otherwise why even have a firewall at all then other than it helps bad actors who might be in play in a variety of ways including misleading people on threads in security forums?
All or nothing.
-
Status update:
I am at a point where I need help. I am not a BSD or UNIX guru. I need help getting pf configured with a NAT behind it. I also have DTrace working, but need guidance with what to run to get it to capture useful data during an attack.
We tested base FreeBSD 10.1-RELEASE. No pf, no NAT, just open to the wild internet. Supermule's attacks would overwhelm my available bandwidth and the box would be unreachable. The instant they stopped, the box was back.
Then I tried to configure pf and screwed myself. I don't have console access to the bare metal FreeBSD box, while sitting on my couch. I'll fix it when I'm in the office today.
After screwing myself, I had yard work to get done before the rains that saturated Texas attempted to do the same to Georgia and deliver a repaired laptop to the owner of a barbecue restaurant.
-
Supermule's attacks would overwhelm my available bandwidth and the box would be unreachable.
Orerwhelming bandwidth is the expected situation, we're concerned about overwhelming the CPU for a fairly high end CPU and a small amount of traffic.
-
From what I could see in top, the attack did not get anywhere near overwhelming a CPU on the bare metal box. Interrupts were fairly low, as well. Unfortunately, the circuit is only 20Mb, but the CPE is a Catalyst 3500 series L3 switch, unlike my home connection.
Then I screwed myself trying to activate pf.
The box is not exactly high end, but it's not a steaming pile of doggie doo with an OS either. It only has the one Intel em interface, not an not an Intel igb. I'm going to scrounge up another GigE NIC today to stick in it, among things I have to do to pay bills.
I'm trying to test methodically, one layer at a time.
- Base FreeBSD 10.1-RELEASE. No pf, no NAT
- Base FreeBSD 10.1-RELEASE. pf, no NAT
- Base FreeBSD 10.1-RELEASE. pf, NAT no ports forwarded
- Base FreeBSD 10.1-RELEASE. pf, NAT one port forwarded to box on LAN
and so on…
-
The bandwith comsumed depends on the handling of the firewall…
It sounds odd, but its not a bandwith attack and the reply vs. no reply from the FW is the difference between 30mbit and 7-8mbit traffic.
Its hard to explain but when I see the traffic graphs its very different when the FW loses packets and when it survives.
-
Supermule's attacks would overwhelm my available bandwidth and the box would be unreachable.
Orerwhelming bandwidth is the expected situation, we're concerned about overwhelming the CPU for a fairly high end CPU and a small amount of traffic.
Exactly, its quite possible there might be too code expected to run when considering the workload Snort has to do when SM tested it with snort running.
Nothing can be done by a fw when the bandwith is tested to the max provided the fw can handle the bandwith in the first place which is going to be different due to different hw in use including differences in cpu's, bus speeds of the mb, nic's used etc etc. Thats a lot of variables to check.
Add in additional packages and services that its expected to run and before long you end up with something like a MS Small Business Server which is expected to handle quite alot of work on one box and then users wonder why networks grind to a halt when everyone logs in at the same time in the morning.
-
Snort is not the issue since it didnt matter if it was there or not.
I wanted Snort on the testbox because its nice and I wanted to test it to see if it did somthing bad/good in regards to the overall load on the box.
When one core hits 100% its gone….
-
Well lots of possibilities but imo until we can get Dtrace working on a pfsense machine its going to be guess work for now.
-
EDIT: Rant over…
-
Any reason why?
Perhaps they feel Dtrace is not the best tool for the job?
Maybe too busy in the short term?
Some other reason?It still doesnt stop us from having a go though does it?
-
-
Yes and the dev's dont want to help at all.
Devs have helped.
I agree with Tim, here. I started a conversation with the Devs last week. Not going into it completely on this thread though.
Long and the short of it was:A) We have a mismatch of problem and solution (DoS attack and Stateful packet filtering device)
B) We need to recreate the problem on FreeBSD 10.1-RELEASE. (Something I am working on with Supermule)
C) DTrace will help locate the issue, and the quickest way to get it going is on a stock FreeBSD box.The message from devs was also educational as to the architecture of BSD in general, and FreeBSD specifically.
When we isolate the issue, I will release documentation. Keep your pants on. If you can help with tracing or logging, or configuring pf then help me. It'll help the community get the desired results (pfSense more resistant to nefarious scripts) much quicker.
-
https://forum.pfsense.org/index.php?topic=24337.0
Very interesting thread!
Do we have any that have something else than em driver nics on the boxes?
-
https://forum.pfsense.org/index.php?topic=24337.0
Very interesting thread!
Do we have any that have something else than em driver nics on the boxes?
So it's a driver and an OS issue, you don't say? ;D
-
We dont know yet since we have tuned the damn thing hence a lot of numbers dont add up in the default settings in pfsense.
So we are working on it :D
-
I do think the last link you posted was excellent with a lot of good discussion, and it might be worth testing non-em boxes. All of my interfaces are em's, so I won't be able to help.
-
I've got a usb2 10/100 plugable.com usb nic currently on my wan interface which I can swap with a gigabit usb3 plugable.com usb nic if need be as well as a sole onboard intel nic built into my little Intel D847 dual core celeron with 8Gb of ram and 30Gb msata ssd, so I can test all 3 formats to see how the different network drivers work on the same system by reassigning the wan interface. Avg CPU load is between 30-40% since installing the latest version of snort on it the other day so it should fall over quite quickly without any system tuning if this thread is anything to go by.
I can do it tomorrow later on in the morning as have appointments first thing as I've now found about about dangling states which was holding me up.
-
I've got a usb2 10/100 plugable.com usb nic currently on my wan interface which I can swap with a gigabit usb3 plugable.com usb nic if need be as well as a sole onboard intel nic built into my little Intel D847 dual core celeron with 8Gb of ram and 30Gb msata ssd, so I can test all 3 formats to see how the different network drivers work on the same system by reassigning the wan interface. Avg CPU load is between 30-40% since installing the latest version of snort on it the other day so it should fall over quite quickly without any system tuning if this thread is anything to go by.
I can do it tomorrow later on in the morning as have appointments first thing as I've now found about about dangling states which was holding me up.
Make sure you know exactly which drivers are loading up to support the interface. Some commands to run in FreeBSD to help:
pciconf -lv
That will list the interfaces and classify them for you. Also go to /var/log/dmesg.boot to see your last boot log. You'll see a section in there that will look something like this:
em0: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xf060-0xf07f mem 0xf7c00000-0xf7c1ffff,0xf7c38000-0xf7c38fff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt ehci0: <intel panther="" point="" usb="" 2.0="" controller="">mem 0xf7c37000-0xf7c373ff irq 16 at device 26.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 hdac0: <intel panther="" point="" hda="" controller="">mem 0xf7c30000-0xf7c33fff irq 22 at device 27.0 on pci0 pcib1: <acpi pci-pci="" bridge="">irq 16 at device 28.0 on pci0 pci1: <acpi pci="" bus="">on pcib1 em1: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xe000-0xe01f mem 0xf7bc0000-0xf7bdffff,0xf7b00000-0xf7b7ffff,0xf7be0000-0xf7be3fff irq 16 at device 0.0 on pci1 em1: Using MSIX interrupts with 3 vectors pcib2: <acpi pci-pci="" bridge="">irq 16 at device 28.4 on pci0 pci2: <acpi pci="" bus="">on pcib2 em2: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xd000-0xd01f mem 0xf7ac0000-0xf7adffff,0xf7a00000-0xf7a7ffff,0xf7ae0000-0xf7ae3fff irq 16 at device 0.0 on pci2 em2: Using MSIX interrupts with 3 vectors pcib3: <acpi pci-pci="" bridge="">irq 18 at device 28.6 on pci0 pci3: <acpi pci="" bus="">on pcib3 em3: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xc000-0xc01f mem 0xf7900000-0xf791ffff,0xf7920000-0xf7923fff irq 18 at device 0.0 on pci3 em3: Using MSIX interrupts with 3 vectors</intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel></intel></intel(r)>
And of course the ever popular ifconfig.
FreeBSD loads NIC drivers (I think) from the kernel, so it might take some hunting to determine which ones are being used.
-
Who can hit me with the attack? There are few scenarios I would like to see how they respond. Can someone PM or email with available timezone so we could try a few of them ?
F.
-
Who can hit me with the attack? There are few scenarios I would like to see how they respond. Can someone PM or email with available timezone so we could try a few of them ?
F.
Supermule can execute the attack. He's UTC + 1.
-
My skype name is: kontaktnetsupply
Its a lot easier that way :)
-
Just be aware everything you give out over Skype will be investigated by MS as per the Skyper agreement before you installed Skype, which means any web addresses, private ip addresses will see at least the Bing bot turning up at.
-
I know mate.
:)
Exactly the reason why I removed my webcam and microphone :D
The material shared over skype has nothing valuable in it.
-
Make sure you know exactly which drivers are loading up to support the interface. Some commands to run in FreeBSD to help:
[b]pciconf -lv[/b]
em0@pci0:0:25:0: class=0x020000 card=0x20448086 chip=0x15038086 rev=0x04 hdr=0x00
class = network
subclass = ethernetThe usb nics are not visible only the USB hubs they are plugged into.
That will list the interfaces and classify them for you. Also go to /var/log/dmesg.boot to see your last boot log. You'll see a section in there that will look something like this:
Motherboard nic
em0: <intel(r) 1000="" pro="" network="" connection="" 7.4.2="">port 0xf080-0xf09f mem 0xf7c00000-0xf7c1ffff,0xf7c29000-0xf7c29fff irq 20 at device 25.0 on pci0
em0: Using an MSI interruptUsb nics ue0 to ue3 same as below have not plugged in the gigabit ones yet.
ukphy0: <generic ieee="" 802.3u="" media="" interface="">PHY 16 on miibus0
ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ue0: <usb ethernet="">on axe0And of course the ever popular ifconfig.
FreeBSD loads NIC drivers (I think) from the kernel, so it might take some hunting to determine which ones are being used.
em0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether
inet6 em0 prefixlen 64 scopeid 0x1
inet 192.168.37.1 netmask 0xffffff00 broadcast 192.168.37.255
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
ue0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=80008 <vlan_mtu,linkstate>ether
inet6 ue0 prefixlen 64 scopeid 0x6
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
ue1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=80008 <vlan_mtu,linkstate>ether
inet6 ue1 prefixlen 64 scopeid 0x7
inet 192.168.60.1 netmask 0xffffff00 broadcast 192.168.60.255
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
ue2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=80008 <vlan_mtu,linkstate>ether
inet6 ue2 prefixlen 64 scopeid 0x8
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
ue3: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=80008 <vlan_mtu,linkstate>ether
inet6 ue3 prefixlen 64 scopeid 0x9
inet 192.168.50.50 netmask 0xffffff00 broadcast 192.168.50.255
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
status: activeIf you need any other info let me know.
Supermule if you want to hit me up, my ip is currently 92.29.124.124, VMware is recording the dashboard so traffic graphs for the interfaces, system activity and Status: System logs: Firewall (Dynamic View), with packet capture running on wan. Current CPUP load is 41 as if with nothing really happening other than those browser windows running along with snort so off to make a cup of tea. ;)</full-duplex></performnud,auto_linklocal></vlan_mtu,linkstate></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></vlan_mtu,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></vlan_mtu,linkstate></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></vlan_mtu,linkstate></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast></usb></generic></intel(r)>
-
Oh and whilst I remember, the wan interface is currently one of the USB nics, the EM0 (Intel nic) using the same drivers as you guys reporting problems is the lan interface so this first test will in effect be a driver test before I swap the wan on the EM0 nic.
-
Hitting you up on port 80 :)
Is it pingable and NAT'ing to something behind it?
-
You can stop it now for that ip address as I'm on another one.
I dont think it was pingable we could for the next test make it pingable but the problem with ADSL is its faster down than up so you may not get consistent pings back anyway.
When you say NAT'able what do you mean by that?
I tried a variety of different settings throughout, like trying to access ip addresses that were closer to me than the ip addresses coming in to port 80, swapped the firewall optimisation around from aggressive to normal to high latency (satellite links) whilst trying to get out but no joy at getting any webpages up throughout, the odd DNS request packet got out seen as a green icon in the dynamic fw log.
Interestingly it seemed to max out at 2.42Mbps yet various speed tests suggest I have a 5Mbps adsl connection. CPU was 100% through out.
FW stayed responsive throughout with snort running, changing various rules made some of the dashboard interface graphs stop updating second by second to maybe updating 10seconds later but I get that anyway when updating some rules or changing things in snort.
The only time I managed to kill the fw was trying to load the RRD graphs for All just a moment ago with the other fw webpages (dashboard, system activity, dynamic fw log) opened along with one or two other fw webpages open when changing rules etc, otherwise I'd have stayed on the old ip address for longer trying different things.
I got to pop out for an hour or so now, but when I get back we can try swapping the USB nic currently handling the wan interface over to the motherboards built in intel nic (em0) to test the driver theory next if you want?
It was funny because this song below was playing in my head throughout the test when I popped my head in to look at the fw as I was cooking some breakfast and eating it at the same time. ;)
www.youtube.com/watch?v=ZHwVBirqD2s
Edit. I could put one of the webservers I have written up if you want port 80 open, but I'd need to load up the relevant vm which I programmed it on and I have hundreds of vm's to go through as I'm in the processes of getting rid of old one's copying necessary code off them where applicable and generally consolidating them which is a lengthy process as I need to make sure I havent missed any source code, dll's, icons, images, help files, etc etc before deleting the vm.
-
No worries mate. Let me know when youre ready. :)
-
The only time I managed to kill the fw was trying to load the RRD graphs for All just a moment ago with the other fw webpages (dashboard, system activity, dynamic fw log) opened along with one or two other fw webpages open when changing rules etc, otherwise I'd have stayed on the old ip address for longer trying different things.
Something to note: When emX runs out of stated the entire pfSense device becomes sluggish. When you increase the states to over 5M, the emX interface goes down but the pfSense box is unaffected.
The console will show the state table getting filled in the first example and during the second test you may see the IRQ interrupt storm. I strongly recommend having the console open during testing because you'll get much more tangible information during testing. I sent almabes a ton of performance links and FreeBSD commands yesterday that should really help. I'm giving him time to digest them and eventually post them.
-
Ran 8M states and it performed well during the tests when it stayed up.
I dont see interrupt storms on the console. I have had a kernel taskq on the core that rises to 100% usage and then packet loss.
-
This is one of the links I sent out last night. http://bsdrp.net/documentation/technical_docs/performance
Use this more for the monitoring tools and the data they capture. It should help immensely. You may have to modify pfSense as some of the tools/modules may not be loaded by default in the distribution. You can grab them from FreeBSD as noted in the article.
-
I've been thinking about this, considering it was the RRD graphs which took out my fw straight away, I dont think the drivers are the issue really but more the SMP support in freebsd 10.1 like Harvy66 mentions in the FreeBSD 11 thread and in post 14 on this thread.
One way I can test this theory is to fire up an old single core AMD laptop with the same usb nics on the wan and see how it performs.
Laptop spec is not identical admittedly to the intel D847 ie only 2 or 4Gb of ram, spin disk instead of msata, but the nics will be the same as will pfsense.
Failing that, if I can get pfsense running on the raspberry pi's I have here, I can test on a RPi1 b (single core) as well as the new RPi2 which is quad core but same 1Ghz, same ram and will have same usb nics although the Arm cpu instruction sets are different.
Worth trying the single core laptop next instead?
-
Just let me know :)
-
Current IP address is 92.28.255.154
I've created an allow ICMP (ping) rule through to the wan so can you give it some pings first, then when I say we can test the EMx intel nic whilst I dust off this laptop at the same time.
I'm not going to attempt accessing the RRD graphs again for now, but everything else will be as before ie
dashboard running, system activity window, dynamic fw logs running along with me navigating around the fw changing rules etc to check responsiveness is still there.Edit. Can you let us know when you have done the pings as I have not seen anything yet.
-
It says no answer to ping here…
-
You can stop it now, I've changed ip address and I hadnt swapped the cables over.
Let me check the rules as I can see one ICMP came in which was allowed but I wasnt sure if that was you or anyone else reading the thread pinging me.
I also found a crash report from lunch time which I'll be submitting but it looks like the fw ran out of memory around the time I tried loading the RRD graphs.
Edit. Give me a few minutes as Unbound (the default DNS server setup in 2.2) doesnt like running with Firewall Optimisation (state timeouts) set to aggressive ie it timeouts the states quickly before the various name servers get a chance to respond when the fw reboots. I've swapped it over to High Latency for now and will leave it on that for the duration of the tests, but if unbound is running when the fw reboots and the state timeouts is too short, that will screw you over as well.
-
Feel free to post findings :)