What is the biggest attack in GBPS you stopped
-
The "Pasta Method" of troubleshooting–throwing things against the wall to see what sticks--won't provide any value in identifying the root cause and resolving the issue. IMHO it's a waste of time.
Eliminate the most basic things first--FreeBSD, PF, and then pfSense. If you can recreate the issue in FreeBSD, poof, there's where the issue resides. Troubleshooting pfSense when the issue could be in FreeBSD is really a waste of time.
-
ok
-
When I run the attacks on myself all the time to harden the damn thing, I have never ever had an interrupt error on the console.
I hardly see interrupt load (0-2%) when hit and thats not the issue IMHO.
The issue is something that acts as a bottleneck on the way through NAT.
Its no issue when NAT is not there and the traffic hits a blocked FW. When it does NAT, then it crashes (1 core hits 100%) and packetloss is observed.
So the difference between no NAT and NAT is what takes it offline.
What model NIC are you using and are you running pfsense on ESXI thus going through ESXI's network stack?
Reason I ask is some NIC's can handle some of the network functionality that would otherwise be handled by the OS.
I wonder what NIC was in use when the photo was taken showing the interrupt storm message, it would be helpful to compare the difference in funtionality to see what work is offloaded to the OS and what is being handled by the NIC.Mer is wondering the same in the post below with the checksum sentence.
@mer:
So NAT takes the inbound packet, rewrites portions of the header, probably redoes checksums, then does it push the mbuf back onto the stack where it gets fed into PF processing again or does it just continue running the PF rules? If it's redoing checksum, is that being offloaded to hardware or is sw doing that?
Dtrace will help isolate the code in freebsd which is affected and it might show us the variables we can tune to reduce the incidence of an intterupt storm but imo we need to be focusing on the interrupt storm as the cause, everything else we have seen is just a symptom of the underlying problem.
Yes, this is definitely the next step. The guidance I deceived from the devs is to get a FreeBSD 10.1 image with dtrace to identify the issue in FreeBSD and subsequently FreeBSD/FP. However, there are certain features of dtrace that are not enabled by default, so it may/will require recompiling the kernel so you can capture those things. I was informed that "there are no dtrace probes currently in sys/netpfil, (look for SDT_PROVIDER_DEFINE) so you’ll be starting from scratch."
Again, it needs to be determined that the issue is not in FreeBSD 10.1 before you troubleshoot pfSense. What's the point in putting any effort into trying to remediate this in pfSense when that may have no bearing on the issue?
Theres no reason why we couldnt setup freebsd with some basic functionality and build up from there. Setting up pf tables for example is time consuming but not difficult, but its time consuming which is why we use pfsense as alot of the functionality is setup for us.
Maybe it would be quicker to compare the XML backups of all those affected to isolate the differences between installations installed?
MS does a nice free XML notepad app which makes it easier in a dual pane tree view on one pane, xml properties on the right pane to make it fairly quick and easy to modify xml files. I know text compare apps exist which make it useful for comparing differences in program code/html/text between versions, so maybe that would be a quicker and easier approach to take?
The XML backup compare approach would be quickest imo and we could have someone/a few people comparing the XML backups, whilst a few others maybe work on setting up freebsd from the ground up, or try a different approach in parallel?
Other alternatives/options is Dtrace but setup time is unknown as is the setup time of freebsd but wil likely be more than trying to get Dtrace running on pfsense, but we might also be able to setup more quickly than Dtrace or FreeBSD the flame graphs also mentioned earlier in the thread.
I personally would like to see Dtrace in pfsense as I think that would be a better low level form of functionality to have in pfsense going forward if you ware western, or going backwards if from the Southern American continent (different regions of the planet view the future differently ie some see it as a path laid out in front whilst others see it as a path behind their head as its unknown what the future holds but I digress). ;)
So whose in favour of what?
Thoughts otherwise we will end up going around in circles and nothing gets achieved with none of us any the wiser as to whats happening and no solution being found (if one can be found in this current version of pfsense as we dont know if it might be resolved freebsd 11 just to chuck that variable in as well).
Lots of variables but we need to sort out a plan otherwise I can only but insert the old cliche "if we dont plan then we plan to fail". ;D
-
The "Pasta Method" of troubleshooting–throwing things against the wall to see what sticks--won't provide any value in identifying the root cause and resolving the issue. IMHO it's a waste of time.
Eliminate the most basic things first--FreeBSD, PF, and then pfSense. If you can recreate the issue in FreeBSD, poof, there's where the issue resides. Troubleshooting pfSense when the issue could be in FreeBSD is really a waste of time.
I agree, first I heard that Snort was also being used in this. What version of Snort is in use as a new version has been released over the last few months.
May I propose we all submit XML backups and I can compare the differences in XML to find the common elements by all those affected?
I think this will be the quickest way to resolve or at least potentially eliminate the odds things to find the common elements which might be affecting things. *
- I say might but if its low level as in deep in the freebsd OS, different packages or parts of the system maybe calling the same parts of the OS at a low level so its not 100% foolproof comparing the XML backups but its a start which shouldnt take too much time.
Dont worry about the encryption in the XML it can be broken easily enough so best to blank your passwords and anything else you want to keep private, but dont say I didnt warn you. ;)
-
Emulating E1000 on Intel Dual Port server adapter on ESXi 4.1 U3.
Intel code is: E1G42ETBLK
-
I installed Snort quite late in the process and it didnt matter on the performance and the issue at hand.
Until i restarted it….
-
Emulating E1000 on Intel Dual Port server adapter on ESXi 4.1 U3.
Intel code is: E1G42ETBLK
Thats quite old like at least a couple years old and thats got a bug in where it can be hacked from the network stack iirc?
Re the testing the fw, I'm still get setup at this end based on what I noticed last night and posted so still double checking rules & the system is ok before the test. What time you going home tonight dont forget timezone your in?
-
CET +1 is the timezone.
I believe they solved that one in U2 or U3.
Going out eating tonight at 7.30PM local time.
-
Got your XML file, I'll compare that to mine and any other's which get pm'ed and I'll try to build a table to show the differences and the common elements to hopefully make it easier to solve.
I'm still curious to see if my home fw running pfsense 2.2.2. can be taken out so if you wanted to do a quick test, my ip is 2.101.3.83. I havent had chance to setup Skype yet as I've still got to get my mail server up and running and I dont allow ping so its not something that needs to be in the test. I've got VM recording the dashboard, pfinfo and system activity plus I'm also using a packet capture (full unlimited on the wan) so I can see whats going on. If the system falls down the ISP will automatically assign a new ip address so for the moment the 2.101.3.83 is mine to play with for now.
Drop me a PM to say when you have done, I'll PM back to let you know if I detect any problems here or not either way.
Edit.
I had PM'ed the above so something got screwy with the forum comms for it to appear here but also explains my post here https://forum.pfsense.org/index.php?topic=94573.0 which is weird as I can access the forum via a free vpn no problem but can no longer access it direct, unless Supermules XML backup triggered a snort alert which is now blocking that machine - will have to check in a moment.
Anyway have you run the script yet against my ip address? I'm still on that IP address and nothing appears to have happened if you have, so let us know Supermule if you have run the script or not.
Much obliged.
-
Anyway still nothing happened this end but Supermule did say he was going to eat tonight whatever that is, so for now I'm still on that ip address if SM pops their head back in later on tonight. I'll update the ip address when it changes next.
If anyone else wants to pm me their pf XML backup file affected who are affected by this scan I can do a comparison to see what are the common elements and what are the exclusive to you elements so hopefully we can start to narrow down what, where and when.
Just remember to blank the bits you want to keep private as encryption and stuff could be useful to the wrong people, etc etc. Once I've got them compiled I can do a table without names showing the common bits so we can then test an example with the common bits, see if it falls down and go from there to further narrow it down in the absence of anything else like Dtrace, flame graphs et al.
Edit.
ISP has forced an IP change so when Supermule touches base again I'll pass on the latest ip address change. The food must be good. :)
-
Anyway still nothing happened this end but Supermule did say he was going to eat tonight whatever that is, so for now I'm still on that ip address if SM pops their head back in later on tonight. I'll update the ip address when it changes next.
If anyone else wants to pm me their pf XML backup file affected who are affected by this scan I can do a comparison to see what are the common elements and what are the exclusive to you elements so hopefully we can start to narrow down what, where and when.
Just remember to blank the bits you want to keep private as encryption and stuff could be useful to the wrong people, etc etc. Once I've got them compiled I can do a table without names showing the common bits so we can then test an example with the common bits, see if it falls down and go from there to further narrow it down in the absence of anything else like Dtrace, flame graphs et al.
Edit.
ISP has forced an IP change so when Supermule touches base again I'll pass on the latest ip address change. The food must be good. :)
Maybe it's not the food but the beer or wine? ;D
-
Just got home from a nice dinner with friends and its 3.39AM here :D
Going to bed and will have a look during the day tomorrow. (saturday).
Got your XML file, I'll compare that to mine and any other's which get pm'ed and I'll try to build a table to show the differences and the common elements to hopefully make it easier to solve.
I'm still curious to see if my home fw running pfsense 2.2.2. can be taken out so if you wanted to do a quick test, my ip is 2.101.3.83. I havent had chance to setup Skype yet as I've still got to get my mail server up and running and I dont allow ping so its not something that needs to be in the test. I've got VM recording the dashboard, pfinfo and system activity plus I'm also using a packet capture (full unlimited on the wan) so I can see whats going on. If the system falls down the ISP will automatically assign a new ip address so for the moment the 2.101.3.83 is mine to play with for now.
Drop me a PM to say when you have done, I'll PM back to let you know if I detect any problems here or not either way.
Edit.
I had PM'ed the above so something got screwy with the forum comms for it to appear here but also explains my post here https://forum.pfsense.org/index.php?topic=94573.0 which is weird as I can access the forum via a free vpn no problem but can no longer access it direct, unless Supermules XML backup triggered a snort alert which is now blocking that machine - will have to check in a moment.
Anyway have you run the script yet against my ip address? I'm still on that IP address and nothing appears to have happened if you have, so let us know Supermule if you have run the script or not.
Much obliged.
-
Anyone care to explain this??
2 first attacks is causing packetloss.
Removing blocked hosts in Snort and it makes the damn thing survive 3 next attacks no problems.
No packetloss and no CPU hits 100%.
What does removing the snort blocked hosts exactly do to pfsense??
Wonder if some setting is over time, resetting something because hours later its back to losing packets until I remove the blocked hosts again.
-
I think snort is blocking and has to inspect every connection before letting it through. Like an extra firewall, but much less efficient. Anything to reduce the work snort has to do will speed things up. Sounds like snort is single threaded in some way. Probably a similar issue with why NAT crumbles when you enabled port forwarding?
-
How about dumping snort rules and then a pfctl -s rules on the pfsense box? It's been a while since I've looked/used snort, but I would think it's looking at packets, but shouldn't be blocking pfsense until a rule match. Then I could see Snort putting an IP address into a table and triggering PF rules based on the table. Removing the blocked hosts from Snort likely would flush a PF table and all associated states.
Again, this is speculation, not based on looking at a Snort/pfSense integration.
-
@mer:
It's been a while since I've looked/used snort, but I would think it's looking at packets, but shouldn't be blocking pfsense until a rule match.
If it doesn't block every packet, there is a chance a packet may get through before blocking. It could inspect packets asynchronously, then signal PF to kill a connection if it sees a "bad" packet after the fact, but most everything I see about snort is it can make your bandwidth lower. If it affects your bandwidth, then it must be acting as a middleman in some way that all packets must go through it. You also runt he issue if packets are coming in faster than snort can process them, then snort can't see every packet.
-
Snort is an IDS rather than an IPS (which limits its usefulness). It gets a copy of the packet, and is not in the data path. When snort decides to block something, it performs the block by adding a pf rule. Look for the snort2c table.
I think snort is blocking and has to inspect every connection before letting it through.
-
The only time I've ever used snort was as a passive sniffer on a mirrored port, so I was unsure how it worked when on the firewall. I assume rules inserted by snort must happen before the past path of checking if a connection already exists, and maybe this is why snort is some times associated with reduced performance.
-
Snort blocks by inserting the target IP address in pfSense pre-defined pf table called "snort2c". That table is created during the pfSense boot process and always exists whether the Snort package is installed or not. The table is rather high up in the chain, so it is one of the first tables (rules) a packet will see. It's not the very first, but it does come before any other user-supplied rules.
The blocking module of Snort is a binary patch integrated into the Snort source code as an output plugin. The plugin takes every Snort alert and checks the IP addresses against any Pass List (do not block list of IPs). If there is no match to a Pass List IP, the IP is then inserted into the snort2c table using FreeBSD system calls.
If the "clear states" option is enabled within the Snort GUI, then the aforementioned blocking plugin will also do a FreeBSD system call to clear all states in the packet filter associated with the IP that was blocked.
Bill
-
What process does it call when clearing states Bill??
