What is the biggest attack in GBPS you stopped
-
The "Pasta Method" of troubleshooting–throwing things against the wall to see what sticks--won't provide any value in identifying the root cause and resolving the issue. IMHO it's a waste of time.
Eliminate the most basic things first--FreeBSD, PF, and then pfSense. If you can recreate the issue in FreeBSD, poof, there's where the issue resides. Troubleshooting pfSense when the issue could be in FreeBSD is really a waste of time.
I agree, first I heard that Snort was also being used in this. What version of Snort is in use as a new version has been released over the last few months.
May I propose we all submit XML backups and I can compare the differences in XML to find the common elements by all those affected?
I think this will be the quickest way to resolve or at least potentially eliminate the odds things to find the common elements which might be affecting things. *
- I say might but if its low level as in deep in the freebsd OS, different packages or parts of the system maybe calling the same parts of the OS at a low level so its not 100% foolproof comparing the XML backups but its a start which shouldnt take too much time.
Dont worry about the encryption in the XML it can be broken easily enough so best to blank your passwords and anything else you want to keep private, but dont say I didnt warn you. ;)
-
Emulating E1000 on Intel Dual Port server adapter on ESXi 4.1 U3.
Intel code is: E1G42ETBLK
-
I installed Snort quite late in the process and it didnt matter on the performance and the issue at hand.
Until i restarted it….
-
Emulating E1000 on Intel Dual Port server adapter on ESXi 4.1 U3.
Intel code is: E1G42ETBLK
Thats quite old like at least a couple years old and thats got a bug in where it can be hacked from the network stack iirc?
Re the testing the fw, I'm still get setup at this end based on what I noticed last night and posted so still double checking rules & the system is ok before the test. What time you going home tonight dont forget timezone your in?
-
CET +1 is the timezone.
I believe they solved that one in U2 or U3.
Going out eating tonight at 7.30PM local time.
-
Got your XML file, I'll compare that to mine and any other's which get pm'ed and I'll try to build a table to show the differences and the common elements to hopefully make it easier to solve.
I'm still curious to see if my home fw running pfsense 2.2.2. can be taken out so if you wanted to do a quick test, my ip is 2.101.3.83. I havent had chance to setup Skype yet as I've still got to get my mail server up and running and I dont allow ping so its not something that needs to be in the test. I've got VM recording the dashboard, pfinfo and system activity plus I'm also using a packet capture (full unlimited on the wan) so I can see whats going on. If the system falls down the ISP will automatically assign a new ip address so for the moment the 2.101.3.83 is mine to play with for now.
Drop me a PM to say when you have done, I'll PM back to let you know if I detect any problems here or not either way.
Edit.
I had PM'ed the above so something got screwy with the forum comms for it to appear here but also explains my post here https://forum.pfsense.org/index.php?topic=94573.0 which is weird as I can access the forum via a free vpn no problem but can no longer access it direct, unless Supermules XML backup triggered a snort alert which is now blocking that machine - will have to check in a moment.
Anyway have you run the script yet against my ip address? I'm still on that IP address and nothing appears to have happened if you have, so let us know Supermule if you have run the script or not.
Much obliged.
-
Anyway still nothing happened this end but Supermule did say he was going to eat tonight whatever that is, so for now I'm still on that ip address if SM pops their head back in later on tonight. I'll update the ip address when it changes next.
If anyone else wants to pm me their pf XML backup file affected who are affected by this scan I can do a comparison to see what are the common elements and what are the exclusive to you elements so hopefully we can start to narrow down what, where and when.
Just remember to blank the bits you want to keep private as encryption and stuff could be useful to the wrong people, etc etc. Once I've got them compiled I can do a table without names showing the common bits so we can then test an example with the common bits, see if it falls down and go from there to further narrow it down in the absence of anything else like Dtrace, flame graphs et al.
Edit.
ISP has forced an IP change so when Supermule touches base again I'll pass on the latest ip address change. The food must be good. :)
-
Anyway still nothing happened this end but Supermule did say he was going to eat tonight whatever that is, so for now I'm still on that ip address if SM pops their head back in later on tonight. I'll update the ip address when it changes next.
If anyone else wants to pm me their pf XML backup file affected who are affected by this scan I can do a comparison to see what are the common elements and what are the exclusive to you elements so hopefully we can start to narrow down what, where and when.
Just remember to blank the bits you want to keep private as encryption and stuff could be useful to the wrong people, etc etc. Once I've got them compiled I can do a table without names showing the common bits so we can then test an example with the common bits, see if it falls down and go from there to further narrow it down in the absence of anything else like Dtrace, flame graphs et al.
Edit.
ISP has forced an IP change so when Supermule touches base again I'll pass on the latest ip address change. The food must be good. :)
Maybe it's not the food but the beer or wine? ;D
-
Just got home from a nice dinner with friends and its 3.39AM here :D
Going to bed and will have a look during the day tomorrow. (saturday).
Got your XML file, I'll compare that to mine and any other's which get pm'ed and I'll try to build a table to show the differences and the common elements to hopefully make it easier to solve.
I'm still curious to see if my home fw running pfsense 2.2.2. can be taken out so if you wanted to do a quick test, my ip is 2.101.3.83. I havent had chance to setup Skype yet as I've still got to get my mail server up and running and I dont allow ping so its not something that needs to be in the test. I've got VM recording the dashboard, pfinfo and system activity plus I'm also using a packet capture (full unlimited on the wan) so I can see whats going on. If the system falls down the ISP will automatically assign a new ip address so for the moment the 2.101.3.83 is mine to play with for now.
Drop me a PM to say when you have done, I'll PM back to let you know if I detect any problems here or not either way.
Edit.
I had PM'ed the above so something got screwy with the forum comms for it to appear here but also explains my post here https://forum.pfsense.org/index.php?topic=94573.0 which is weird as I can access the forum via a free vpn no problem but can no longer access it direct, unless Supermules XML backup triggered a snort alert which is now blocking that machine - will have to check in a moment.
Anyway have you run the script yet against my ip address? I'm still on that IP address and nothing appears to have happened if you have, so let us know Supermule if you have run the script or not.
Much obliged.
-
Anyone care to explain this??
2 first attacks is causing packetloss.
Removing blocked hosts in Snort and it makes the damn thing survive 3 next attacks no problems.
No packetloss and no CPU hits 100%.
What does removing the snort blocked hosts exactly do to pfsense??
Wonder if some setting is over time, resetting something because hours later its back to losing packets until I remove the blocked hosts again.
-
I think snort is blocking and has to inspect every connection before letting it through. Like an extra firewall, but much less efficient. Anything to reduce the work snort has to do will speed things up. Sounds like snort is single threaded in some way. Probably a similar issue with why NAT crumbles when you enabled port forwarding?
-
How about dumping snort rules and then a pfctl -s rules on the pfsense box? It's been a while since I've looked/used snort, but I would think it's looking at packets, but shouldn't be blocking pfsense until a rule match. Then I could see Snort putting an IP address into a table and triggering PF rules based on the table. Removing the blocked hosts from Snort likely would flush a PF table and all associated states.
Again, this is speculation, not based on looking at a Snort/pfSense integration.
-
@mer:
It's been a while since I've looked/used snort, but I would think it's looking at packets, but shouldn't be blocking pfsense until a rule match.
If it doesn't block every packet, there is a chance a packet may get through before blocking. It could inspect packets asynchronously, then signal PF to kill a connection if it sees a "bad" packet after the fact, but most everything I see about snort is it can make your bandwidth lower. If it affects your bandwidth, then it must be acting as a middleman in some way that all packets must go through it. You also runt he issue if packets are coming in faster than snort can process them, then snort can't see every packet.
-
Snort is an IDS rather than an IPS (which limits its usefulness). It gets a copy of the packet, and is not in the data path. When snort decides to block something, it performs the block by adding a pf rule. Look for the snort2c table.
I think snort is blocking and has to inspect every connection before letting it through.
-
The only time I've ever used snort was as a passive sniffer on a mirrored port, so I was unsure how it worked when on the firewall. I assume rules inserted by snort must happen before the past path of checking if a connection already exists, and maybe this is why snort is some times associated with reduced performance.
-
Snort blocks by inserting the target IP address in pfSense pre-defined pf table called "snort2c". That table is created during the pfSense boot process and always exists whether the Snort package is installed or not. The table is rather high up in the chain, so it is one of the first tables (rules) a packet will see. It's not the very first, but it does come before any other user-supplied rules.
The blocking module of Snort is a binary patch integrated into the Snort source code as an output plugin. The plugin takes every Snort alert and checks the IP addresses against any Pass List (do not block list of IPs). If there is no match to a Pass List IP, the IP is then inserted into the snort2c table using FreeBSD system calls.
If the "clear states" option is enabled within the Snort GUI, then the aforementioned blocking plugin will also do a FreeBSD system call to clear all states in the packet filter associated with the IP that was blocked.
Bill
-
What process does it call when clearing states Bill??
-
C'mon.
Troubleshoot FreeBSD first.
Why is that so difficult?
-
Its not Tim :D
I just want to know why it behaves like that. Currently bombing Anthonys provided IP. Hope he is stilla alive :D
-
It's an absolute waste of time to do anything with snort. It's built on top of pfSense, which in turn is built on top of FreeBSD.
When you troubleshoot a web server, you usually start with trying to ping the server, check to see if the port is open, and then look at the applications built on top of it. Why on God's green earth would you even look at snort?
If you want to resolve the issue, if you truly want to resolve the issue, eliminate the mast basic layers before you go up to the applications. If' stopped engaging in this ruse because of the distractions.
I can almost guarantee that the issue is with the FreeBSD networking drivers. I haven't seen a thread created on their forums to address or resolve the issue. There's no more point to this thread other than being a distraction from methodically trying to identify the root cause.
