pfSense and Snort DDOS and syn flood
-
This is long but this is my story, question at the end....
So I started battling a DNS DDOS (at least thats what I am calling it) This is where 1000s of remote IPs hit my DNS server with recursive requests for domains like cisco.com, atlassian.com or ferc.gov etc...
I have recursion disabled my DNS server but it still responds with the root name servers so they send like 75kb I send like 600kb this bogs the server down... (I finally figured out the . forward zone which stops the root name server response)
In the beginning I was using DNS logs to build lists of IPs to block,,.... So I created a "BadActor" list and added it to the pfSense firewall to block traffic from any IP on the list port 53. This became monotonous So I wrote 5 Snort rules to block the IP of any IP making these requests.
After a few days these bogus DNS requests slowed significantly and then suddenly I started getting syn flood attack from the same group of IPs... So I wrote 4 rules to block the syn flooding.
I looked at the Snort2c table and 1000s, 10s of 1000s of ips were coming in at one point there were 86k ips blocked. Most of these entries were entire C-Blocks ie: 131.108.128.0 - 131.108.128.255
Ok so I wrote a script to look at the Snort2c IP list and converted the 86k ips into 357 blocked c classes like 131.108.128.0/24 and added those to the "BadActors" list and changed the rule to block on any port.
My thinking was to offload work from Snort and just ban those bad IPs in the firewall so after I updated the list I cleared the snort alerts and blocked and they instantly refiled with the same IPs that were blocked in the "BadActors" list.
OK Questions
Wouldn't blocking these IPs in the firewall stop Snort from looking at and alerting on them?
I regularly watch the alert list to see if general rules are blocking legitimate IPs but because there are so many of these alerts coming from my custom rules I can't see any other alerts.
Is there a way to have my custom Snort rule block the IP but NOT add an alert?
Thanks
-
Snort runs outside the
pffirewall engine. It sits on the path between the NIC and the firewall engine. Below are two flow diagrams I have shared countless times here on the forum.
Snort on the WAN is going to see and process those packets regardless of the inbound rules you put on the
pffirewall.Move Snort to your LAN interface instead. Then the firewall rules on the WAN can help filter the "already known" bad actors, and Snort on the LAN can catch the "new bad actors" as they attempt to reach your DNS server (or use the DMZ interface if your DNS server is in a DMZ). You can periodically review your alert logs and add the new bad actors to your WAN rules (hopefully you've created an alias for that).
If you are letting external hosts hit the DNS Resolver in pfSense, then that's just a bad idea all together. In fact, why run your own DNS server anyway? Let the big boys with huge pipes and lots of CPU horsepower host that function for you.
-
@bmeeks said in pfSense and Snort DDOS and syn flood:
Snort runs outside the
pffirewall engine. It sits on the path between the NIC and the firewall engine. Below are two flow diagrams I have shared countless times here on the forum.Snort on the WAN is going to see and process those packets regardless of the inbound rules you put on the
pffirewall.Move Snort to your LAN interface instead. Then the firewall rules on the WAN can help filter the "already known" bad actors, and Snort on the LAN can catch the "new bad actors" as they attempt to reach your DNS server (or use the DMZ interface if your DNS server is in a DMZ). You can periodically review your alert logs and add the new bad actors to your WAN rules (hopefully you've created an alias for that).
If you are letting external hosts hit the DNS Resolver in pfSense, then that's just a bad idea all together. In fact, why run your own DNS server anyway? Let the big boys with huge pipes and lots of CPU horsepower host that function for you.
I am a small hosting provider. so I have always had my own DNS server but moving it is something I am looking into.
Is this what snort would look like on the LAN side
I only tried for a few minutes but,. when I tried moving snort to the LAN side my custom rules stopped working...
This is one of the DNS rule ( I am not positive this stopped working because there may not have been attempts during the test):
block udp any any -> any 53 (msg: "MY Suspicious DNS Query ferc"; content: "ferc|03|gov", nocase,fast_pattern; sid:1000003; rev:1;)
This is one of the syn flood rule I am using:
block tcp any any -> $HOME_NET 443 (flags: S; msg:"Poss Syn TCP DoS 443 dst"; flow: stateless; detection_filter: track by_dst, count 15, seconds 5; sid:1000009; rev:1;)I tried switching to this while doing the test on the LAN side still no joy
block tcp any any -> $HOME_NET 443 (flags: S; msg:"Poss Syn TCP DoS 443 src"; flow: stateless; detection_filter: track by_src, count 15, seconds 5; sid:1000008; rev:1;)I have searched and I believe the answer is not but Is there a way to have my custom Snort rules block the IP but NOT add an alert?
Thanks for the help
-
@jmanatee said in pfSense and Snort DDOS and syn flood:
I have searched and I believe the answer is not but Is there a way to have my custom Snort rules block the IP but NOT add an alert?
No, that's not possible. The blocking mechanism used in pfSense depends on a custom binary plugin compiled into Snort. That plugin triggers off the alert log hook in the Snort binary. With no alert, there is no call to log and thus "no hook" for the blocking plugin to recognize.
@jmanatee said in pfSense and Snort DDOS and syn flood:
block udp any any -> any 53 (msg: "MY Suspicious DNS Query ferc"; content: "ferc|03|gov", nocase,fast_pattern; sid:1000003; rev:1;)
For this rule, I would put the specific IP address of my DNS server in the Destination IP field.
@jmanatee said in pfSense and Snort DDOS and syn flood:
This is one of the syn flood rule I am using:
block tcp any any -> $HOME_NET 443 (flags: S; msg:"Poss Syn TCP DoS 443 dst"; flow: stateless; detection_filter: track by_dst, count 15, seconds 5; sid:1000009; rev:1;)Did you verify that $HOME_NET is correctly populated with the move from WAN to LAN? Not knowing your IP addressesing scheme or whether NAT is in play, I can offer any advice beyond making sure that HOME_NET and EXTERNAL_NET are set properly. The default for EXTERNAL_NET is !$HOME_NET -- which is usually sufficient. I would use $EXTERNAL_NET as the "source" in your rules to be sure I did not inadvertently alert on internal traffic (unless you want to rule to trigger from internal hosts talking with each other).
-
@bmeeks said in pfSense and Snort DDOS and syn flood:
@jmanatee said in pfSense and Snort DDOS and syn flood:
I have searched and I believe the answer is not but Is there a way to have my custom Snort rules block the IP but NOT add an alert?
No, that's not possible. The blocking mechanism used in pfSense depends on a custom binary plugin compiled into Snort. That plugin triggers off the alert log hook in the Snort binary. With no alert, there is no call to log and thus "no hook" for the blocking plugin to recognize.
@jmanatee said in pfSense and Snort DDOS and syn flood:
block udp any any -> any 53 (msg: "MY Suspicious DNS Query ferc"; content: "ferc|03|gov", nocase,fast_pattern; sid:1000003; rev:1;)
For this rule, I would put the specific IP address of my DNS server in the Destination IP field.
@jmanatee said in pfSense and Snort DDOS and syn flood:
This is one of the syn flood rule I am using:
block tcp any any -> $HOME_NET 443 (flags: S; msg:"Poss Syn TCP DoS 443 dst"; flow: stateless; detection_filter: track by_dst, count 15, seconds 5; sid:1000009; rev:1;)Did you verify that $HOME_NET is correctly populated with the move from WAN to LAN? Not knowing your IP addressesing scheme or whether NAT is in play, I can offer any advice beyond making sure that HOME_NET and EXTERNAL_NET are set properly. The default for EXTERNAL_NET is !$HOME_NET -- which is usually sufficient. I would use $EXTERNAL_NET as the "source" in your rules to be sure I did not inadvertently alert on internal traffic (unless you want to rule to trigger from internal hosts talking with each other).
I will fix the DNS rule to have the DNS servers IPs
pfsense is setup as a 1:1 NAT with all my public IPs as virtual IPs -> to NAT IPs
In pfSense I created the alias "srvips" which contains my private IPs 192.168.2.0/24
In Snort under Pass Lists I have a "passlist_srvips" which has the pfsense alias "srvips"I also have all 5 check boxes checked
Local Networks
WAN Gateways
WAN DNS Servers
Virtual IP Addresses
VPN AddressesI looked at HOME NET it is set for "passlist_srvips" and it has all all my local IPs, DNS Servers and virtual IPs
But when I look at EXTERNAL NET the list is identical to Home HOME NET That is odd isn't it?
Not sure where I should go from here....
-
@jmanatee said in pfSense and Snort DDOS and syn flood:
I looked at HOME NET it is set for "passlist_srvips" and it has all all my local IPs, DNS Servers and virtual IPs
But when I look at EXTERNAL NET the list is identical to Home HOME NET That is odd isn't it?
What do you have the HOME_NET and EXTERNAL_NET drop-downs set for under the INTERFACE SETTINGS tab for your Snort LAN interface? In the majority of cases, leaving those set to "default" is sufficient. But if you are using 1:1 NAT with Virtual IPs that's going to present a different "IP address" to the Snort instance on the LAN. That's because when the traffic reaches Snort on the LAN interface (as the traffic is outbound to the clients on the LAN) the IP address is going to be different than what the Snort interface was seeing when run on the WAN interface. Remember that Snort sees traffic before the firewall rules are applied to inbound traffic. You may have to rethink how you populate HOME_NET (and EXTERNAL_NET) when running on the LAN side.
-
@bmeeks said in pfSense and Snort DDOS and syn flood:
@jmanatee said in pfSense and Snort DDOS and syn flood:
I looked at HOME NET it is set for "passlist_srvips" and it has all all my local IPs, DNS Servers and virtual IPs
But when I look at EXTERNAL NET the list is identical to Home HOME NET That is odd isn't it?
What do you have the HOME_NET and EXTERNAL_NET drop-downs set for under the INTERFACE SETTINGS tab for your Snort LAN interface? In the majority of cases, leaving those set to "default" is sufficient. But if you are using 1:1 NAT with Virtual IPs that's going to present a different "IP address" to the Snort instance on the LAN. That's because when the traffic reaches Snort on the LAN interface (as the traffic is outbound to the clients on the LAN) the IP address is going to be different than what the Snort interface was seeing when run on the WAN interface. Remember that Snort sees traffic before the firewall rules are applied to inbound traffic. You may have to rethink how you populate HOME_NET (and EXTERNAL_NET) when running on the LAN side.
Where is the INTERFACE SETTINGS tab in snort? I think this is what you mean
Currently I have
In Snort Interfaces -> WAN Settings -> General Settings -> Interface I have WANIn Snort Interfaces -> WAN Settings -> Choose the Networks Snort Should Inspect and Whitelist
Home Net is passlist_srvips
External Net is Default
Pass List is a list of IP from some of my clients -
@jmanatee said in pfSense and Snort DDOS and syn flood:
Currently I have
In Snort Interfaces -> WAN Settings -> General Settings -> Interface I have WANIn Snort Interfaces -> WAN Settings -> Choose the Networks Snort Should Inspect and Whitelist
Home Net is passlist_srvips
External Net is Default
Pass List is a list of IP from some of my clientsThe INTERFACE SETTINGS tab is the one that opens when you edit a Snort interface in the package GUI. The settings you show above should be correct.
Are you not getting alerts or not getting blocks? If you are not seeing alerts, have you cleared the snort2c table? If the IPs were placed in snort2c by the previous Snort instance when it was running on the WAN, and if you have not cleared them, they will still be enforcing and therefore blocking the traffic inbound from the WAN and it will never reach the LAN to be seen by the new Snort instance there.
Snort "blocks" by using a system call to populate the snort2c pf table. That table is referenced by a hidden built-in pfSense rule to block any source or destination IP that resides in that table. The table is only cleared by a reboot, by manual user action, or by a cron task if you have configured that on the GLOBAL SETTINGS tab via the "Remove Blocked Hosts" option.
-
@bmeeks said in pfSense and Snort DDOS and syn flood:
@jmanatee said in pfSense and Snort DDOS and syn flood:
Currently I have
In Snort Interfaces -> WAN Settings -> General Settings -> Interface I have WANIn Snort Interfaces -> WAN Settings -> Choose the Networks Snort Should Inspect and Whitelist
Home Net is passlist_srvips
External Net is Default
Pass List is a list of IP from some of my clientsThe INTERFACE SETTINGS tab is the one that opens when you edit a Snort interface in the package GUI. The settings you show above should be correct.
Are you not getting alerts or not getting blocks? If you are not seeing alerts, have you cleared the snort2c table? If the IPs were placed in snort2c by the previous Snort instance when it was running on the WAN, and if you have not cleared them, they will still be enforcing and therefore blocking the traffic inbound from the WAN and it will never reach the LAN to be seen by the new Snort instance there.
Snort "blocks" by using a system call to populate the snort2c pf table. That table is referenced by a hidden built-in pfSense rule to block any source or destination IP that resides in that table. The table is only cleared by a reboot, by manual user action, or by a cron task if you have configured that on the GLOBAL SETTINGS tab via the "Remove Blocked Hosts" option.
Currently I am on the WAN interface and Oh boy I am getting alerts and blocks My syn flood rule was so busy I couldn't see any other alerts. The snort2c table had 86k ips in it.
This is what I am trying to solve. I have now blocked the bulk of bad actors in the firewall but the alerts are still coming in so I guess the best solution is to move to the LAN interface but when I tried my rules stop working....
Should the External Net list be identical to the Home Net list?
Do you have suggestions on how to correctly populate the Home Net and External Net lists?
I am foggy on src vs dst with snort on LAN vs WAN
On the WAN interface inbound traffic the src IP is the public IP
On the LAN interface inbound traffic is src the public IP or the virtual ip or the NAT IP?
If on the LAN side and traffic is coming in to the server (Syn request) is that src? Is my servers ACK response src
So confused...
-
@jmanatee said in pfSense and Snort DDOS and syn flood:
Should the External Net list be identical to the Home Net list?
No, those two should not be identical. The IP addresses listed in EXTERNAL_NET should have a "!" character in front of each of them to signify the logical NOT operator. At first glance they may look the same, but examine closely for the "!" character. If it is missing, then something is not right for sure.
@jmanatee said in pfSense and Snort DDOS and syn flood:
On the LAN interface inbound traffic is src the public IP or the virtual ip or the NAT IP?
It should be the Virtual IP if I understand your setup correctly. Using an example INBOUND packet from the bad guy, here is how it will flow:
Bad Guy on web --> pfSense WAN port interface --> WAN pf rules --> LAN pf rules (if present) --> Snort --> LAN port interface --> LAN client
If I understand your setup correctly, then the Snort instance on the LAN is going to see the "bad guy's IP" as one of your 1:1 NAT virtual IPs. That's because the NAT processing is happening within
pfon the WAN interface as the traffic enters that interface.When you have Snort on the WAN, remember it is going to see the traffic before
pfdoes, so no NAT has taken place at that point for inbound traffic. But on the LAN instance, the WAN'spfrules have already acted on the inbound traffic from the bad guy and it has been processed through NAT and any "deny" rules that were present on the WAN.You will probably need to remove your Virtual IPs from the Pass List if you want to block the traffic when running Snort on the LAN. Run a packet sniff on the LAN to see what is actually coming across in terms of IP addresses. You can do that under DIAGNOSTICS in the pfSense menu.
-
@bmeeks said in pfSense and Snort DDOS and syn flood:
@jmanatee said in pfSense and Snort DDOS and syn flood:
Should the External Net list be identical to the Home Net list?
No, those two should not be identical. The IP addresses listed in EXTERNAL_NET should have a "!" character in front of each of them to signify the logical NOT operator. At first glance they may look the same, but examine closely for the "!" character. If it is missing, then something is not right for sure.
@jmanatee said in pfSense and Snort DDOS and syn flood:
On the LAN interface inbound traffic is src the public IP or the virtual ip or the NAT IP?
It should be the Virtual IP if I understand your setup correctly. Using an example INBOUND packet from the bad guy, here is how it will flow:
Bad Guy on web --> pfSense WAN port interface --> WAN pf rules --> LAN pf rules (if present) --> Snort --> LAN port interface --> LAN client
If I understand your setup correctly, then the Snort instance on the LAN is going to see the "bad guy's IP" as one of your 1:1 NAT virtual IPs. That's because the NAT processing is happening within
pfon the WAN interface as the traffic enters that interface.When you have Snort on the WAN, remember it is going to see the traffic before
pfdoes, so no NAT has taken place at that point for inbound traffic. But on the LAN instance, the WAN'spfrules have already acted on the inbound traffic from the bad guy and it has been processed through NAT and any "deny" rules that were present on the WAN.You will probably need to remove your Virtual IPs from the Pass List if you want to block the traffic when running Snort on the LAN.
When reading about Snort when I first set this up years ago I was under the impression you put Snort on WAN to protect from outside threats and you put Snort on LAN to protect from things like maleware or viruses already inside trying to reach outside. This is a very basic understanding I have
Would snort on LAN see the public IP of the bad guy? Could snort on LAN block it or would it only see the virtual IP or NAT IP If this is in fact the case I have to leave Snort on WAN
Interestingly the bulk of these IPs are coming from Brazil so I used pfBlocker to geoip block Brazil the attack is still occurring I see it in the pfblocker logs but not in Snort alerts. How is pfblocker before snort? I though pfBlocker only added firewall rules.
Thanks for the help
-
@jmanatee said in pfSense and Snort DDOS and syn flood:
When reading about Snort when I first set this up years ago I was under the impression you put Snort on WAN to protect from outside threats and you put Snort on LAN to protect from things like maleware or viruses already inside trying to reach outside. This is a very basic understanding I have
This is not really true. Think about how a packet from the Internet traverses your firewall. It comes into the WAN, hits the WAN firewall rules, then is routed out the LAN or whatever the destination interface needs to be (based on the destination IP). But if you run Snort on that internal destination interface, the packet still must pass through Snort before it can reach the final destination client. Refer back to the diagrams I posted originally in this thread. So, it does not matter where you run Snort in terms of it being able to see traffic flowing to internal hosts. You run Snort to protect your internal clients -- not your firewall.
The downside of running Snort on the WAN is two-fold. First, Snort will expend CPU and RAM resources analyzing traffic that the
pffirewall rules on the WAN may already be configured to block. You are seeing that issue now when you run Snort on the WAN -- the IPs are in thepfsnort2c table and are blocked from reaching your internal hosts, but Snort on the WAN is still expending CPU and RAM resources analyzing that "blocked" traffic. The second downside is that all of your internal hosts will be hidden behind the NAT IP. You will notice that all of the alerts show only your public IP as the "source" or "destination" for any internal hosts. That makes tracking down a malware infected internal host difficult.Running Snort on an internal interface lets the WAN firewall rules filter out the crap initially thus lessening the workload on Snort. It also means internal host IPs are not masked by NAT.
-
@jmanatee said in pfSense and Snort DDOS and syn flood:
Would snort on LAN see the public IP of the bad guy? Could snort on LAN block it or would it only see the virtual IP or NAT IP If this is in fact the case I have to leave Snort on WAN
Yes, in a normal setup (without your 1:1 NAT) Snort on the LAN or other internal interface would see the real public IP of the bad guy. But your 1:1 NAT setup with virtual IPs changes things. Snort on the internal interface is only going to see the virtual IP that resulted from the 1:1 NAT.
-
@jmanatee said in pfSense and Snort DDOS and syn flood:
Interestingly the bulk of these IPs are coming from Brazil so I used pfBlocker to geoip block Brazil the attack is still occurring I see it in the pfblocker logs but not in Snort alerts. How is pfblocker before snort? I though pfBlocker only added firewall rules.
If you are seeing this behavior with Snort running on the WAN, then it does not make sense. Snort definitely sees traffic before the
pffirewall when running on the WAN. Now, if you mean Snort running on the LAN, then yes it would make sense as the WAN rules would be filtering the traffic before it gets to Snort on the internal LAN interface. -
For now I will leave it on WAN and over time setup a virtual sever to test and play with configuring Snort on LAN
Thank you for all the help and taking the time to explain this to me.
Jon
