Taming the beasts… aka suricata blueprint



  • This topic will become the definitive guide to setting up firewalling, IP lists, and an IDS/IPS running on pfsense. You are now offered the blue and the red pill. The choice is entirely up to you. If you think you already have enough experience setting up security systems, please feel free to take the blue pill and go on about your life. The rest of you are welcome to take the red pill.

    Remember, there are two rules: 1)You will do exactly as I say and 2)The responsibility for anything that happens (house burns down, your dog hangs itself from the ceiling fan) lies squarely on the recognized computer security industry leaders.

    Why firewalling? Because I’m tired of seeing allow any rules everywhere. Such rules should only be temporary solutions, but what I see is them being recommended as general best practices, without an accompanying “This is a temporary solution” tag alongside them. I admit I did the same mistake multiple times, and it’s now time to stop making this mistake. There are exceptions of course, (“There are always exceptions to a rule, but never a rule to the exceptions” - Me) like CARP’s SYNC interface. That interface is a direct connection between 2 bastion systems (bastion:it’s the “if that system is compromised, we all f***ed” system). A compromise to any one of them would only be possible through 1) bug 2) brute forcing and 3) direct access. In any one of those cases, having tighter rules on that interface will only stop the attacker for enough time to pop open his can of coke. It’s not like malware getting onto your system and fireing up a webserver. Entire categories can be shifted from suricata to pfsense, by using proper in/out filtering, thereby reducing the load on your systems. More on this later on.

    Why IP lists? Contrary to industry leaders with a combined experience of multiple thousands of years of experience, SECURITY THROUGH OBSCURITY IS REAL SECURITY . Multiple seminars/presentations/the world’s top security researchers can be proven wrong with just a single sentence: If I can’t see you, it’s highly unlikely I’ll shoot at you. There, we just demolished ideas and principles cast in stone in the security industry. You have already taken a significant step in your training. Keep going and you will not regret taking the red pill. An IP list keeping track of all of X botnet’s hosts, saves us the trouble of having to analyze each and every single packet going in and out of our network for that botnet. It’s the difference between a millisecond of delay, and buying a new system because your current one cannot keep up with the load.

    Why an IDS/IPS? Again, contrary to popular industry’s beliefs, the best place for placing an IDS/IPS is OUTSIDE your network. Sun Tsu gives us a hint in his book “The Art of War” why this is important to be placed outside your network. Think of it as your spies. Its job is identifying the bad guys, before they are allowed to cause any harm. The spies then tell the gatekeepers (pf) that a bad host shouldn’t be allowed in. Kinda like the theoretical job of the NSA, without the domestic spying. Our spies will help our systems to be kept protected, by weeding out the majority of the “bad” guys, without our systems even noticing.

    In this topic I assume that you know how to install pfsense and suricata, and set up pfsense and suricata interfaces.

    General FAQs:
    “Who are the beasts?” pfSense + suricata.

    “Why did you post this topic in this subforum?” Because ultimately, this topic is about suricata. The first couple of posts will be how to set everything up to make our life easier with suricata.

    “Why are you always adding posts, and very rarely editing your posts?” This comes down to a unique phenomenon known as the pfsense forum black hole. It’s a documented (and scientifically proven) fact that editing an old post will create a time-space distortion, greatly affecting the gravity in the forum, which in turn compacts entire galaxies into single dots (black holes). I’m really sorry, but the only way to get the most up to date information is going through the entire thread.

    “You’ll cost us our jobs for $deity’s sake! Stop posting this information!” I’m at a point where I realised that most security industry leaders would have a hard time poking their eyes out with a fork (a Cypriot saying, showing complete inability to perform a job). Frankly, the sooner they all starve to death, the better it will be for the industry.

    “I’m an NSA/CIA/other-alphabet-soup-agency agent. Where can I find information on your whereabouts to, err…, have an intelligent conversation with you. It has nothing to do with a drone flying overhead.”  Take the blue pill and GTFO.

    Here we go!

    Firewalling
    Always whitelist, NEVER blacklist. In other words, start with everything disallowed (pfsense’s default is exactly that) and only allow what is absolutely needed. The same applies for outgoing filtering. Head over to Firewall>aliases> and set up a ports alias. The ports you should allow outgoing are:

    | 21 | FTP, if you have the need to access hosts over FTP |
    | 22 | SSH, if you remotely administer systems |
    | 25 | SMTP if planning to send email (technically your ISP should only allow outgoing 25 to its relay hosts) |
    | 80 | HTTP, if planning to access any website, essential for updating systems |
    | 123 | NTP, maybe not needed depending on your preferences |
    | 443 | HTTP/S, see HTTP note |
    | 465 | SMTP/S, see SMTP note |
    | 547 | DHCP, only needed on interfaces that pfsense will automatically provide IPs for |
    | 993 | IMAP/S, if you want to access a remote IMAP account |
    | 995 | POP3/S, if you want to access a remote POP3 account |
    | 1024 to 65535 | unprivileged ports, you don’t have any control over these, make a note to remember what ports are privileged |

    Name the alias something that is easy to remember, like I don’t know, how about “outgoing_ports”?

    Create another alias, pfsense_ports

    | X | pfsense’s webgui port. See note |
    | Y | pfsense’s ssh port. See notes below |

    A note on the webgui and ssh port: The port chosen must be X , where 0 > X < 1024. Same goes for Y. The first allowed port is 1 and the last allowed port is 1023. Choose a port not in use by any other service (DHCP, NTP). These first 1024 ports are called privileged ports, and greatly help us in administering our systems. A relatively random port (eg not 0, not 81) will allow for some security through obsurity.

    Done on the alias page, go Firewall>Rules:

    Set up a rule on the LAN (admin) interface to allow access to the webgui port along with the ssh port. This is your way in to your pfsense. If you screw this step up, you will be locked out of the system. Double check this rule and double check it again:

    | Action | Pass |
    | Disabled | UNTICKED!!! |
    | Interface | Your admin interface, typically LAN. |
    | TCP Version | IPv4+IPv6. Saves you the trouble of updating it in the future |
    | Protocol | TCP |
    | Source | Admin interface subnet, ie “LAN subnet” |
    | Destination | Admin interface address, ie “LAN address” |
    | Destination port range | Start typing your port alias’s name. It should eventually show “pfsense_ports” |
    | Description | Enter a meaningful description to you |

    If you have already double-double checked the rule, check it once more. If you think this is an unnecessary step, the door is there and you are free to leave. Remember rule 1. Hit save then apply and wait for a bit.

    Next up Floating tab:
    Set up a rule but make these changes:

    | Action | Block |
    | Quick | TICKED!!! |
    | Interface | Hold CTRL and click on all interfaces EXCEPT LAN(admin) and SYNC |
    | Direction | any |
    | Source | any |
    | Destination | any |

    DON’T CHANGE DESTINATION PORT RANGE!!! Had to add this since I confused a few people already 😛

    Those are pretty much the only changes you need to make. Save and apply the rule. When adding other floating rules, make sure this rule stays at the absolute top of the list.

    Next go to System>Advanced>Admin Access and >>>TICK!!!<<<

    | WebGUI redirect |
    | WebGUI Login Autocomplete |
    | Anti-lockout |

    Make sure the ports (webgui+ssh) match those you put in the alias.
    Save/apply

    FROM NOW ON, PRETEND WAN DOES NOT EXIST <<< Wan should not have any rules if dealing with a home network. If running servers behind pfsense, see note near the bottom about WAN. I’m assuming you already know how to set up forwards/public IPs and know what you are doing. >>>

    Interface’s rules tab: Set up a DNS rule on each interface (except floating, that’s not an interface! and SYNC), following the example for the pfsense_ports. Destination should be interface’s IP >>>OR<<< the CARP IP.  If you wanted this to be simpler (in the future proof concept, not time spent setting up) then create a port alias using ports that will provide services to clients, like DNS/DHCP(if all internal interfaces are DHCP)/NTP, then set up the rule using that alias. See? You are already getting the hang of it. See DNS FAQ below. Save/apply.

    Head over to an interface’s tab and set up a an allow rule. Source should be the interface’s subnet. The destination should be any, and for the ports use the outgoing_ports alias created above. Destination should be any. Otherwise identical to the webgui rule. Warning! This allows any host to access any other host on other interfaces using those ports. If this is not needed (and generally it shouldn’t), finish the rule, and head over to the floating rules tab. Let’s say you have a logging interface (interface dedicated to syslog logging, NOT industry’s best practice, MY best practice since traffic for that host from any point on the network must pass through the firewall). You don’t want access to this interface from clients on the wifi, or the kids’s interface. Set up a floating BLOCK rule with all the interfaces you DON’T want to access it, and apply it. MAKE SURE QUICK IS TICKED!!!

    Back to where we left. Nobody likes his internet being down (I grew tired of having to explain that the Internet was designed to survive a nuclear holocaust without it being down, if you can’t beat them, join them). So hurry up with the other interfaces as well.

    Warning: Be careful with the SYNC interface if running CARP. See introduction. It’s safe if you create an allow any any there. That is the only interface where we set up allow any rules for simplicity, since technically it’s a direct connection between the two boxes. If an attacker has physical access to your hosts, then all bets are off, tighter rules will not save you. The same does not apply to other internal interfaces (eg a wireless network) since an attacker is able to attack from the building next door, which is not technically physical access.
    The important thing to remember when setting up other interfaces is to change the source as the interface’s subnet. Each interface should be on a different subnet. This takes care of “alien” traffic.

    For the incoming part of the firewalling “chapter” you just need to allow access to ports that you host servers/processes for. The same principles apply as with the outgoing part. Only allow what is absolutely needed.

    Finished (for now ;)). You set up an alias, set up rules for accessing pfsense, set up commonly used ports, limited traffic between interfaces… After reading the unprivileged ports FAQ below, you’ll understand that some traffic can be completely stopped at the firewall level. By the end of this thread, you will know how to use suricata to also identify and ban the hosts responsible for privileged > privileged traffic. What if such a host used this traffic, then tried to legitimately connect to your webserver? It would first be denied by the firewall, with no indication to the attacker (block rule), suricata would pick up on that traffic, ban the host, and by the time he tries to connect again, he cannot.

    Firewalling FAQs:
    “Why not set up rejecting rules? I like the yellow box better!” Rejecting rules should only and always be set up when dealing with IP lists on internal interfaces, not trying to hide your firewall from the network. It’s better if you do your best to delay the attacker and allow the firewall/IDS/IPS system to do its job, than just outright giving away that the port is open, but traffic is rejected. An attacker wants to connect to your network. That traffic should be delayed for as long as possible, to fool the attacker into thinking that nothing is there. Ringing the phone and you picking up only to say “I’m not talking to you!” and hanging up is tease bait. The attacker will call again and again. These are the rejecting rules. Ringing the phone a couple of times with no answer is likely to push the attacker into giving up. These are the blocking rules. The same does NOT apply to internal interfaces though. You are trying to access a site whose IP is listed on an IP list. Your browser can sit there for (what looks like) forever, trying its best to connect, then finally giving up a minute later. Would you want that, or your browser immediately throwing up a page cannot be displayed message?

    “Why are you only allowing TLS ports? What about regular IMAP?” If your email host refuses to allow secure connections to your email account, it’s time for you to move on to a different host, no questions asked. On a side note, if your host doesn’t set up DMARC records for your accounts, you should have already moved to a different host by the time you are reading this. Only support the good guys, the rest can burn in hell. I have no regrets for them.

    “You are paranoid about security, yet still only allow FTP? WTF?” I’m allowing explicit FTP over TLS. It’s regular FTP from the point you initiate the connection up to the point where you ask the server “hey, is it ok if we do this over tls?”, then hand over your credentials. The port is the same, regular FTP. It’s beyond the scope of this guide to explain anything more.

    “What are the unprivileged ports?” They are ports designed to be used by “client” programs as the initiating point in a connection. Let’s say you want to connect to a website. Your client (browser) initiates a connection using port 34521. The destination port is 80, which is a privileged (aka “official” server port). That’s why I recommended having your webgui port on a privileged port. For example, show hands, who in here knew that port 80 > port 80 traffic should NOT be allowed through a firewall? Anyone? You in the back? No one? How about port 22 > port 80? Still no one? More on this when the suricata part is done.

    “What about DNS and NTP?” DNS should be handled per interface. Only allow dns traffic destined for the pfsense interface address. If using OpenDNS and other “remote” DNS services, then of course allow traffic for those specific addresses as well. General DNS (allow any any dns rule) is not recommended.  The same applies for NTP. If you don’t want your time getting hijacked then only allow it internally on the network.

    “What is this “alien” traffic you mention?” Let’s say your network is in the 192.168.1.xxx range. Suddenly a host comes on and starts communicating using a 192.168.45.xxx range IP. There are 3 causes for this: 1) that host was connected to another network and has just got back online using your network. A smartphone for example. It should be able to figure out that it’s using the wrong network, and ask for a new IP from your DHCP server or 2) the traffic is spoofed to use a different IP or 3) That host definately does not belong to the network. In any case, alien traffic should be blocked. There are certain DoS attacks that can be completely stopped from originating from your network simply by using these rules.

    “I just installed pfsense and don’t want to learn all this!!!” Wax on, wax off young grasshopper. Spending the (just saying) 1 hour to do all these is spending a single hour to be absolutely sure you did it correctly. Remember, a firewall is not something you change daily. You have learned about aliases (which will come handy later on), how to use them in the rules, the difference between rejecting/blocking rules, what floating rules are and how to use them to ease up on the management, the difference between privileged/unprivileged ports…A simple “wax on, wax off” session was enough to give you what industry leaders spent their entire lives and entire fortunes getting certifications for, and still not understanding it.

    “I set up everything like you told me, now my SuperCoolMessenger doesn’t work!” I might come across as the all knowing guy, but I’m far from it. If that SuperCoolMessenger uses a privileged port for communication, and you did not allow it, then it’s expected behavior to break it. Set up the rule/edit the rule accordingly to make it work. Likewise, set up rules that are specific to your use case. The rules recommended here are minimal rules to help you get started with your training, and allow for basic connectivity. ICMP (ping,traceroute) for example is not covered here. You do know how to add it though ;). Take FTP for example. The explained rules allow for passive FTP which doesn’t use port 20 (FTP data port).

    Congratulations. You have now completed part one of your training. You can already slap in the face any guy that tells you “I work in the computer security industry and know WTF I’m talking about”. Your system is now behind a VERY capable firewall that filters both incoming and outgoing traffic, something that even the ISPs don’t do because, and I quote “what?we’ll end up with an ACL that cannot be managed if we added every host our clients told us”. That was my upstream ISP’s response to me saying a host of mine was under a 15 month (yes, fifteen month) DoS attack, with the complete lack of co-operation even from the attacker’s ISP and law enforcement. But then again why should they be bothered? They are not getting payed from it. Citing a guy for parking is more profitable that dealing with the actual crimes. No need to name any names here, wouldn’t want to get sued by complete and utter idiots. It would be an embarrassment.

    Next up is setting up IP lists. This will allow us to weed out the known “bad” guys, thereby saving our IDS/IPS from having to deal with them. Have patience and rest now, you have a lot of training ahead.

    Until I finish the next part though, feel free to throw me in a bonfire, flame away, scream your arguments at me, or just take notes and point out corrections. Remember, you have to be quick with the corrections due to the forum’s black hole. Otherwise, I’ll have to add a new post just for them.

    Class dismissed!



  • Great article. One comment about the unprivileged ports.

    1024 to 65535 unprivileged ports, you don’t have any control over these, make a note to remember what ports are privileged
    Name the alias something that is easy to remember, like I don’t know, how about “outgoing_ports”?

    Since pfSense is a state-full firewall, should it not be able to sense the switch of communication to the high ports, and open them up automatically? Thus we do not need to open all of them explicitly, but only the ones that correspond to remote services that your network needs to establish new connections with.

    Another thing I would add, for those with multiple LAN interfaces, the ‘allow outside’ rule, destination should be set to Not (!) and the alias with a list of all other LAN interfaces. This is to prevent different LAN subnets from talking to each other. I assume you do not want them talking to each other, otherwise what is the point of setting up multiple LAN subnets anyway. This way you do not have to add the floating BLOCK rule; as you said earlier, we should whitelist not blacklist.

    Excellent, comprehensive FAQ, I enjoyed reading it, thank you.


  • Developer Netgate Administrator

    An allow all rule on the CARP interface isn’t technically safe, either. 🙂
    Especially if you disallow general access to the GUI from non-management workstations, which is a good idea but not always followed… It’s more important on guest networks. As long as they’re taking the time to lock down their network, they may as well keep the GUI safe, too.

    pass pfsync from SYNC subnet to any
    pass TCP from SYNC subnet to SYNC subnet on pfsense_ports

    Otherwise your users on LAN can reach the WebGUI on the secondary by using its IP address on the sync interface (it will route, after all). pfsync traffic won’t route since it’s (directed) multicast so that’s safe, but not any other services on the firewall.

    Another couple tricks:
    Users on LAN can reach the firewall GUI by using the WAN address or SYNC IP address from the LAN, or any other internal interface. This isn’t so easy to block on 2.1.x, but on 2.2 you can use the “This Firewall (self)” macro to block traffic going to any IP address on the firewall.



  • @G.D.:

    Since pfSense is a state-full firewall, should it not be able to sense the switch of communication to the high ports, and open them up automatically? Thus we do not need to open all of them explicitly, but only the ones that correspond to remote services that your network needs to establish new connections with.

    No. It doesn’t matter if the firewall is stateful or not, pfsense’s default is to block everything, even high ports. If you drop the high ports from the outgoing_ports alias, It’s very likely you’ll break a lot of stuff (messengers/voip/etc). An allow any any rule is exactly that, allow everything going out. My way limits the outgoing traffic somewhat. Yes you could certainly allow only ranges to ports that are actually used.

    @G.D.:

    Another thing I would add, for those with multiple LAN interfaces, the ‘allow outside’ rule, destination should be set to Not (!) and the alias with a list of all other LAN interfaces. This is to prevent different LAN subnets from talking to each other. I assume you do not want them talking to each other, otherwise what is the point of setting up multiple LAN subnets anyway. This way you do not have to add the floating BLOCK rule; as you said earlier, we should whitelist not blacklist.

    A host on the DMZ initiates a connection to the LAN. DMZ is in the 192.168.43.xxx range. Lan is in the 192.168.23.xxx range. Different subnets, as explained earlier. I’ll try to explain the process as simple as possible. The DMZ shouts “hey, who has host 192.168.23.46?”. pfsense wakes up from its sleep, and says “hey, here, give me that packet, I know how to get to 192.168.23.46”. pfsense takes the packet and does what a router does, it routes it to the LAN subnet.
    Having different subnets is not a guarantee that they will not talk to each other. Having a firewall with correctly set up rules is (almost :p). If the rules say “if a packet comes on DMZ (or opt2,opt3, ie a floating rule), and wants to be delivered to LAN, if the address is any, just throw it out the window”. Having a NOT floating rule allows you to add even the destination interface (yes, even LAN) in the selection. Since the rule says “if the address DOESN’T match LAN…” The rule will be “invalid” on the LAN interface and have no effect. There is a catch though. Put yourself in pfsense’s place. A packet comes in on DMZ with an address 192.168.23.185. What does the rule say? If the address is NOT LAN. The source address is on the LAN though, therefore this packet does not match the rule. The packet passes through, and depending on the rest of the rules, it could actually get routed between the subnets 😄

    @jimp:

    An allow all rule on the CARP interface isn’t technically safe, either. 🙂
    Especially if you disallow general access to the GUI from non-management workstations, which is a good idea but not always followed… It’s more important on guest networks. As long as they’re taking the time to lock down their network, they may as well keep the GUI safe, too.

    pass pfsync from SYNC subnet to any
    pass TCP from SYNC subnet to SYNC subnet on pfsense_ports

    Well said. A small correction though. allow sync subnet > sync subnet. That in combination with the floating rule I suggested should completely take care of pfsense’s ports.

    @jimp:

    Otherwise your users on LAN can reach the WebGUI on the secondary by using its IP address on the sync interface (it will route, after all). pfsync traffic won’t route since it’s (directed) multicast so that’s safe, but not any other services on the firewall.

    Another couple tricks:
    Users on LAN can reach the firewall GUI by using the WAN address or SYNC IP address from the LAN, or any other internal interface. This isn’t so easy to block on 2.1.x, but on 2.2 you can use the “This Firewall (self)” macro to block traffic going to any IP address on the firewall.

    Actually that scenario shouldn’t apply if everyone uses common logic. Every interface (unless specifically needed) should only have access to pass traffic between that interface and the wan side.
    A floating rule to handle this is a blocking quick rule, with all of the interfaces selected (except sync) and the source set as any, destination sync subnet. Every traffic destined for sync should now be dropped on all interfaces. On that note, setting up carp with the secondary IP on the sync subnet, isolates every sync action (states/config changes) from other interfaces, since the 2 hosts are directly communicating on a dedicated line. This is my preferred way to set up carp. The admin interface is the admin interface, with access to mostly everything (since I need to administer servers on other interfaces), accesses the webgui, and the carp members talk to each other over their dedicated line, with no other traffic reaching that line. Technically the SYNC interface should be completely safe this way (except physical access). I’m not arguing that tightening SYNC rules is wrong, and I would actually recommend it as well. The point being though, if an attacker has access to unplug the network cable between the 2 hosts, connect it to a switch, then connect the switch to a malicious host in order to launch an attack on the 2 SYNC hosts, the fortress has already fallen and is burning to the ground 🙂

    Edit:
    Walked away then came back with an idea. If the documentation specifically says use an IP for the secondary CARP member that is on the SYNC subnet, on the master, then by default traffic is limited to that interface. How about adding a default floating rule for this to new installations, as set up out the box? (yes I do understand that it >might< break some set ups, but progress is progress 🙂 ). That will save new(ish) users from making the mistake of allowing foreign traffic on SYNC. Maybe make it a tick to enable option so that everybody is happy?

    The same can be applied to the Webgui+SSH. A default floating rule on all interfaces except the admin interface, would also save a lot of people from misconfigurations. Again tick to enable?



  • Good Read. 🙂
    Was hoping to find a detailed explanation and setup about Suricata itself, so waiting for I’m assuming part 3 in this guide.

    A default rule to isolate as you will the SYNC interface might actually not be a bad idea. I would be amazed if even 10% of all HA setups would be doing this atm.

    Maybe I should read this again when i’m not running a fever…



  • Between writing the next part (IP lists), checking the lists, making sure everything works, refreshing my rule configs (what good are howtos if you don’t follow them, right?), I can’t help thinking why the SYNC interface is not kept completely isolated (even removed from the rules page) and default rules not set.



  • Hi

    Thanks, thought no smart human being would put information in this matter. Miss some pictures? I know some stuff have pictures somewhere (maybe references) but all together is always good. Some  20 min. Youtube video? asking to match.

    Great stuff. One more time, thanks.



  • If you drop the high ports from the outgoing_ports alias, It’s very likely you’ll break a lot of stuff (messengers/voip/etc).

    Blocking rouge communicators is why a lot of people start looking into filtering outbound traffic by destination port in the first place. So, I would say, in this case ‘breaking a lot of stuff’ is a good thing.

    Having different subnets is not a guarantee that they will not talk to each other.

    If you delete all firewall rules, there will be no crosstalk between pfSense routed subnets.

    Instead of first allowing the traffic, and then blocking it, you could tweak your allow outside rules so that traffic from each of the subnets is allowed to go anywhere except other local subnets.



  • @G.D.:

    Blocking rouge communicators is why a lot of people start looking into filtering outbound traffic by destination port in the first place. So, I would say, in this case ‘breaking a lot of stuff’ is a good thing.

    The rules I recommended use the outgoing_ports as a destination, not as a source. In that case, filtering the high (unprivileged) ports >will< break a lot of useful stuff. I’m not arguing that sometimes breaking them is not useful, but breaking let’s say (simple example, don’t know the actual port off the top of my head) skype for the general public isn’t as useful. Giving a general recommendation is better, then people can tweak that to fit their actual use case. That’s why the snort blueprint and now this topic exist. Because rule maintainers made the assumption that identifying everything on the network is useful. It’s actually not useful at all.

    Let’s examine snort’s example. A rule exists to identify a simple HTTP request. While useful (in theory) to identify HTTP requests, the majority (all in use?) of OSs out there will eventually need to issue such a request in order to perform OS/program updates. Following that, a host that will never issue an HTTP request has no place on the public Internet, since it will never update. Using a 40 year old OS is just asking for it.

    The same principle can be applied here. I could instead say delete all rules and you are absolutely safe. In theory it is useful. But in practice I’m trying to provide a secure starting point without having to spend the rest of my life supporting it. Minimal breakages then tweaks is more acceptable than total breakage.
    @G.D.:

    If you delete all firewall rules, there will be no crosstalk between pfSense routed subnets.

    Instead of first allowing the traffic, and then blocking it, you could tweak your allow outside rules so that traffic from each of the subnets is allowed to go anywhere except other local subnets.

    That’s what the general floating rules suggested do. They block traffic from interface X destined for Y,Z. The interface’s rules should allow for traffic destined for WAN. The good thing about floating rules is instead of creating new rules in the future when you add an interface, you edit the rule and CTRL+click the new interface and done. Also see previous paragraph of why breaking everything isn’t as good as it sounds.



  • To show how this guide can be tweaked I’ll provide an example for an individual interface. This interface provides internet access to WIFI clients. Ideally, all internal facing interfaces should be close to this.

    In order, as shown on the interface’s rule page:

    rule 1:

    | Action | PASS |
    | Disabled | NOT TICKED! |
    | Interface | The interface you are setting up the rule for |
    | TCP/IP Version | IPv4+IPv6. Saves us the trouble of modifying it in the future |
    | Protocol | TCP/UDP |
    | Source | not NOT TICKED, type Interface’s net. Nothing else here |
    | Destination | not NOT TICKED, type Interface’s net. Nothing else here |
    | Destination port range | from DNS to DNS |
    | Log | NOT TICKED |
    | Description | DNS |

    rule 2:

    | Action | PASS |
    | Disabled | NOT TICKED! |
    | Interface | The interface you are setting up the rule for |
    | TCP/IP Version | IPv4+IPv6. Saves us the trouble of modifying it in the future |
    | Protocol | UDP |
    | Source | not NOT TICKED, type Interface’s net. Nothing else here |
    | Destination | not NOT TICKED, type Interface’s net. Nothing else here |
    | Destination port range | from NTP to NTP |
    | Log | NOT TICKED |
    | Description | NTP |

    rule 3:

    | Action | PASS |
    | Disabled | NOT TICKED! |
    | Interface | The interface you are setting up the rule for |
    | TCP/IP Version | IPv4+IPv6. Saves us the trouble of modifying it in the future |
    | Protocol | TCP see note |
    | Source | not NOT TICKED, type Interface’s net. Nothing else here |
    | Destination | not NOT TICKED, type any. Nothing else here |
    | Destination port range | from (other), select outgoing_ports to (other), select outgoing_ports |
    | Log | NOT TICKED |
    | Description | Outgoing ports |

    NOTE: We have already broken multiple things here. Remember, our rule only applies to TCP connections. VOIP for example generally uses UDP. If you don’t care about that, leave the rule as is. If you broke something you want, then set up the following optional rule:

    OPTIONAL RULE SEE NOTE ABOVE. Do NOT set up this rule unless you read the note above 3 times and once more to make sure you understood it completely.

    | Action | Pass |
    | Disabled | NOT TICKED! |
    | Interface | The interface you are setting up the rule for |
    | TCP/IP Version | IPv4+IPv6. Saves us the trouble of modifying it in the future |
    | Protocol | UDP see note |
    | Source | not NOT TICKED, type Interface’s net. Nothing else here |
    | Destination | not NOT TICKED, type any. Nothing else here |
    | Destination port range | from 1024to 65535 |
    | Log | NOT TICKED |
    | Description | Outgoing ports UDP |

    Reasoning for the rule: Most UDP traffic passes through high ports. There are exceptions of course, like NTP and DNS. Add rules according to your use case.

    At this point, your interface is set up. The problem is that “noise” traffic will get logged as blocked, since you should generally enable default rule logging (harmless traffic, broadcast traffic, etc.). To clean up our logs:

    Rule 4 (assuming you did not set up the optional rule above):

    | Action | Block |
    | Disabled | NOT TICKED! |
    | Interface | The interface you are setting up the rule for |
    | TCP/IP Version | IPv4 |
    | Protocol | any |
    | Source | not NOT TICKED, type any. We select any here since broadcast will not match our interface’s subnet. |
    | Destination | not NOT TICKED, type any. |
    | Log | NOT TICKED |
    | Description | Tidy logs |

    Rule 5 (assuming you did not set up the optional rule above):

    | Action | Block |
    | Disabled | NOT TICKED! |
    | Interface | The interface you are setting up the rule for |
    | TCP/IP Version | IPv6 |
    | Protocol | any |
    | Source | not NOT TICKED, type any. We select any here since broadcast will not match our interface’s subnet. |
    | Destination | not NOT TICKED, type any. |
    | Log | NOT TICKED |
    | Description | Tidy logs |

    Those 5 rules (assuming you did not set up the optional rule) are the only rules that exist on this interface. Again, for MY use case.

    This guide (as mentioned) is not a spoon feed me guide. Everything I suggested should be created as suggested. Of course I can’t cover each and every single use case out there. That’s where you tweak the suggested configuration to suit your needs.



  • IP lists
    Now that you have completed the first part, it’s time to put the knowledge you have gained to good use. We’ll be setting up IP lists to help us relieve suricata from the burden of analyzing unneeded packets, by reducing the number of active rules.

    There are two critical missing features from pfsense though, which would greatly help us in maintaining our lists. The first is that the URL (and URL table) aliases are missing the part where you tell them when to download updated lists. As it stands now, the auto update will check occasionally (update frequency dropdown) but not as often as we would like, with out a way to specify different times for different lists and if there was no update will not check for another week (please correct me if I’m wrong). The second one is that having duplicate entries (multiple lists containing an IP) doesn’t get handled, so we end up using more RAM than we would like, and passing a packet through multiple inspections before deciding what to do with it. A feature request for the pfsense team is to add a way to say list 1 should be updated at 1:20am, every day, list 2 should be updated once per hour, list 3 should be updated once per week on Sunday 11:30pm, and so on. In other words, add a way to be able to setup scheduling for the alias updates. Bonus points if de-duplication is handled as well, with a way to say de-duplicate the lists on, say for example, Monday at 3:00am.

    A forum member, BBcan177, was kind enough to create a script containing the necessary functions missing. The script was designed to keep snort IP reputation lists up to date, but we’ll adapt it to our needs.

    We’ll use aliases to keep a large number of IPs in a rule. This allows us to set up quick floating rules for a number of interfaces, keeping our per interface ruleset to a minimum. Remember, incoming should always be blocked, outgoing should always be rejected. In the future, when you add an interface, instead of copying existing rules to that interface, you just edit the existing quick floating rules and CTRL+click the new interface and you are done :).

    If BBcan177 passes by this thread, please provide the script for public downloading. I do understand that the script is released under GPL, but I’m not willing to take credit for the script by providing the download.

    Ok, after acquiring the script, rename “pfiprep v2.2.8 pfiprep.txt” to a simpler name (I don’t like spaces or versioning, since it will interfere with our cron job later on let’s say badips.sh for example). Rename pfiprep v2.2.8 pfiprepman.txt" to “badipsman.sh”. No longer needed, script is renamed to a simpler name on github. Then we’ll change a couple of things that need to be changed in order for pfsense to do our bidding.

    System>Advanced>Firewall/NAT.

    Change:

    | Firewall Maximum States | 1000000 (1,000,000) |
    | Firewall Maximum Tables | 10000000 (10,000,000) |
    | Firewall Maximum Table Entries | 10000000 (10,000,000) |

    Why so high limits? States is what pfsense will keep track of. Unless running mini-pfsenses (<512MB) then you are safe to increase this limit. Tables is self explanatory, saves us the trouble to increase it when we create 1mil aliases :P. Table Entries is what actually needs to be increased. The lists can contain a huge number of IPs. We’ll not be running anywhere close to 10mil per list, but it doesn’t hurt to give us a little leg room.

    TIP WHEN USING CARP
    Make the following changes on the CARP slave first, until you reach and follow the BLUE text

    Download the scripts (Diagnostics>Command Prompt) and upload them on the CARP master. More on this later.

    With the limits lifted, it’s time to get the script onto pfsense. Head over to Diagnostics>Command Prompt.

    1. Use the upload part to well… upload both parts of the script. After hitting upload, look to the top to see where the file ended up. It should show something like “Uploaded file to /tmp/badips.sh”.
    2. Command prompt “mkdir /home/badips” then “ls -la /home/badips” to check that owner is root and group wheel.
    3. Use the command part to “mv /tmp/pfiprep* /home/badips/”.
    4. “chmod +x /home/badips/pfiprep*” to make them executable.
    5. “ls -la /home/badips/” and make sure the owner is root and group wheel.
    6. “mkdir /usr/local/www/badips” this is the folder that will hold our de-duplicated copies of the lists, in order for us to import it into the aliases, and various bits and pieces needed by the script. “ls -la /usr/local/www/badips” and make sure its drwxr-xr-x, owner root, group  wheel. The permissions need to be correct so that we can read the list file from other hosts (keep a single SPAMers list on pfsense, and pull it from multiple email servers for example) and the script can write to it. They don’t need authenticated access to pull the lists, since it’s a simple file on pfsense’s webserver. Yes you do need to allow access to the webserver (webgui) first. Depending on your use case it’s either acceptable or not. I don’t pull the files from other hosts. The files need to be in a web accessible place anyway so that the URL alias can pull in the lists.
    7. “mkdir /usr/local/www/aliastables” for the script’s tier functionality
      7) You need to convert the script from DOS endlines to UNIX ones using the tr command. Enter the next commands as shown, replacing the name with your custom one:
      No longer needed since moving the script to github. Use the Download gist link on github to get a tar archive with the two scripts

    Remember to use the following when editing the script:

    userfolder=/home/badips (NO TRAILING /)
    pfdir=/usr/local/www/badips/ (don't mix them up!)
    

    Diagnostics>Edit File>/usr/local/badips.sh (or whatever you called it) and make the two absolutely needed changes above. While editing the script, read through the comments and make any necessary changes (other than userfolder and pfdir). FIRST TIME RUNING IT CHANGE “bypass=no” TO “bypass=yes”, otherwise the script will fail to run. Now is the time to select your favourite lists.

    NOTE: Some IP lists are almost guaranteed to include False Positives. Choose the minimum required and adjust accordingly. See comments in script when removing a list. Eventually I’ll provide a complete list of all the lists (no pun intended) that I personally use, with the least amount of false positives). I prefer to keep the stuff about the guide that stays relatively unchanged (don’t forger the forum’s black hole problem) at the start, then add the things that do change over time. Look later in the thread for the recommended lists.

    ssh into your pfsense, change to /home/badips.
    try to run the script with ./pfiprep.

    If you are not using CARP, skip below the BLUE text. If you are using CARP, MAKE SURE YOU FOLLOW THE BLUE TEXT.
    NOW MAKE THE SAME CHANGES ON MASTER!!! Use Diagnostics>Command prompt to download the scripts from the slave and upload them on the master. This way the desired lists will get replicated on master.
    AFTER completing the changes on the master as well, you need to edit both master’s and slave’s script and change “bypass=yes” to “bypass=no”. FROM NOW ON, ONLY DO THE CHANGES ON MASTER, SINCE REPLICATION WILL TAKE CARE OF REPLICATING OUR ALIASES AND FLOATING RULES ON THE SLAVE AS WELL. You can thank me later.

    Edit the script and change “bypass=yes” to “bypass=no” after initial run
    If you don’t have the cron package installed, install it now.
    Services>Cron.
    Add a new cron entry as shown below:

    | minute | 40 (since no other cronjobs are set up near that timeframe) |
    | hour | * |
    | mday | * |
    | month | * |
    | wday | * |
    | who | root |
    | command | /usr/bin/nice -n20 /home/badips/pfiprep >> /home/badips/download.log 2>&1 |

    NOTE: When I first installed the script it simply refused to download the lists. I left it overnight, and it downloaded the lists then did it’s dedup job correctly. I recommend walking away leaving it enabled, then continue the next day with the following.
    Removed because as pointed out by another member, I’m an idiot  ;D

    Diagnostics>Command prompt>Command
    "ls /usr/local/www/badips/"
    This will give us a list of the files that we need to point our URL table aliases to. It will be different for you (if you enabled different IP lists). I will only provide an example for a single random list (it might, or it might NOT be recommended for use in the end), it’s just an example to show how to use the aliases.

    We’ll pick “dShield.txt”. Head over to Firewall>Aliases>URLs>click “+” button
    I’ll give an example for a single alias, then you need to create as many aliases as you have lists enabled.

    | Name | dshield |
    | Description | Simple description to help you identify what the alias is for |
    | Type | URL Table |
    | URL | https://127.0.0.1:X/badips/dShield.txt | |
    | Update Freq. (days) | 1. This is the lowest you can go with pfsenses defaults.see note |

    NOTE:Let’s say X=43. For URL the https://127.0.0.1:43/badips/ part will be same for all other aliases, since it’s the directory where our finished lists are located. the dShield.txt part is what file you want to use, and can be found by using the “ls /usr/local/www/badips/” command we used previously.
    NOTE: Update frequency should be the lowest possible value (1). This means the lists will be checked once per day by pfsense. The script will take care of maintaining the lists (checking for updates,downloading,deduping) externally, since there are lists that will ban your IP if you check for updates too often. pfsense will just check the local file, and if an update is made, it will refresh its aliases. This is where URL Tables individual scheduling could help if it was implemented directly into pfsense, since we could bypass the script and directly check/refresh everything in pfsense and some lists can be updated 4 times a day, while others once a week. Hope the pfsense developers add this feature.

    Keep adding other aliases as needed. When done, Firewall>Rules>Floating.
    Add a new rule to the BOTTOM of other existing rules.
    The rule should be:

    | Action | Block |
    | Disabled | NOT TICKED! |
    | Quick | NOT TICKED!!! |
    | Interface | WAN and WAN ONLY! If running multiple WANs then select all WAN connections. NOT internal ones. |
    | Direction | any |
    | TCP/IP Version | IPv4, since currently the lists are for IPv4 addresses |
    | Protocol | any |
    | Source | not NOT TICKED, type “Single host or alias”. start typing ds… and dshield should pop up. Select it. Make sure the red box says dshield |
    | Destination | not NOT TICKED, type any. |
    | Log | NOT TICKED |
    | Description | Inbound dshield |

    Save and apply. Mouse over dshield (in the Destination column), and a pop up will show up showing some IPs in it. If the popup is empty, it means that the alias is not correct. If it shows 1.1.1.1 it means the list was empty to begin with (handled by the script).

    Next up, our internal rule using this alias:

    | Action | Reject |
    | Disabled | NOT TICKED! |
    | Quick | NOT TICKED!!! |
    | Interface | ALL internal facing interfaces EXCEPTSYNC. If you select SYNC, you risk breaking syncs since a list could contain private addresses.NO WANs!!! |
    | Direction | any |
    | TCP/IP Version | IPv4, since currently the lists are for IPv4 addresses |
    | Protocol | any |
    | Source | not NOT TICKED, type any. |
    | Destination | not NOT TICKED, type “Single host or alias”. start typing ds… and dshield should pop up. Select it. Make sure the red box says dshield |
    | Log | NOT TICKED |
    | Description | Outbound dshield |

    Save and apply.

    Keep adding the floating rules after adding your desired URL table aliases. Remember, there are two rules to each alias. One is WAN facing, blocking and one is internal (LAN,DMZ,opt1,opt2,except SYNC!), rejecting.

    You have now set up the IP lists to download automatically (the script also has other functions), set up a way for pfsense to become aware of them,  and added the necessary rules.

    Keep checking this topic for a list of the recommended lists to use. I’ll add this eventually.

    So far we have limited traffic using pfsense.  How far are we in our suricata goal though?
    An example is the emerging-tftp.rules category. This category contains rules that are explicit to port 69, and guess what? We already handled this in pfsense, so no need to enable that category.
    Another example is the  emerging-compromised.rules rules. Since this category detects traffic to IPs listed in Emerging Threat’s Compromised list, and we’ll use this list, I’ll let you figure out if it needs to be enabled 😉

    That completes (almost, the recommended lists are missing) part 2 of this guide. Next up… suricata!

    As always, feel free to call me an idiot (as above), flame me, or add your comments and corrections 😄

    Keep checking this threat, the interesting stuff has just begun…



  • Recommended lists.
    This MAY or MAY NOT get changed in the future. Please go through the thread in the future to see if an updated version is available. I’ll try to keep the updates to posts, for as much as I can, after that, I’m afraid to cause the forum’s black hole.
    The left column is my alias name, and the right column is the file (/usr/local/www/badips/$file), See post above how to add this to the aliases

    abuse_palevo AbusePalevo.txt
    abuse_spyeye AbuseSpyeye.txt
    abuse_zeus AbuseZeus.txt
    alienvault ALIENVAULT.txt
    atlas_attacks Atlas_Attacks.txt
    atlas_botnets Atlas_Botnets.txt
    atlas_fastflux Atlas_Fastflux.txt
    atlas_phishing Atlas_Phishing.txt
    atlas_scans Atlas_Scans.txt
    atlas_ssh Atlas_SSH.txt
    ciarmy CIArmy.txt
    danger_rules DangerRulez.txt
    drg_http DRG_http.txt
    drg_ssh DRG_SSH.txt
    drg_vnc DRG_VNC.txt
    dshield dShield.txt
    et_comp ET_Comp.txt
    feodo_bad Feodo_Bad.txt
    feodo_block Feodo_Block.txt
    geopsy Geopsy.txt
    iblock_badpeer IBlock_Badpeer.txt
    iblock_bt_fs IBlock_BT_FS.txt
    iblock_bt_hijack IBlock_BT_Hijack.txt
    iblock_bt_spy IBlock_BT_Spy.txt
    iblock_bt_web IBlock_BT_Web.txt
    iblock_onion IBlock_Onion.txt
    infiltrated Infiltrated.txt
    malc0de malc0de.txt
    malware_group MalwareGroup.txt
    mdl MDL.txt
    nothink_bl NOThink_BL.txt
    nothink_malware NOThink_Malware.txt
    nothink_ssh NOThink_SSH.txt
    openbl OpenBL.txt
    p24_spamcop p24Spamcop24.txt
    shunlist Shunlist.txt
    spamhaus_drop Spamhaus_drop.txt
    spamhaus_edrop Spamhaus_edrop.txt
    sri_attackers SRI_Attackers.txt
    sri_cc SRI_CC.txt
    tor TOR.txt
    virbl VirBL.txt
    vmx VMX.txt
    watchguard WatchGuard.txt

    A bit of searching around in the script should be enough to match my list to the lists enabled in the script. I could provide a script snippet showing the enabled lists, but meh 😛
    This particular list of lists (again, no pun intended) is in use on a production network, providing connectivity to servers and clients. I can’t remember the last time I had to fiddle around with the lists due to a false positive. I do remember the last time was removing alienvault from the lists. Been running like this for a few months now (on an old customize version of the script, and that’s why I had to edit the above post multiple times, migrated to a newer version).

    It’s a total of 44 lists. Double that (each list needs 2 floating rules) and you only have to set 88 floating rules.
    A reason why I don’t use the script’s tier functionality (combining multiple lists to a single alias) is because since this is a production network I might have to identify false positives and make changes fast to it. Haven’t had to change the floating rules for a while though :D.

    Now, class, finish your homework by the next time. And make sure you brush up on the previous parts when that time comes . Hint: writing custom suricata rules for use on a network gateway. Yes diving straight to the deep waters. You’ve already got so far in your training, yet still fail to understand the reason for all this work…tsk…tsk…young grasshoppers…
    You are already ahead of the curve, just the information provided in these two parts could land you a 5K/month job as a network administrator for a Fortune 500 company. Trust me, you know stuff the competition doesn’t even imagine exists. Next up, we’ll take your training to the next level and offer you the chance to become one of the best network administrators/security experts in the world by awakening the beast…



  • Suricata

    Everyone should have completed the two previous parts of the guide before moving on to this part. This sets everything up so that when the beast is truelly awaken, it will work in your favor, instead of against you. Some instructions in this part (and explanations) apply equally to suricata and snort. I’ll mention only suricata, since this IS the suricata topic after all.

    I know most of you are anxious and want to dive right in. First things first. We need a “wax on, wax off” session first.

    Who will be our attackers
    Bet you didn’t see that coming.  Our attackers are persons/corporations/state-funded-actors that are not previously identified. Since we are using IP lists to block traffic from known sources, if the attacker’s packets pass through pfsense (reaching the wan side is ok, since known are dropped) then the attacker has not attacked a critical sensor system (one that adds to the lists). Keeping with snort’s topic’s NSA theme “the SIGINT simply isn’t there”.

    Since we have already identified the “usual suspects”, we are dealing with, let’s say 20% of the total bad traffic (assuming 80% is known).

    How will you know that packets are arriving from an unknown bad IP?

    1. Those packets will reach a legitimate port and pass across pfsense to a server/process listening behind pfsense. In this case they will be analyzed in transit (or intercepted if running in IPS mode) by our fire breathing dragon (suricata).
    2. Those packets will NOT reach a legitimate (open) port and will be dropped by pfsense’s default block rule. We could monitor the default rule and set up bans based on that. We’ll do it in a far more interesting way, using suricata.

    The way suricata works on pfsense can be thought as being as close to the network as possible. Packets (technically copies of packets so far, since we are not running in IPS mode) are passed through suricata at the same time pfsense gets the actuall packet. So let’s say a bad source packet arrives at our firewall. The packet is copied, then the original is forwarded to pfsense at the same time the copy is forwarded for analysis. The original will be dropped by pfsense, but the copy can fire up an alert.

    Why is this useful? We can leverage our suricata systems to keep their own IP lists, in the form of banned hosts. Bad packets (bad refers to a closed port destination) will be dropped by pfsense with no indication to the attacker, and at the same time our attacker will be identified and put on our own list. If the attacker decides to perform another port scan, for example, a week later, suricata will still pick up on that and fire up a new alert, refreshing the initial ban day. So instead of being 28-7=21 days from now, with the refreshed alert is will be 28 days from now (pfsense is already dropping the original packets, since that host is now on an internal alias, and the packets will not be logged). You can see that as long as the attacker attacks us, he will be kept on a perpetual ban. The harder they try, the worst it is for them.

    NOTE: The rules explained in this part are rules that should be run on each and every single gateway system. They are rules especially designed for such use, since our bastion (first systems that can be reached from the network) must be our bastions, our “impenetrable” outer perimeter. Quotes because as proven by history, even the mightiest of the castles have fallen.

    QUIZ: Who remembers why low (privileged ports) are useful?

    Using the low ports we can detect port scans looking like legitimate traffic reaching “official” server ports. Let’s examine a random example:
    The attacker sends traffic for port 514 (syslog), containing a legitimate syslog entry.
    We have 2 choices to deal with this traffic. Set up a rule especially for it, or set up a general unused ports rule. Remember, suricata will add a new alert each time a rule is triggered, which means if an attacker scans 80 low ports, there will be 80 alerts logged for it, making the blocked tab page a mess. If the alert matches the previous one though, only the timestamp is updated, which keeps everything running smoothly. Further analysis can be performed through the logs.

    In other words, packets from a source NOT our home network (or belonging to an external network) and a source port of any, reaching a port NOT in use on our home network, will alert us.
    Congratulations young grasshopper, you have just created the world’s top snort/suricata rule. It is internally referred to in The Company as “The Golden Standard for writing IDS/IPS rules” or, depending on the day “The two rules to rule them all”. Why is it the golden standard? Those of you that failed to see that it is impossible to cause a false positive by this rule so far are dangerously close to failing the class. Unless you ignored a port in use (which shouldn’t happen all that often) a packet that triggers this rule is a packet that will not route across pfsense. Simply put: a legitimate alert.
    Be extremely careful with the knowledge. It can be used for good, or it can cause great harm. These two (a TCP and a UDP), the “The two rules to rule them all”, rules accounts for 10K monthly banned hosts (hosts set up to be unbanned after 28 days, but trigger the rule before that).

    Let’s examine why the rule is so successful. We have already mentioned its ability not to cause false positives. One of its other abilities is that it acts as close to the actual protocol as possible. There is no need to pass a packet through deep packet inspection to see what is actually in it, since it is a packet that would otherwise be dropped. Saves us the processing and provides a further ability: detection of traffic that would otherwise be missed by suricata.

    “Wax on, wax off” session: HTTP traffic comes in wanting to initiate a connection to pfsense’s HTTP port. We are already running HTTP servers behind pfsense, but they use a different IP for that. A legitimate connection attempt to a non legitimate destination.

    So, a packet has arrived that is sourced NOT from our network, having a source port of any, and is destined for a NOT HTTP server, to an HTTP port. Get it? 😉

    The next question will separate those that want to use this topic as a way of getting a 10K/month job (5K was when you finished a previous part, you are now in the position to basically walk in to any company you want and tell them to hire you for an arbitrary salary) and those that are truelly interested in network security.

    Should a low (<1024) port communicate directly with another low port?
    The answer is no. Low ports should always be used as the DESTINATION of the connection, NOT the source.
    So, a packet that claims to come from a port which starts at 0 and ends at 1023 shouldn’t be reaching a port that starts at 0 and ends at 1023. If most potential employers failed to see why you are one of the best there is in the network security industry, tell them the above rule. They will not be able to find the pen to sign your contract fast enough.

    You now have the necessary knowledge of setting up custom rules for use in pfsense’s suricata use case (a network gateway). This allows you to shift processing from Layers 5-7 of the OSI model (actually opening up the packet and looking inside, which is useless if using encryption), to Layer 4 (TCP is layer 4, you are forgeting the first level, physical (cables)). This offers lower load, since you are spending less time processing the traffic, while at the same time not reducing your security. Simply put, you have just awaken the beast…

    Those of you that reached this far and “saw the light” will go on in their lives becoming the top of your classes in the best universities of the world, and upon graduating from said universities, will pursue a career in network security, quite possibly for an alphabet-soup-agency. What ever your future goals are, I have a simple request from you: “Always try to the best at what you do. There is the first place and there is the last place. Second place is not an option.”

    You have now set up the battle so that the sun is behind you, your shield is on your left, the hill is on your right, and the enemy is straight ahead. Sun Tsu was right, you are now only dealing with determined attackers. Suricata is screaming “bring it on”.

    That completes the “static” part of this topic. Since old posts cannot be edited, I wanted to put this as close to the top as possible. The IP lists list and the enabled suricata rules will be posted from time to time to this thread, so please check the entire thread for the most up to date version.

    FAQs
    "Why didn’t you just gave away your custom rules?" Because quite frankly I’ve spent 17 years of my life studying and don’t want to see my efforts being used as a way for an idiot to get a high paying job. Those that truelly want to learn about suricata (and snort) rules have already picked up the hints and are on their way to writing their rules already.
    “Why did you give away all this?” You having a more secure network is inversely proportional to my time spent sifting through logs, trying to identify infected/malicious/compromised remote hosts :D. That and the fact that setting up IDS/IPS systems my way has been proven to be the best way to set them up.
    “Who are you?” As mentioned in the first post, “Security through obscurity”.
    “Who do you work for?” As mentioned in the first post, “Security through obscurity”.

    Next up, and after much anticipation, suricata enabled rules. 😄



  • I have been working on a script that downloads over 50 different Blocklists and performs a duplication check to reduce the size of the data. It can download .CSV, .TXT, ,GZ, .ZIP files and also scrape from certain websites that post only a web copy of their Blocklists.

    ie : ET, Spamhaus, IBlock,  dShield, Atlas, Alienvault etc… I have been researching Blocklists for several Months and have found the current list to be beneficial in Blocking Malicious IPs.

    It utilizes a tool called “Grepcidr” to make the de-duplication work.

    It also looks at the number of IP addresses found in a /24 range and can condense the list and enter a /24 block instead. This is done in three ways

    1. Using a “max” variable, if it finds over the Max variable it will perform a local Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries on an individual Blocklist Basis.
    2. Using a “dmax” variable  if it finds over the dmax variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries at the end of the download process on all of the Blocklists together.
    3. Using a “pmax” variable, if it finds over the dmax variable it will process a /24 Block excluding Country Code whitelist at the end of the download process on all of the Blocklists together.

    So I set max to 5, dmax to 5 and pmax to 50 in my setup.

    Depending on how aggressive / conservative an admin wants to configure the processes or disable them completely and just use the de-duplication processes.

    I have been testing it for several weeks without the need for pfBlocker to reload the Alias tables. Options also exist to place the Repeat Offender Ranges into a “Match” List that can be used with “Floating Rules” to report when these suspected ranges are in your network.

    I also found a way to use the Maxmind database to make a Country Code Specific Blocklist, excluding whitelisted countries.

    Here is a snapshot of the IP count for each list (mail server lists not included)

    Orginal list of approx 700,000 IPs (worst of the worst!)

    ===[ [b]Blocklist IP Counts ]===========================

    240818 total
      127685 /home/user/pf/BadIPs.txt
      47095 /home/user/pf/IBlock_Badpeer.txt
      23423 /home/user/pf/ALIENVAULT.txt
        6406 /home/user/pf/Geopsy.txt
        6282 /home/user/pf/SRI_Attackers.txt
        5307 /home/user/pf/VMX.txt
        4818 /home/user/pf/IBlock_Onion.txt
        3692 /home/user/pf/Infiltrated.txt
        2904 /home/user/pf/IBlock_BT_Spy.txt
        2813 /home/user/pf/HackedReport.txt
        1570 /home/user/pf/ET_Block.txt
        1423 /home/user/pf/IBlock_BT_Web.txt
        1139 /home/user/pf/DRG_http.txt
        1057 /home/user/pf/MDL.txt
        1037 /home/user/pf/ET_Comp.txt
        559 /home/user/pf/SnortBL.txt
        446 /home/user/pf/Atlas_SSH.txt
        378 /home/user/pf/Greensnow.txt
        371 /home/user/pf/CIArmy.txt
        358 /home/user/pf/Spamhaus_CC.txt
        357 /home/user/pf/IBlock_BT_FS.txt
        275 /home/user/pf/MTA.txt
        239 /home/user/pf/NOThink_Malware.txt
        188 /home/user/pf/Maxmind_Proxy.txt
        152 /home/user/pf/DRG_VNC.txt
          85 /home/user/pf/dShield_Top.txt
          83 /home/user/pf/Blut_TOR.txt
          76 /home/user/pf/DRG_SSH.txt
          68 /home/user/pf/BotScout.txt
          66 /home/user/pf/Juniper_Spam.txt
          56 /home/user/pf/Snort64.txt
          49 /home/user/pf/HoneyPot.txt
          40 /home/user/pf/malc0de.txt
          39 /home/user/pf/IBlock_BT_Hijack.txt
          39 /home/user/pf/DangerRulez.txt
          37 /home/user/pf/NOThink_BL.txt
          36 /home/user/pf/MalwareGroup.txt
          28 /home/user/pf/Feodo_Block.txt
          23 /home/user/pf/Spamhaus_edrop.txt
          21 /home/user/pf/OpenBL.txt
          17 /home/user/pf/WatchGuard.txt
          15 /home/user/pf/dShield_Block.txt
          14 /home/user/pf/SRI_CC.txt
          10 /home/user/pf/Atlas_Fastflux.txt
          8 /home/user/pf/Shunlist.txt
          8 /home/user/pf/Atlas_Phishing.txt
          7 /home/user/pf/Atlas_Botnets.txt
          6 /home/user/pf/NOThink_SSH.txt
          4 /home/user/pf/Atlas_Attacks.txt
          3 /home/user/pf/Atlas_Scans.txt
          1 /home/user/pf/Spamhaus_drop.txt    <–-- These below are actually empty with
          1 /home/user/pf/ISC_top10.txt                        a “1.1.1.1” as a placeholder.
          1 /home/user/pf/Feodo_Bad.txt
          1 /home/user/pf/AbuseZeus.txt
          1 /home/user/pf/AbuseSpyeye.txt
          1 /home/user/pf/AbusePalevo.txt

    (Here is the status of the pfSense Alias Tabes (Groups/Tiers) update process)

    ===>  Sanity Checks PASSED, Updating Group Lists  <===

    Updating  [ IR_PRI1 ] [  ET_Comp ET_Block Spamhaus_drop Spamhaus_edrop Spamhaus_CC CIArmy AbuseZeus AbuseSpyeye AbusePalevo dShield_Top dShield_Block SnortBL ISC_top10 Snort64 ]
    no changes.

    Updating  [ IR_PRI2 ] [  ALIENVAULT Atlas_Attacks Atlas_Botnets Atlas_Fastflux Atlas_Phishing Atlas_Scans Atlas_SSH SRI_Attackers SRI_CC HoneyPot ]
    no changes.

    Updating  [ IR_SEC1 ] [  MDL NOThink_BL NOThink_SSH NOThink_Malware DangerRulez Shunlist Infiltrated DRG_SSH DRG_VNC DRG_http Feodo_Block Feodo_Bad WatchGuard VMX Geopsy MTA Maxmind_Proxy BotScout HackedReport Juniper_Spam Greensnow ]
    no changes.

    Updating  [ IR_SEC2 ] [  MalwareGroup OpenBL malc0de ]
    no changes.

    Updating  [ IR_SEC3 ] [  BadIPs ]
    no changes.

    Updating  [ IR_IB ] [  IBlock_BT_Hijack IBlock_BT_FS IBlock_BT_Web IBlock_BT_Spy IBlock_Badpeer ]
    no changes.

    Updating  [ IR_TOR ] [  IBlock_Onion Blut_TOR ]
    no changes.

    pfSense Table Stats
    –-----------------
    table-entries hard limit  800000
    Table Usage Count        512459

    Thanks to jflsakfja, for helping to test it out.  😉 😉

    Here is a link to GitHub(Gist) where you can download the files. This provides more functionality than what is available in the vanilla pfBlocker.

    [  [b]https://gist.github.com/BBcan17/67e8c456cb399fbe02ee  ]

    There are installation instructions in the script. If you need more help send me a PM or a post.
    Please be careful when working in the shell.

    **  [ From time-time please review any Revisions in the Gist ]**



  • "also scrape from certain websites that post only a web copy of their Blocklists. "

    That last part could become quite handy indeed.
    Thanks for sharing your script.



  • UPDATE:    I have updated my script pf IP Reputation Manager to v2.3.1

    https://gist.github.com/BBcan17/67e8c456cb399fbe02ee

    If anyone needs any help to install please let me know.

    Your Feedback is always welcome.



  • @jflsakfja great write up! I have to agree with you; block everything and open up what is needed… To bad I don’t do this on my box at home, well my lan is open but my other interfaces are not. So i’m half way there.

    @BBcan177 Thank you so much for sharing. When jflsakfja mention the script the other day, I started to search everywhere (including github) but couldn’t find it 😞 I’ve been a big fan of pfBlocker but there are too many dups because of list overlap and it doesn’t allow you to import other formats… Maybe someday your script and pfBlocker can get married and have a honeymoon…

    I’m using the script basically out of the box without making any major changes to it (Other then disabling the TOR EXIT lists, enabling 2 list from the MAILSERVER section and BadIPs.

    Since pfIPRep creates and updates the following pf Tables:
    IR_PRI1
    IR_PRI2
    IR_PRI3
    IR_SEC1
    IR_SEC2
    IR_SEC3
    IR_IB
    IR_TOR
    IR_MAIL
    IR_CC
    I use them for my URL Alias name. This way, pfSense doesn’t create a new pf Table

    Url Aliases:

    I’ve created my floating rules like this:

    Example of my Outbound Block rule:

    Notice in the the description, I started it with the alias name… This could be used for label matching within your USER: ruleset if needed.

    Example of my inbound Reject rule:










  • Thanks Cino, I’m glad that you are using the Script  😉

    I suggest that you add “enable logging” on the Floating Rules so you can see what is getting Blocked.

    I have also added your recommendations for the Widget. (pfIP_Reputation.widget.php)

    The widget can be found in my Github GIST.

    Copy and save that file in  [  [b]/usr/local/www/widgets/widgets  ]

    From pfSense Status:Dashboard

    Click the “+” Icon and add the new “pf_IP_Reputation” Widget"

    You may also notice that there is also a “mastermatch” alias that can also be placed into the Floating Rules as a “MATCH” Rule. These are IP Ranges that are in the Safe Country List, but are repeat offending IP Ranges. This can help to indicate malicious behaviour.



  • @Cino:

    Example of my Inbound Block rule:

    Example of my Outbound Reject rule:

    FTFY 🙂
    Other than that, everything looks good.
    EDIT!!! MISSED THE QUICK CHECKBOX. TICK THAT!

    A small update on the suricata rules. I’m delaying their release because I wanted to enable all the rules and work my way towards an FP free install, just like snort. The trouble is that I’ve had to do all the work I did over a few years, in a few weeks. The good news is that most disabled rules are the same on snort and suricata. The medium news is that I still haven’t hit all the FPs :p.

    I haven’t made up my mind if I’ll post on the topic recommendations to disable rules no longer needed, since they are handled by the custom rules mentioned previously. Maybe I’ll add a small note to them that they can be disabled if using custom rules.



  • I have posted version 2.3.2 of pf IP Reputation Manager to my Gist.

    https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-pfiprepman_2-3-2

    No recent changes to the pfiprep script. You can replace the pfiprepman script completely with this one.

    I have also added some code to the pfSense php script      /usr/local/www/diag_dns.php

    ScreenShot  http://imgur.com/L9GeYa2

    This will allow you to click on a Blocked IP in the Firewall Log, and it will show you which Blocklists have the Blocked IP Listed. At the bottom are several Threat Sources that can be referenced as well. When 2.1.4 comes out, I will post a new Patch as there are some other changes being made in this file also.

    I have created a “patch” that can be applied to pfSense v 2.1.3 (Haven’t tested it on other versions, but should work?)

    If you don’t have the pfSense Package “System Patches”, it is available in the pfSense System:Packages list under “System Patches”

    Click the “+” Icon to add a new Patch
      Enter a Description (diag_dns.php Patch)
      In the Patch Contents Dialog Box - Copy/Paste from my Gist the contents of this
      link below:

    https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-diag_dns-php_patch

    After you paste the patch, look for this line and change it to your pf folder location.

    ```
    $query_list = grep {$hosttrim} /YOUR/BLOCKLIST/FOLDER/* | sed 's/^.*[a-zA-Z]\///';

    
      These changes need to be made also:
    
      Base Directory - **/usr/local/www/**
    
      Click **"Save"**
    
      Click **"Test"**
    
      Look at the top of the patch that it can be successfully added, then
      Click **"Apply"**


  • Here is a good example about why Reputation in a /24 network is important to monitor:

    A recent DNS amplification attack caught by Snort.

    The IP Range is listed by 5 different Blocklists, but none of them have that exact IP Range.

    The Country code for 204.124.183.211 is in the US. I currently have my script set to not apply a /24 block to a repeated offender IP Range based in the US.

    ( See ScreenShot )    http://imgur.com/xkRXSFi



  • Suricata Disabled Rules

    ALWAYS DISABLE RULES! SUPPRESSING HAS OTHER USES, HIDING KNOWN FALSE POSITIVES IS NOT ONE OF THEM!

    What this list is
    This list is the definitive guide on what rules should be edited/deleted upstream. Ideally the ET guys will come along and the next iteration of this list will be blank. Practically I’ve spent months trying to get certain rules deleted, with no success. An example:
    2010228 ET POLICY Suspicious Microsoft Windows NT 6.1 User-Agent Detected <<< so WTF is windows 7 then??? DELETE UPSTREAM

    Windows 7 was (since RTM), is (currently publicly available versions) and always will be, 6.1. Yes it doesn’t make sense. It’s actually the bitter truth. Windows 7 IS Windows 6.1. Other than Microsoft’s NSA ties, there is nothing suspicious about it. An OS used by the majority of PC users should not generate a “suspicious” alert when performing the initial updates after a clean install.

    How to use this list
    First of all head over to the interface’s categories page. Go through the list and enable all categories NOT having “DO NOT USE!” in front of them. Leave the rest disabled. There are some suricata internal rules, that will not be shown on the categories page.

    After that, move over to the rules page. Select the first entry in the dropdown, and enable all rules. Then manually search for the IDs listed, and click the icon next to them. Open a second browser tab that shows the dashboard. Look at the CPU load. Back on the rules page, hit apply. Back to the dashboard, and wait for the CPU load to return to idle. When that’s done, THEN move to the next category in the dropdown.
    That’s one way to do it. The other way is to enable all in a category,hit apply, and wait for the CPU to idle again, before moving to the next category. THERE ARE 2 EXCEPTIONS TO THIS (the following 2 rules MUST be disabled after hitting enable all, BEFORE HITTING APPLY!

    1. decoder-events > 2200003 SURICATA IPv4 truncated packet
    2. emerging-policy > 2010815 ET POLICY Incoming Connection Attempt From Amazon EC2 Cloud

    Rule 1 will generate a billion alerts per second (trust me on this) and rule 2 is (here’s the first revelation of it on the entire internet) responsible for 500MB+ RAM usage. Disabling this rule makes sure that the suricata interface does not ran out of memory while trying to load its rules.

    With our selected rules enabled, it’s time to head over to the alerts tab. Refresh it after a couple of minutes and watch for rule IDs listed here. Click the icon next to them to force them to a manually disabled state (and never having to deal with them again). While disabling a rule, make sure to keep an eye out on your second tab (dashboard). Only remove a host from the blocked list (icon next to IP) AFTER the cpu returns to idle, otherwise you risk generating an alert again since the rule has not yet been unloaded.
    Note to bmeeks: Pretty please bring back the old way of handling manually disabled rules. Manually disabling a rule from either the alerts tab or the rules page, should turn the rule into a manually disabled rule (pale yellow). Currently the rules page turns it into the rule’s default state. This is NOT recommended when using this list. Having both setting to manually disabled, allows the list to be used as it was meant to be used. Enable all, then find the 10 that need to be disabled, disable them, and apply. Rinse, repeat.

    Explanations:DO NOT use the list without reading this!
    DO NOT USE! = categories that should be disabled
    xxxxxx (number) = rule’s ID
    text before <<< = rule’s explanation
    text after <<< = why the rule was disabled

    before xxxxxx = WILL ONLY BE FOUNT ON SOME CATEGORIES.Rules that generated false positives on a suricata production network. I go with the enable all, then manually disable based on the alerts route. These are rules that generated alerts, but the rest of the rules in that category (NO # in front) are known FPs, but have NOT yet generated an FP on the suricata install. Hope that makes sense.

    and no # rules explained. BLUE rules are rules that generated FPs on suricata

    decoder-events
    2200029 SURICATA ICMPv6 unknown type  <<< breaks IPv6 tunnels
    2200079 SURICATA ICMPv6 invalid checksum <<< breaks sixxs’s IPv6 tunnel

    emerging-chat
    2001595 ET CHAT Skype VOIP Checking Version (Startup)
    #2002157 ET CHAT Skype User-Agent detected <<< obvious
    Hope that clears it up. Sorry for that, but I had to start from the snort topic’s list and work my way forward, so I had to copy over the list and manually disable the alerts as they are generated. You are welcome!

    The Definitive Enabled Rules List
    Getting a 500 internal error while trying to upload the list. Will try again (either edit this post, or a new post) The list cannot be uploaded either to this reply, or a new reply, due to a 500 error. Will see where I can paste it so that I can also edit it in the future, which should also help with keeping the thread clean.
    List is here:
    https://github.com/jflsakfja/suricata-rules/blob/master/list.txt



  • Writing custom rules for deep packet inspection
    Suricata’s intended use is intercepting the packets as they are passing through the network, reassemble them (if needed), open them up and peek inside, looking for specific patterns, and deciding whether to alert or not (or drop, if running in IPS).

    We’ll examine a specific example of a known spammer trying to send spam to our email server.
    The logs show this:
    2014-06-24T16:35:52+03:00 emailserver1 postfix/smtpd[4324]: NOQUEUE: reject: RCPT from unknown[192.168.1.1]: 554 5.7.1 Service unavailable; Client host [192.168.1.1] blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=192.168.1.1; from= example@example.comto= example2@example2.comproto=ESMTP helo=<[192.168.1.1]>

    The red part shows that the email was rejected because we have set up our email server so that it first checks if the sender is a known spammer, then decides whether to accept the email or not.
    The blue part is the server’s reply sent to the client (the spammer).

    Why not use the IP directly in an alias on pfsense? This is not the scope of this post. This post tries to show how to write custom rules based on what you are trying to detect.

    Back on topic. We know so far that we are looking for a packet that is originating from our email server (port 25 since that is unencrypted), and could end up anywhere on the internet, containing a specific reply.

    So far the rule is:
    alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any
    This will not get you far, since you detected ALL traffic originating from port 25, with no specific contents.

    Examining our reply sent back to the client we see that the easiest way to detect this specific traffic is to use the part in red shown below. This is the common part for all replies, and states the reason why the email was denied:
    554 5.7.1 Service unavailable; Client host [192.168.1.1] blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=192.168.1.1

    Ok, now we know what we are looking for, but it’s time for us to understand our alert.
    Our chosen message will be:
    Known SPAMMER
    The classification for this alert will be bad-unknown (use http://manual.snort.org/node31.html#SECTION00446000000000000000 and choose the most appropriate classtype for your rule)
    Our rule version (revision) will be 1
    Our sid (signature ID) must be a high value so that it will not interfere with other rules. Each rule MUST have a unique ID! I recommend using a value in the 9mil range. In this rule we use 9900003 because we have other custom rules before it.

    Don’t forget that we are only interested in established connections (actively communicating) and the traffic direction is from the server to the client.

    Putting it all together gives us this:
    alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any (msg:“Known SPAMMER”; flow:established,from_server; content:“blocked using”; classtype:bad-unknown; sid:9900003; rev:1;)

    An example of the alert generated:
    2014-06-24T16:09:49+03:00 pfsense1 suricata[21435]: [1:9900003:1] Known SPAMMER [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 192.168.200.1:25 -> 192.168.1.1:63651

    Since our IP is not banned, suricata will try to ban both IPs (source+destination) (ALWAYS set it up like that) but ignore the source.

    We have caught our first offender using suricata’s deep packet inspection!  😄

    The spammer will return after 28 days (since his IP will be unbanned) and try to send another spam email. He will be blocked again for a further 28 days. 1 email per month is nothing compared to 1000 per day.

    Taking apart the rule and finding flaws (if any)
    The rule should only fire up on messages originating from our email servers. Good!
    The rule cannot generate a false positive since the email WILL be rejected regardless by our email server. Good!
    The rule shows a generic alert message. Good AND Bad!
    The rule’s alert message is good in the sense that it does not give too much information about the origin of the block (no list’s name appears).This is the bad part. This also allows us to keep the alerts tab clean, since a known spammer is a known spammer regardless of where the intel is coming from. If an IP is listed on list ABC on one month, and list DEF on the next month, two alerts will be logged in the alerts tab. For now it’s nothing. Detecting a portscan using my previously recommended rule would end up with a thousand alerts logged on the alerts tab for that IP (NOT if using a generic “Incoming connection to unused port” message). This is the good part!

    Congratulations. You have now finished your training and are part of the industry’s elite in network security. You are one of the best in this field! Use the knowledge wisely./example2@example2.com/example@example.com



  • great work jflsakfja!

    You may want to add decoder-events - 2200013 IPv6 truncated packet to the list for noise. My alert file is mostly this alert. What is strange, none of these show up in the GUI, only in the Alert file itself… Because there is no IP, i’m thinking the GUI doesn’t display it. But that’s for another thread after I confirm my findings.

    
    06/25/2014-06:27:46.251740  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.250371  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.318718  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.251489  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.319106  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.319352  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.543947  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 0A B8 B0 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.319496  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.543837  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 0A B8 B0 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.986163  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 05 E7 64 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.986289  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 05 E7 64 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    
    


  • Getting the suricata logs on a remote syslog, and I’m running IPv6 tunnels because my upstream provider has vowed not to support IPv6 until AFTER they are completely cut off from the Internet (it will be the last ISP on earth that moves to IPv6). It might be due to the fact that their “3rd level” support technicians have no clue what an IP (4 or 6) address looks like. This is the same company that has to pay a daily fine for “excessive profits”.

    I’ve not seen that alert so far, but I’ll keep an eye out for it, thanks for pointing it out.

    So far the list seems to have stabilized a LOT, and I’m thinking about clearing up the list (# and no # rules). What gets on my nerves is that although the list originates from the snort topic (which is almost a year old!) most of the rules are still there.



  • I’m wondering since i’m dual stack IPv4/IPv6(DHCP for both on the WAN), that is why i’m seeing this alert… My ISP(TWC) doesn’t officially support IPv6 but its enabled and working pretty well. Better then IPv4 traffic at times…



  • I also dual stack, but have not seen that alert yet.  Same here, I get better speeds over the tunnel, than the native IPv4.

    On a side note, I’m updating the list. Mostly cleaning it up, moving the DO NOT USE! categories to the top so it’s easier to find them, removing the # and no # rules…a line added here, a line taken away there…deleting some rules, updating the counts. I said I’m rarely mistaken (mathematically proven by the world’s top universities, and the possibility of me being mistaken comes out to 0.000000000001%) but ffs guys. 5 rules on top of a count that says 4 rules are disabled should be noticed by someone by now :D.

    EDIT: Finished cleaning up. The list is now starting fresh for suricata (old rules coming over from snort that did not FP within a reasonable timeframe were removed), cleaned up and ready to go. Enjoy!



  • whitelists were renamed to passlists not long ago. That is a definite bug in the suricata package. please post a link pointing to that post, on the most recent version(ed?) topic on suricata or re-post the same thing there. I’m not the suricata package maintainer, bmeeks is  ;D

    bmeeks got a tunnel working a few days back, so IPv6 support for both packages should improve.



  • @jflsakfja:

    I also dual stack, but have not seen that alert yet.  Same here, I get better speeds over the tunnel, than the native IPv4.

    On a side note, I’m updating the list. Mostly cleaning it up, moving the DO NOT USE! categories to the top so it’s easier to find them, removing the # and no # rules…a line added here, a line taken away there…deleting some rules, updating the counts. I said I’m rarely mistaken (mathematically proven by the world’s top universities, and the possibility of me being mistaken comes out to 0.000000000001%) but ffs guys. 5 rules on top of a count that says 4 rules are disabled should be noticed by someone by now :D.

    EDIT: Finished cleaning up. The list is now starting fresh for suricata (old rules coming over from snort that did not FP within a reasonable timeframe were removed), cleaned up and ready to go. Enjoy!

    You’re dual stack and using a Tunnel? When I mean dual stack; I mean receiving IPv4 IPv6 addresses on the same WAN interface… The alerts are on that interface… Once I get my stuff fine tune, i’m going to copy it over to LAN and see if there is a difference.  I should had started with my LAN interface first but oh well…

    I noticed the counts were wrong, figured your eyes were getting crossed or something from the long post  😮 Thanks again for a great thread!

    @avink use this topic https://forum.pfsense.org/index.php?topic=73906.msg428032#msg428032 to report the issue



  • Statistics on suricata’s system usage

    While idling, the system is using about 2% CPU and 23% RAM (4GB,32bit so not all available).

    Under normal load (not extreme “ZOMG! multiple 10Gbps duplex laser links into space!”) the CPU occasionally spikes to <10% for a brief moment, and the RAM stays about the same.

    While reloading rules (when hitting save on the rules page or disabling a rule from the alerts tab) the CPU spikes to 50% with it occasionally going higher (seen up to 92% for a few seconds). CPU is a dual core, so 50% means 100% of one core. RAM usage drops at the start of the reload, and continues to slowly climb to ~23% while the rules are reloaded into RAM.

    System has been set up according to this topic, with about 50 custom rules. It’s the network gateway to a small datacenter, which also provides Internet connectivity for hosts outside the datacenter.

    Matcher is AC, suricata interface is WAN.

    @Cino : Having a tunnel means somehow getting the IPv6 addresses onto pfsense. Unless you are doing some pretty complicated stuff (routing packets to Mars, then Venus, then the end of the galaxy :D) all other interfaces that need to use those addresses (except the tunnel’s “terminating” WAN interface) ARE dual stacked ;). Clients on the LAN side can connect either using an IPv4 address or completely stop using IPv4 and use an IPv6 address. That is the definition of dual stacked 😄



  • @Cino, Thanks.  I removed my message from this thread.



  • @jflsakfja:

    @Cino : Having a tunnel means somehow getting the IPv6 addresses onto pfsense. Unless you are doing some pretty complicated stuff (routing packets to Mars, then Venus, then the end of the galaxy :D) all other interfaces that need to use those addresses (except the tunnel’s “terminating” WAN interface) ARE dual stacked ;). Clients on the LAN side can connect either using an IPv4 address or completely stop using IPv4 and use an IPv6 address. That is the definition of dual stacked 😄

    Maybe I don’t understand but then again I think I do  😉  My WAN has both an IPv4 and a IPv6 address, and so does my LAN/Clients… Before my ISP provided native IPv6; the WAN was IPv4 and I had a separate WAN(Tunneled) Interface for IPv6.  That being said, my current WAN/LAN/Clients are all dual stacked. When I only had IPv6 via a tunnel, the LAN and Clients were dual stacked while my WANs were not. Does that make sense?

    Question: With a IPv6 Tunnel setup, do you run suricata on that interface also? Or just the IPv4 WAN Interface?

    Edit: Noticed emerging-rbn-malvertisers.rules emerging-rbn.rules were not in the DO NOT USE list



  • They were supposed to remove those categories since they are no longer maintained. I’ll add them in the do not use.

    Yes that’s what I meant with dual stack.

    No need to run suricata on the IPv6 tunnel. It still alerts as is.



  • @jflsakfja:

    They were supposed to remove those categories since they are no longer maintained. I’ll add them in the do not use.

    Do you think they will? after reading the comments in your enable rule list; hell will have to freeze over…



  • Supposed is the key word  😄



  • Thank you for taking the time to write up and
    giving us this great suricata blueprint,

    maybe you can help me with this setup

    I would like to put pfsense in front of an
    existing firewall, but just as an
    Suricata IDS and also use the IP Reputation Manager script
    is this possible, if it is how would i go about doing
    so. The system that i am using has 3 ports, 1 port will be for managing
    pfs as a Suricata IDS  and the other 2 ports 1 will be connected to the wan of the other firewall
    and the other port would be connect to the isp modem.

    I would like the modem to not see the pfsense Suricata IDS box to act like its not even their and still pass the external address to the
    other firewall behind the pfsense Suricata IDS  while the pfsense Suricata IDS is still catching the nasty stuff and blocking them.



  • @Cino:

    great work jflsakfja!

    You may want to add decoder-events - 2200013 IPv6 truncated packet to the list for noise. My alert file is mostly this alert. What is strange, none of these show up in the GUI, only in the Alert file itself… Because there is no IP, i’m thinking the GUI doesn’t display it. But that’s for another thread after I confirm my findings.

    
    06/25/2014-06:27:46.251740  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.250371  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.318718  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.251489  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.319106  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.319352  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.543947  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 0A B8 B0 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.319496  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.543837  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 0A B8 B0 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.986163  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 05 E7 64 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    06/25/2014-06:27:46.986289  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 05 E7 64 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
    
    

    The GUI is not currently showing these alerts because it is not properly detecting them.  The Suricata alert log output is modified to produce a CSV (comma separated values) format.  There is a built-in PHP function that can parse a CSV file and split the result into fields.  For now the pfSense PHP code is counting fields and only showing on the ALERTS tab those alerts that parse into 13 (I think it is 13) fields.  The decoder events currently don’t output 13 distinct fields, and hence are dropped by the ALERTS tab PHP code.

    I can fix that in an upcoming update.

    Bill



  • @jflsakfja:

    whitelists were renamed to passlists not long ago. That is a definite bug in the suricata package. please post a link pointing to that post, on the most recent version(ed?) topic on suricata or re-post the same thing there. I’m not the suricata package maintainer, bmeeks is  ;D

    bmeeks got a tunnel working a few days back, so IPv6 support for both packages should improve.

    I do finally have a working IPv6 tunnel, and I did see the bug report in a separate post and responded. I will fix it.

    Bill



  • @Arist:

    Thank you for taking the time to write up and
    giving us this great suricata blueprint,

    maybe you can help me with this setup

    I would like to put pfsense in front of an
    existing firewall, but just as an
    Suricata IDS and also use the IP Reputation Manager script
    is this possible, if it is how would i go about doing
    so. The system that i am using has 3 ports, 1 port will be for managing
    pfs as a Suricata IDS  and the other 2 ports 1 will be connected to the wan of the other firewall
    and the other port would be connect to the isp modem.

    I would like the modem to not see the pfsense Suricata IDS box to act like its not even their and still pass the external address to the
    other firewall behind the pfsense Suricata IDS  while the pfsense Suricata IDS is still catching the nasty stuff and blocking them.

    What you are looking for is running pfsense (+suricata) as a transparent bridge in front of the “normal” firewall. Suricata might have trouble deciding its home net values if running as a bridge (snort did) but that’s easily corrected by manually entering the home net.

    A transparent bridge is not visible on the network. If you send a packet to one interface, and the rules allow it, it will pop up through the other interface. You can actually do some pretty clever stuff with it, provided you are using a single host (no CARP). Think of it as merging the two interfaces into a single interface, with filtering applied.

    A couple of years back I was able to access a public server hosted behind a transparent bridge, from a host on a NATed interface on that transparent bridge (let’s say the admin interface), using the server’s public IP (universally understood as NOT possible to do). Then again I’m the only person on the planet that managed to get IPv6 working through 30 year old switches 🙂

    That said, my personal recommendation is NOT to run pfsense like that. If you are trying to protect a small network, put pfsense directly as the core router (which allows you to move onto CARP if you so wish), which also saves time (money) + space (1 pc instead of 2 daisy chained firewalls) + power. Then set up firewalls on network hosts, along with other security measures (brute force protection for example) working together to protect your hosts.

    EDIT: Clarification: I’m not saying don’t ever use transparent bridges. If all you need is a single firewall host, it’s actually better to run it as a transparent bridge since the host is not visible from the network. Depending on law mandated paranoia (called certifications in the industry), that might actually be exactly what you should use.
    There are also downsides, an example is that the firewall hosts themselves don’t have internet access (cannot check for updates), as set up in a plain vanilla transparent bridge (permission is hereby granted to correct me if I’m wrong).



  • Awesome post, thanks!

    I’m trying to install pfiprep, and am getting an error when installing one of the dependencies.

    $ pkg_add -r grepcidr
    tar: Failed to set default locale
    

    Can someone please tell me what to do to fix this?


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy