Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM)
-
Hi everyone,
I’m building a high-performance IDS on a Raspberry Pi 5 (16GB RAM) running a 64-bit Lite OS. My goal is to use Suricata as a real-time data source for a Python-based Machine Learning pipeline designed to detect Zero-Day attacks and anomalies.
I’m currently deciding between Suricata 7.0.x (Stable) and 8.x.
Since my ML model depends on a consistent and reliable data stream from eve.json, I have a few questions:
Stability: Is Suricata 8 mature enough on ARM64 for a production-heavy research project, or should I stick to 7.0.x for better log consistency and uptime?
I/O Bottlenecks: To avoid losing packets during real-time log export, I’m planning on using an NVMe SSD. Are there specific eve-log configurations (like batching) that help with high-throughput Python parsing?
Optimization: I’m already planning to use Vectorscan for pattern matching. Are there any other ARM-specific tweaks (like CPU affinity or AF_PACKET fanout settings) that are 'must-haves' for the Pi 5 architecture?
Thanks in advance for the help!
-
@CH-Imene said in Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM):
My goal is to use Suricata as a real-time data source for a Python-based Machine Learning pipeline designed to detect Zero-Day attacks and anomalies
Reading that info makes me thing : a pure non AI, human brain, might be able to find a 'zero day'.
Software like Suricata that uses 'rules' that have to match bit-wise the content of the data to be analyzed can only exist if the potential risk has been found upfront, and translated into a rule.pfSense uses Suricata as an 'addon' package. To actually (know how to) use Suricata, you need to start , example, here.
Suricata can access most of the data flowing trough pfSense, as most of traffic is TLS encrypted. This means Suriata only sees ... random bits.
This means that you have to add a proxy service on pfSense. The idea is that all TLS (web, mail, etc etc and don't forget all the apps used by devices) traffic has to be intercepted by pfSense (and TLS decoded), so that the proxy can send the request (TLS encode it again) on behalf of the LAN device's and - here it comes, it will the the proxy that receives the answer back - and can now decode the TLS (as it was the sender) so it can inspect all the data. Then it encrypts it again, and sends it to the requesting device.
Does that seem convoluted to you ? Great. As it is. This is high end Harvard like networking. Not something that you implement in an afternoon.Added to that :
Machine learning ? Using script languages ? (serious ?)
Then why are all nVidea GPU out of stock ? These don't use Python, C or Basic. Afaik, ML (AI) needs power and pure raw massive speed. So 'native machine language' it will be be. An interpreted languages are way to slow.Still, it can be done. Sure.
The proxy part shouldn't be implemented on the main network router like pfSense, but a dedicated (LAN based ?) server device.
The data inspection (like Suricata) part neither : it will need its own (LAN) device. -
Hi!
Thank you for the insights! I really appreciate the detailed breakdown.To clarify, I am a student working on this as my Final Year Project (PFE). I am fully aware that Suricata is rule-based and that TLS makes deep inspection difficult. That is exactly why I am proposing a hybrid system:
Suricata (Signature-based): To catch the 'easy' known threats using standard rules.
ML Layer (Anomaly-based): To monitor traffic patterns (like flow duration and packet sizes) for 'strange' behavior that doesn't exist in the rules. If the ML detects a high-risk anomaly, it can trigger a block even if the attack is a Zero-Day.
I’m not trying to build a 'Harvard-level' enterprise router, but rather a realistic prototype on a Raspberry Pi 5 (using Kali Linux 2026.1.1 and NVMe) to see how much Edge security we can achieve with limited hardware.
Being a beginner, I’m focused on the 'Anomaly Detection' side to overcome the limits of traditional bit-wise rules. I’d still love to hear any tips you have on optimizing Suricata’s output for this kind of research!
-
@CH-Imene said in Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM):
Suricata (Signature-based): To catch the 'easy' known threats using standard rules.
You can't successfully accomplish this due to the inherent encryption of all the payload data in each packet. The only way the signatures can function correctly is when they see unencrypted or plaintext data. The only way that can happen is if you configure a MITM (man-in-the-middle) proxy so the TLS/SSL session encryption can be broken for inspection and then re-established for forwarding. If you let Suricata scan encrypted data packets, it is highly likely to generate many false positives due to the random nature of the encrypted data bytes haphazardly matching the conditions of a particular rule.
The only thing you can reasonably do with encrypted data is see into the SNI header for HTTPS, but even that will soon be encrypted as well with the new web standards. IDS/IPS at the network perimeter is rapidly losing its usefulness because of encryption.
Email traffic is almost 100% encrypted now as is 99.9% of all web traffic. Even DNS is rapidly switching to encrypted packets with the increasing use of DoT and DoH by clients.
Large enterprise customers that have strict device management controls can enforce use of a MITM proxy by placing the required certificates on their company-owned clients and redirecting traffic flow accordingly. That's not generally feasible for home users or even small business users.
-
Hi @bmeeks
I plan to use Suricata as a tool within my security system, not as the entire system. Suricata will be configured with rules for three known attacks, which is sufficient for this part of the project. The Machine Learning layer will handle the rest, analyzing data for unknown or abnormal behaviors.I am fully aware that Suricata works based on existing rules (either built-in or custom ones I provide), so its role is mainly to detect predefined threats, while the AI complements it by detecting new patterns.
I would be very happy if you could tell me whether this approach is possible or not, and I would also greatly appreciate any additional suggestions you might have, as this is my final year project!
-
@CH-Imene said in Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM):
Hi @bmeeks
I plan to use Suricata as a tool within my security system, not as the entire system. Suricata will be configured with rules for three known attacks, which is sufficient for this part of the project. The Machine Learning layer will handle the rest, analyzing data for unknown or abnormal behaviors.I am fully aware that Suricata works based on existing rules (either built-in or custom ones I provide), so its role is mainly to detect predefined threats, while the AI complements it by detecting new patterns.
I would be very happy if you could tell me whether this approach is possible or not, and I would also greatly appreciate any additional suggestions you might have, as this is my final year project!
The rules generally work by inspecting packet data payloads looking for specific byte patterns that match say a packed virus executable or some command and control sequence from a control server to a newly installed bot. But when the packet payload is encrypted due to SSL or TLS session connections, the byte patterns are randomized and the rule will no longer match on the content. The whole idea of encryption is to render the packet payload unreadable and undecipherable by anyone except the intended recipient.
I think you need to research exactly how Suricata rules work to detect threats, and then examine how that will work when the payload data being inspected is encrypted and thus just a bunch of randomized bytes. IDS/IPS worked great years ago before the vast majority of network traffic was encrypted. But the widespread adoption of end-to-end encryption has rendered most IDS/IPS rules toothless.
Now, if for the purposes of your project the three known attacks are conducted using "in the clear and non-encrypted payloads", then Suricata will work fine for that. But what we are trying to tell you is that does not represent the real world today where almost all traffic is encrypted.
You will only be able to match source and destination IP addresses and ports with encrypted traffic, and with some SSL connections, you can see the SNI and therefore the host or domain name the traffic is targeting. But that is not always super helpful, and the firewall engine is much more efficient at seeing and blocking on IP addresses and ports than Suricata. It will use much less CPU for that than a tool such as Suricata or Snort.
-
@bmeeks said in Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM):
Email traffic is almost 100% encrypted ...
The transport, yes.
As I have my own 'postfix' that handles all incoming and outgoing mails for all my domains, my 'inboxes' = the place where my mails are stored, are actually plain text.
Mails, which are actually files, the entire thing, the body and the headers have to be readable so I (the mail server, with tools) can do some basic SPF, DKIM, DMARC and other checks on it.
I feed incoming and outgoing (!) mails trough a filter called 'amavis', which contains a anti spam and anti virus, and it can read ZIP and other archive files, it will ban/block EXE, COM and other executable files. I could even throw Suricata at it, but never did this.Is this save and secure ? I don't know. I'm not going for presidential elections ( ^^ ) and I consider my server as save as I'm a low profile guy : I don't deal with secret state info, medical records, fiscal stuff or financial traffic.
I guess, when we all start to use encrypted mails, a bit like what PGP proposes, the mail body itself encrypt, services like gmail, hotmail etc will be closing their business as 'they' can't exploit your mails anymore, so they can't make $/€ anymore.
-
@Gertjan said in Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM):
@bmeeks said in Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM):
Email traffic is almost 100% encrypted ...
The transport, yes.
As I have my own 'postfix' that handles all incoming and outgoing mails for all my domains, my 'inboxes' = the place where my mails are stored, are actually plain text.
Mails, which are actually files, the entire thing, the body and the headers have to be readable so I (the mail server, with tools) can do some basic SPF, DKIM, DMARC and other checks on it.
I feed incoming and outgoing (!) mails trough a filter called 'amavis', which contains a anti spam and anti virus, and it can read ZIP and other archive files, it will ban/block EXE, COM and other executable files. I could even throw Suricata at it, but never did this.Is this save and secure ? I don't know. I'm not going for presidential elections ( ^^ ) and I consider my server as save as I'm a low profile guy : I don't deal with secret state info, medical records, fiscal stuff or financial traffic.
I guess, when we all start to use encrypted mails, a bit like what PGP proposes, the mail body itself encrypt, services like gmail, hotmail etc will be closing their business as 'they' can't exploit your mails anymore, so they can't make $/€ anymore.
While I was not clear about it in my reply, the context for my 100% encrypted remark was the transport layer and scanning/securing email at the network perimeter as it traverses the firewall. I guess someone could run their mail server on the firewall and do the email transport decryption and subsequent scanning there, but that would open up an entirely new level of potential vulnerabilities
. -
@bmeeks said in Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM):
I guess someone could run their mail server on the firewall
A Mail server on the firewall, like pfSense ?
No way ... not for me, not even behind pfSense on a LAN. -
@Gertjan said in Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM):
@bmeeks said in Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM):
I guess someone could run their mail server on the firewall
A Mail server on the firewall, like pfSense ?
No way ... not for me, not even behind pfSense on a LAN.Yeah, I meant it as a joke. Not good to run a mail server on your firewall.
-
@bmeeks said in Best Suricata version for IDS + AI Anomaly Detection on RPi (16GB RAM):
Not good to run a mail server on your firewall.
Oh shit - it's not, damn now I have to redo a bunch of stuff..
Just a joke - hehehe
