The next update to the Suricata 5.x package on pfSense will contain a new option for configuring Suricata to export performance stats over a Unix socket to Telegraf. It will support the input.suricata plugin.
Suricata can produce EVE JSON logs, and that data can be either written to a conventional text file or it can be made available to a Unix socket. So if someone produces a log data parser for EVE JSON, then Suricata can easily be adapted to feed data over the Unix socket. I am not familiar with Telegraf since I've never used it. So I don't know what it is capable of in terms of digesting Suricata's EVE JSON logs. The new feature I mentioned came from a Redmine Feature Request submitted a while back. And that request was specifically for Suricata performance stats (things like packets processed, packets dropped due to load, etc.).
Thanks my friend will check it out, and I will ask my isp about that because I am seeing a whole range of ips in the same scope as my public wan ip as well as ips that look to be going to different ip addresses not related to me at all and are on the same subnet as my public wan.
10$ dif for 200 vs gig - yeah that is a no brainer.. What I would go for gig as well at that sort of price point and difference.. What I really want is the up.. But if I went gig from my 500/50 it only goes to 1000/100 - and its like 20+ more a month.. And I really can not justify the download side.. I have no real use for it, and the up is only for my friends and family to share my plex.. And 50 is handling the current load without any issues.
But if I could get 1000/1000 I would jump on it for sure if only 20 more, for 10 = no brianer..
So if your going to do gig/gig - 1100 prob bit under powered.. 3100 would be what you would want. Just ordered 4th 3100 for work ;) pretty happy.. Just wish they would let me use them for some other devices with more umph... Its been a slow process.. But out of the blue my team lead said today - hey order another one of those firewalls ;)
Found an answer, took me long enough given it was right in front of me the whole time...
On Line 60 in the YAML, you can disable Stats - that probably cuts down 80% of the garbage data in EVE.
You can further disable logging (in EVE) under metadata for DNS, TLS, TCP, HTTP, etc. -- YMMV, but I feel keeping that stuff is fine since you can filter it out using something like Kibana or Splunk readily.
The Suricata GUI package on pfSense is designed to make the deployment of an IDS/IPS somewhat simpler for users new to such technology. If you are at an advanced level where you want to integrate with multiple other systems and construct on-the-fly rules using script tools, then you really should abandon the GUI part of the package and simply use the Suricata binary itself. You can do that by simply installing Suricata from FreeBSD ports. You are going to have to install all of the other scripting language dependencies from there anyway.
I am not in favor of loading up the Suricata package with a ton of new dependencies when the vast majority of users would likely not need them for a basic IDS/IPS. I'm talking about things like Python, Go, (and heaven forbid one old suggestion even needed Java! Can you imagine the security holes your firewall would have with Java installed on it?).
There is a Github site for all of the pfSense packages here. You are free to submit pull requests there. I usally am asked for my opinion, but the pfSense developers have final say in what is accepted into the package.
Go to the FLOW/STREAM tab and start increasing the TCP Stream Flow Memcap setting. The default is 32 MB (if I recall correctly), but with high core-count processors the default value may need doubling or even quadrupling in order for Suricata to start. The default value works fine on dual and quad-core processors, but higher core counts need much more Stream Memory. In your case, witih 16 cores, I would start with 256 MB and go up from there until Suricata starts reliably.
Search this sub-forum for the same error (stream memcap) and you should find posts similar to yours with the solution. One of the posts in the past included the formula to use for calculating the amount of required memory based on your CPU core count.