Suricata 2.0.3 Package Preview

Mr. Jingles

Gentlemenz, could we please maintain the FreeBSD spirit? I am getting very sad at seeing that Gonzo & JFL are getting into a fight. Both are people who are doing great invaluable services to the world.

Please, Gents: don't fight. It ain't worth it. Really.

@jflsakfja:

Then please explain the post count. It should be over 300, but instead it's 30. Either the forum blew up (did someone edit an old post causing the blackhole I've been mentioning) or they were deleted.

Yeah, I see something like 335 (currently) under your profile. I don't see any [deleted]. (I can't see the contents of deleted posts, but the ID shows up with that notation.) Probably a db issue, which will likely go away on a rebuild.

Hit the link for your username, then "show posts" and you should end up on a page like:

https://forum.pfsense.org/index.php?action=profile;area=showposts;u=###

where '###' is the actual account ID (it's an integer, not 'jflsakfja'). Looks normal to me.

In that case the forum did blow up and I publicly apologize for blaming (or insinuating a blame on) you or anyone else.

See? I'm not so bad, once you get to know me (in agent Smith voice).

Edit: post count back to normal. Beers all round on me.

Great.

Should we talk about this?

@jflsakfja:

The release did NOT happen <24 hours after he got the changes to you. At least have the decency to tell the truth. The package was given to you 11 days ago with the intent to merge it upstream. Bill said we found a few last minute bugs, wait till we can fix them. The package was again released for merge on the 30th, which to my books is not <24 hours.

The last public "notes" were that the package was released for merging. I didn't see any public announcement that the package was waiting on patches to be merged into it before it was available.

Don't think I'm an overreacting idiot (not saying I'm not, but…) it's the fact that even if the IPv6 bug existed in the new package, it would still be a tremendous improvement over the old package that was available. Having a bug that affects a certain number of people while waiting for a fix to it is better than having a bug that affects all people that use the package. And that's why I suggested that it should be on the top of your priorities list.

Ultimately I trust Bill's judgment. That's why I posted my opinion that we should go ahead with the new package even if the bug was there, IF Bill agreed.

And I'm one of the dozen people on the planet that acknowledge when they f*** up and apologize. I therefore apologize, in public, a second time for speaking without knowing all the details.

Supermule

Testing my smite count….1...2...3

EDIT: uhhhh it goes up when Gonzo comes on every evening GMT time.... how nice!

@Supermule:

Testing my smite count….1...2...3

EDIT: uhhhh it goes up when Gonzo comes on every evening GMT time.... how nice!

There are those of us that the forum gives a second chance in life, you are clearly not one of them ;D

@jflsakfja:

The last public "notes" were that the package was released for merging. I didn't see any public announcement that the package was waiting on patches to be merged into it before it was available.

Don't think I'm an overreacting idiot (not saying I'm not, but…) it's the fact that even if the IPv6 bug existed in the new package, it would still be a tremendous improvement over the old package that was available. Having a bug that affects a certain number of people while waiting for a fix to it is better than having a bug that affects all people that use the package. And that's why I suggested that it should be on the top of your priorities list.

Ultimately I trust Bill's judgment. That's why I posted my opinion that we should go ahead with the new package even if the bug was there, IF Bill agreed.

And I'm one of the dozen people on the planet that acknowledge when they f*** up and apologize. I therefore apologize, in public, a second time for speaking without knowing all the details.

You and others make a mistake if you think you're going to have visibility into everything. Not everything will be publicly announced.

In the end, the bug (which was long-standing, but still something that would "fail open" which is unacceptable), was found and fixed (because I asked Bill to take another look). pfSense is better for it and so is Suricata.

You did more than post your opinion, you called me a liar, but assuming that your apology above applies to this as well, I accept, and the matter can be dropped.

Even if you didn't, I think my point stands.

@Supermule:

Testing my smite count….1...2...3

EDIT: uhhhh it goes up when Gonzo comes on every evening GMT time.... how nice!

What would you like it to be?

[Edit: I've zeroed it. Let me know if that's not what you wanted.]

bmeeks

@digdug3:

After updating to Suricata 2.0.3 pkg v2.0.1 I got a lot of errors in the Widget and the Alert-tab.
Clearing the alert-logs and blocked-tab logs took care of it.

Thanks for the quick "avink" bug-fix!

Edit: Found another bug, after updating and editing an interface, the Barnyard tab gives the following error:
Fatal error: Can't use function return value in write context in /usr/local/www/suricata/suricata_barnyard.php on line 99

Edit: the "avink" bug was patched by Renato shortly after the initial merge to production and the pkg version bumped to 2.0.1

I will fix this as well. It was a last-minute patch to fix a problem another user reported with Snort that also impacted Suricata. I tested it on 2.2 pfSense with no issues. If you have 2.1 instead, maybe it's related to the difference in PHP versions. No matter, I know how to fix it and will submit a patch for this ~~and the Avink bug~~.

Bill

digdug3

Thanks Bill!

dotOne

As some people already discovered suricata seems to have problems with PPPoE connections and the log is filling up with these messages:

6/9/2014 – 20:58:10 - <error>-- [ERRCODE: SC_ERR_DATALINK_UNIMPLEMENTED(38)] - Error: datalink type 0 not yet supported in module DecodePcap
6/9/2014 – 20:58:10 - <error>-- [ERRCODE: SC_ERR_DATALINK_UNIMPLEMENTED(38)] - Error: datalink type 0 not yet supported in module DecodePcap
6/9/2014 – 20:58:10 - <error>-- [ERRCODE: SC_ERR_DATALINK_UNIMPLEMENTED(38)] - Error: datalink type 0 not yet supported in module DecodePcap</error></error></error>

According to the suricata website PPPoE should be supported so I did some searching and playing around on my test firewall.
What I did is instead of listening on the PPPoE interface I changed the listening interface on the configuration file to the underlying physical interface (em0) in my case and restarded suricata by hand

suricata.yaml:
pcap:
- interface: em0
checksum-checks: auto
promisc: yes

/usr/pbi/suricata-amd64/bin/suricata -i em0 -D -c /usr/pbi/suricata-amd64/etc/suricata/suricata_2925_pppoe0/suricata.yaml –pidfile /var/run/suricata_pppoe02925.pid

It looks like suricata can perfectly handle this and strips the PPPoE part looking at the actual data.
There are few messages reporting unrecognized ppp frames, but there are only a few of them. Will do some more checking but it looks suricata is running fine now.
It's logging and blocking as it should

So far so good.

Raul Ramos

Thanks @avink, i really appreciate this information.

I will try this (maybe tomorrow) on my real pfsense and report back.

bmeeks

@avink:

As some people already discovered suricata seems to have problems with PPPoE connections and the log is filling up with these messages:

6/9/2014 – 20:58:10 - <error>-- [ERRCODE: SC_ERR_DATALINK_UNIMPLEMENTED(38)] - Error: datalink type 0 not yet supported in module DecodePcap
6/9/2014 – 20:58:10 - <error>-- [ERRCODE: SC_ERR_DATALINK_UNIMPLEMENTED(38)] - Error: datalink type 0 not yet supported in module DecodePcap
6/9/2014 – 20:58:10 - <error>-- [ERRCODE: SC_ERR_DATALINK_UNIMPLEMENTED(38)] - Error: datalink type 0 not yet supported in module DecodePcap</error></error></error>

According to the suricata website PPPoE should be supported so I did some searching and playing around on my test firewall.
What I did is instead of listening on the PPPoE interface I changed the listening interface on the configuration file to the underlying physical interface (em0) in my case and restarded suricata by hand

suricata.yaml:
pcap:
- interface: em0
checksum-checks: auto
promisc: yes

/usr/pbi/suricata-amd64/bin/suricata -i em0 -D -c /usr/pbi/suricata-amd64/etc/suricata/suricata_2925_pppoe0/suricata.yaml –pidfile /var/run/suricata_pppoe02925.pid

It looks like suricata can perfectly handle this and strips the PPPoE part looking at the actual data.
There are few messages reporting unrecognized ppp frames, but there are only a few of them. Will do some more checking but it looks suricata is running fine now.
It's logging and blocking as it should

So far so good.

Thank you for the extra research. This is good information that maybe I can use to fix PPPoE with Suricata. I do not have a PPPoE connection to test with anymore since I switched from DSL to cable modem a couple of years ago. So the trick appears to be maybe capturing the actual physical interface behind the PPPoE interface.

I am sending you a PM so we can communicate about this some more offline.

Bill

Raul Ramos

@mais_um:

Thanks @avink, i really appreciate this information.

I will try this (maybe tomorrow) on my real pfsense and report back.

Tried on a em0 interface and it looks cool. Thanks again.

binaryjay

Thank you for the package… I was having stability issues with snort crashing when under heavy load repeatedly and decided to give suricata a try the DAY before the 2.0 package went up... :) Anyway I upgraded of course... and I have not had the service go down a single time yet. I thought snort was supposed to be the more stable of the two? Anyway...

The only issue I've run across was what I suspect to be suricata filling my /var partition to 100% with... something... causing everything else on the box to go haywire. I couldn't do much else but force a reboot and it has been fine since and will need to monitor growth of /var more carefully and see if it happens again. Logs were not persisted after reboot (thanks nanoBSD ram disk).

The reason I blame suricata was because it was the first time I've seen this happen and the only thing that changed was adding suricata and removing snort. I do have most of the logging turned OFF and the automatic log management turned ON with a very small size limit.

If I can come up with some actual proof of what happened (if it occurs again) I will post back.

This is on nanoBSD.

bmeeks

@binaryjay:

Thank you for the package… I was having stability issues with snort crashing when under heavy load repeatedly and decided to give suricata a try the DAY before the 2.0 package went up... :) Anyway I upgraded of course... and I have not had the service go down a single time yet. I thought snort was supposed to be the more stable of the two? Anyway...

The only issue I've run across was what I suspect to be suricata filling my /var partition to 100% with... something... causing everything else on the box to go haywire. I couldn't do much else but force a reboot and it has been fine since and will need to monitor growth of /var more carefully and see if it happens again. Logs were not persisted after reboot (thanks nanoBSD ram disk).

The reason I blame suricata was because it was the first time I've seen this happen and the only thing that changed was adding suricata and removing snort. I do have most of the logging turned OFF and the automatic log management turned ON with a very small size limit.

If I can come up with some actual proof of what happened (if it occurs again) I will post back.

This is on nanoBSD.

Suricata will produce a LOT of log output if some of the logging options are enabled. I highly suspect it is filling up the /var partition. I more or less left the defaults at the "out-of-the-box" values. Go to the LOGS MGMT tab and check the box to enable automatic management of logs. This will prune and rotate the logs. For nanoBSD, you should probably reduce both the LOG LIMIT sizes and RETENTION PERIODS to numbers somewhat smaller than the defaults. If keeping the logs is a big deal, then you would want to export them off the box somehow. Using Barnyard2 and outputting to a remote syslog server is one way of doing that.

Suricata's log files are all in /var/log/suricata and sub-directories underneath.

Oops…went back and read your original post more carefully and saw where you had enabled the management. How busy is your network? The logs management task only runs once every 5 minutes, and it is possible on extremely busy networks for some of the log files (say EVE, http or dns) to get quite large in 5 minutes when compared to the amount of free space that is likely to be on a nanoBSD partition.

Bill

binaryjay

@bmeeks:

@binaryjay:

Thank you for the package… I was having stability issues with snort crashing when under heavy load repeatedly and decided to give suricata a try the DAY before the 2.0 package went up... :) Anyway I upgraded of course... and I have not had the service go down a single time yet. I thought snort was supposed to be the more stable of the two? Anyway...

The only issue I've run across was what I suspect to be suricata filling my /var partition to 100% with... something... causing everything else on the box to go haywire. I couldn't do much else but force a reboot and it has been fine since and will need to monitor growth of /var more carefully and see if it happens again. Logs were not persisted after reboot (thanks nanoBSD ram disk).

The reason I blame suricata was because it was the first time I've seen this happen and the only thing that changed was adding suricata and removing snort. I do have most of the logging turned OFF and the automatic log management turned ON with a very small size limit.

If I can come up with some actual proof of what happened (if it occurs again) I will post back.

This is on nanoBSD.

Suricata will produce a LOT of log output if some of the logging options are enabled. I highly suspect it is filling up the /var partition. I more or less left the defaults at the "out-of-the-box" values. Go to the LOGS MGMT tab and check the box to enable automatic management of logs. This will prune and rotate the logs. For nanoBSD, you should probably reduce both the LOG LIMIT sizes and RETENTION PERIODS to numbers somewhat smaller than the defaults. If keeping the logs is a big deal, then you would want to export them off the box somehow. Using Barnyard2 and outputting to a remote syslog server is one way of doing that.

Suricata's log files are all in /var/log/suricata and sub-directories underneath.

Oops…went back and read your original post more carefully and saw where you had enabled the management. How busy is your network? The logs management task only runs once every 5 minutes, and it is possible on extremely busy networks for some of the log files (say EVE, http or dns) to get quite large in 5 minutes when compared to the amount of free space that is likely to be on a nanoBSD partition.

Bill

Not busy at all, it is a home network. The most workout it ever gets is bittorrent on a client being forced through openvpn in pfSense. I didn't notice any of the suricata logs ballooning unreasonably during the time I was actually monitoring an tweaking it after install. I did further reduce all of the log sizes and retention periods and it hasn't happened since but I don't feel like what I had them set to before was unreasonable.

My /var partition has not grown at all since I last wrote so I guess it was some fluke occurrence.

Anyway all is smooth running for now, thanks!

bmeeks

@mais_um:

@mais_um:

Thanks @avink, i really appreciate this information.

I will try this (maybe tomorrow) on my real pfsense and report back.

Tried on a em0 interface and it looks cool. Thanks again.

Just be aware that with this setup you are not getting the pure PPPoE traffic. What Suricata sees will include some PPPoE header info and stuff that may occasionally confuse it.

I did some research offline with avink and also swapped e-mails with some of the pfSense developers. The underlying issue with PPPoE using Suricata is that Suricata has no handler for the data link type DLT_NULL that FreeBSD returns for a PPPoE interface. The Suricata binary expects other values, but not the DLT_NULL (which is zero). It will take a patch to the Suricata binary in order for it to support PPPoE on FreeBSD. Snort seems to work with the DLT_NULL data link type that FreeBSD returns.

Bill

geudrik

Bill,

Sincerely appreciate your insight here re: why Suricata is choking on PPPoE links. It looks like a separate sensor hanging off a span is in my future :)

Pat