Snort free Registered rules MDS fail

bmeeks

@andrzejls said in Snort free Registered rules MDS fail:

@bmeeks
OK, I got new oink code. and run "update rules" with same result, should I use "force update"?. I have free account. I followed link that you provided and had no problem getting to that site. I can only get "Community" not "Registered" nor "Subscription".

That makes no sense. What do you mean by this statement: I can only get "Community" not "Registered" nor "Subscription"?

There is a sign-in button or icon at the site. Can you sign-in with a valid email address and password at the site? If so, then either the Registered or Subscription Rules will become available. The "Community" rules are available free to anyone including anonymous users. But the other two require a valid registration with the Snort team in order to access and download them. If you cannot sign-in and access either the "Registered" or "Subscription" rules at the web site, then your Oinkmaster code is not going to work. Where did you get that code? If you got somebody's from the Internet, then it is highly likely to have been cancelled.

andrzejls

@bmeeks
Yes, you are absolutely right. I did not realized, that I needed, nor seen "sign in" button under "Registered" column. Once I signed in, gotten new oink code and run "update rules" all is working fine.
I thought that just login into Snort.org I would be recognized and I would not need to do additional sign in into Registered rules in order for oink code to work. It was misunderstanding on my part. As I said, I am new to Snort and going through this process I learned a lot. Thank you for your help and understanding my "nub" status in Snort.

bmeeks

@andrzejls said in Snort free Registered rules MDS fail:

@bmeeks
Yes, you are absolutely right. I did not realized, that I needed, nor seen "sign in" button under "Registered" column. Once I signed in, gotten new oink code and run "update rules" all is working fine.
I thought that just login into Snort.org I would be recognized and I would not need to do additional sign in into Registered rules in order for oink code to work. It was misunderstanding on my part. As I said, I am new to Snort and going through this process I learned a lot. Thank you for your help and understanding my "nub" status in Snort.

You do not have to be continually signed-in for the Oink code to work, but perhaps in your case with a new code at least one successful sign-in was required in order to fully activate the new code.

Glad you have it working now.

andrzejls

@bmeeks
Once again, thanks for your help!

andrzejls

@bmeeks
Here we go again. Oct.1st. and Snort fail to update rules. I did not changed/modify anything yet I am back to where I was few days ago. This is getting kind of enjoying and/or stupid. Is this is normal with Snort or I do not understand something?.
Screenshot from 2022-10-01 15-34-49.png

bmeeks

While I realize it may not be of great help to you, this really looks like a problem specific to your setup. If the rules servers were down or malfunctioning I would expect to see lots of folks posting here about it. There are around 25,000 or so Snort and Suricata users on pfSense around the world according to some stats I was given by the Snort Subscriber rules team two years ago.

I can think of really only two things it could be:

A problem with your registered user status and/or the oinkcode; or
An issue with your WAN IP address such as it is maybe getting blocked by the Snort rules server (sort of unlikely, to be honest).

But since it worked for a bit and then stopped, maybe look into whether or nor your public WAN IP is on a list resulting in it being blocked. What frequency do you have the "check for updates" set to? Once a day is plenty. The Snort rules are only updated on Tuesday and Thursday each week if I recall. They are not updated frequently. The Emerging Threats rules do get updated daily. If you check too often, that might result in a restriction against your oinkcode.

In what country are you located? Perhaps that is an issue ???

And I will mention this just to be thorough -- not suggesting this is your case.

Is the Oinkmaster code you have truly yours registered by you to an email address controlled only by you? Is it shared by anyone else? If the same code hits the Snort servers from multiple IP addresses, I could see how they might restrict it since the codes are not intended to be shared.

johnpoz

@bmeeks could it just be the files are being corrupted on download, I would think if he was blocked in some way, or his code was blocke/restricted why would it say the bad md5 checksum? Wouldn't it just say failed? How is there anything to get a md5 off of if nothing downloaded?

andrzejls

@bmeeks
Thanks for your timely response.
You are right stating that ~25K Snort users do not have issue that I have and, most likely, it is a problem resulting in some setup of my install/config of Snort.
I do not know as to why there would be a problem with my Snort user status. I use my real email address that I use every day, not a fake one. I do not share my email with anyone.
I set up updates to 1 (one) day intervals at 4AM. I am physically located in North Carolina USA and my ISP is Spectrum so my public ip should not and is not blocked. I use NordVPN , occasionally, on 1 (one) laptop connected/hardwired to the pfsense router (static ip). My nordVPN on that laptop is setup to use NORDLYNX technology and my LAN 192.xxx.xxx.xxx/24 is Whitelisted in nordVPN settings. pfSense router is not setup with VPN.
Should I regenerate my ionk code in snort.org? Should I run "Update Rules" or "Force Update"?

bmeeks

@andrzejls said in Snort free Registered rules MDS fail:

@bmeeks
Thanks for your timely response.
You are right stating that ~25K Snort users do not have issue that I have and, most likely, it is a problem resulting in some setup of my install/config of Snort.
I do not know as to why there would be a problem with my Snort user status. I use my real email address that I use every day, not a fake one. I do not share my email with anyone.
I set up updates to 1 (one) day intervals at 4AM. I am physically located in North Carolina USA and my ISP is Spectrum so my public ip should not and is not blocked. I use NordVPN , occasionally, on 1 (one) laptop connected/hardwired to the pfsense router (static ip). My nordVPN on that laptop is setup to use NORDLYNX technology and my LAN 192.xxx.xxx.xxx/24 is Whitelisted in nordVPN settings. pfSense router is not setup with VPN.
Should I regenerate my ionk code in snort.org? Should I run "Update Rules" or "Force Update"?

The VPN might be an issue if traffic happened to go out that route. Just guessing, though.

The difference in Update Rules and Force Update is this:

Update Rules downloads the MD5 checksum files for each rule archive and compares the content of that file (one line of text representing the md5sum of the much larger gzip archive) to the MD5 checksum file stored locally. The locally stored file is saved from the last time the rules changed. So, if the locally stored MD5 file matches what is posted on the Snort rules website that means the gzip rules archive file has not changed, and there is nothing to actually update.
Force Update begins by deleting the locally stored MD5 file thus guaranteeing the "new file" test will fail and thus download the full gzip rules archive file.

In your case, the last time you posted your full log, you are getting an HTTP Response Code 422 when attempting to download the Snort files. It downloads the MD5 file first, so that is the first error you see in the log. But it does not matter which option you use - Update Rules or Force Update - it will still fail the same way if your box cannot successfully download the files.

Here is the official definition of HTTP Response Code 422: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422. Unfortunately, there is no way of knowing precisely what that means in the case of the Snort rules download. Could be an Oinkcode problem, or might be something else the server on their end does not like.

bmeeks

@johnpoz said in Snort free Registered rules MDS fail:

@bmeeks could it just be the files are being corrupted on download, I would think if he was blocked in some way, or his code was blocke/restricted why would it say the bad md5 checksum? Wouldn't it just say failed? How is there anything to get a md5 off of if nothing downloaded?

He is getting a 422 HTTP RESPONSE error. That is a somewhat generic error from the Snort rules server. Could be any number of things.

The bad checksum error is happening because the curl download is likely generating an empty checksum file that fails the test. The Snort rules server does not send back any specific error messages other than the generic HTTP RESPONSE codes.

The most common cause of the bad checksum error is using a RAM disk without enough space to hold the entire file, but the OP says he is not using a RAM disk.

andrzejls

@bmeeks
Just to confirm, I am not using RAM disk.

andrzejls

@bmeeks
I can and I did download "snortrules-snapshot-29200.tar.gz" file from snort.org, so this is not a problem, I think.

andrzejls

@bmeeks said in Snort free Registered rules MDS fail:

Could be an Oinkcode problem

Just to make sure that I am not making errors or not following Snort proper procedure, what is the proper procedure to obtain "oink" code. If I got more than once "oink" code, should I use the most recent one or it does not matter?.

fireodo

@andrzejls said in Snort free Registered rules MDS fail:

If I got more than once "oink" code, should I use the most recent one or it does not matter?.

Hi,

i guess you should use the latest one and discard the old one. (MaxMind GeoIP Key is following this procedure)

my 2 cents,
fireodo

andrzejls

@fireodo
Thanks. Yes, that is what I use; latest oink code.

bmeeks

Sorry to be late replying again. Had to be away from home for a bit.

I am stumped over your error. As I mentioned, the vast majority of the time this error is caused by a RAM disk and not having enough free space on it to hold the files as they are downloaded. Since you have confirmed you are not using a RAM disk, that eliminates that as a cause.

Just to be complete, are you showing plenty of available space on your disk? I would assume so or you would likely be experiencing other pfSense problems. 256 MB or more of free space should be plenty to download the files and unpack them.

The fact you can login and download the file directly from the Snort site indicates your registration or subscription with them is good. Like @fireodo suggested, make sure you are using the most recently issued Oinkcode if you have gotten more than one. Not sure if older ones would still be good.

To perform the exact same type of download as the Snort package on pfSense is attempting, try this URL in your browser of choice:

https://www.snort.org/rules/snortrules-snapshot-29200.tar.gz?oinkcode={$oinkid}

In the URL above, replace the braces "{}" and the content within with your Oinkcode. So like this as an example:

https://www.snort.org/rules/snortrules-snapshot-29200.tar.gz?oinkcode=123456789

This should result in downloading the rules file using your Oinkcode in the exact same way as the Snort package is attempting on pfSense.

Your error message shows the download succeeded, but when the package code tested the md5sum of the gzip file it downloaded against what the Snort rules website reported was valid, the checksum did not match and thus the downloaded gzip is considered corrupted.

andrzejls

@bmeeks
Hi, Thank you for your support and help. I followed your instructions and I was able to download "snortrules-snapshot-29200.tar.gz" file without any problem. It got me thinking about checksum and offloading with Realtek cards. In System/Advanced/Networking there is a setting for "Hardware Checksum Offloading". I check that field and rebooted pfsense as instructed. Long and behold Snort updated successfully. I use old EVGA nForce 750i SLI MB with Reatek build in 1GB NIC for WAN. I think that was the issue all along. Did not realize, I am new to Snort, that this could be an issue with Snort. Proof in the putting will be tomorrow when Snort run through schedules update.

johnpoz

@andrzejls said in Snort free Registered rules MDS fail:

Hardware Checksum Offloading

The checkmark disables it..

If that fixes it - that is a weird one for sure.

bmeeks

Hmm... the checksum setting causing that is definitely unusual. Not that disabling the setting is bad -- it is actually suggested for Snort when inspecting traffic -- but weird that it would cause a corrupted file.

My assumption is that if that setting was causing large downloads to become corrupted you would expect many other issues related to corrupt downloads such as slow websites when browsing, failed Windows updates on Microsoft clients, etc.

It is definitely true that Realtek NICs and FreeBSD are not best friends, though .

johnpoz

@bmeeks Yeah I would think if that was causing such a problem you would see its effects elsewhere as well.