Sort 4 Not Downloading VRT Rules

[NollipfSense]

@bmeeks Noticed that Snort 4.0 is not downloading the subscriber VRT rules, and when I did a force update, it seems to crash...how to fix?

[bmeeks]

Working just fine for me. Here is the update check from today. The times shown are US Eastern Daylight Savings Time --

Starting rules update...  Time: 2020-03-23 13:30:00
	Downloading Snort Subscriber rules md5 file snortrules-snapshot-29150.tar.gz.md5...
	Checking Snort Subscriber rules md5 file...
	Snort Subscriber rules are up to date.
	Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
	Checking Emerging Threats Open rules md5 file...
	Emerging Threats Open rules are up to date.
The Rules update has finished.  Time: 2020-03-23 13:30:01

I suspect you've got something wrong in your firewall configuration. Are you running other things such as perhaps Squid or Squidguard? If so, they can monkey with HTTPS downloads. The Snort package does a very simply call to curl with PHP to download rules.

99% of the time in the past, when users post about issues downloading rules, it's caused by either using a RAM disk that does not have enough free space, or they have configured some other package that monkeys with HTTP/HTTPS transfers such as Squid or Squidguard and have it misconfigured.

[NollipfSense]

@bmeeks Yes, I am running Squid; however, just the antivirus clamd...and, yes, I am using RAM disk but have lots of room. The only other package that deals with HTTP/HTTPS would be pfBlockerNG-dev. If I directly download the rules from Snort's website, where to put it?

[bmeeks]

You can't just download the tarball and put it somewhere. The package does not work that way without you significantly rewriting the PHP code. It has to download and extract the contents itself. There is a lot of activity that happens under the cover with rules updates.

One of those two packages (most likely Squid) is your problem with the rules download. Disable Squid completely (you may even have to reboot the firewall to reset things) and try the Snort update again.

[NollipfSense]

@bmeeks Okay, I disabled Squid, rebooted pfSense, then update Snort but response...then, I force an update and still no response; however, it did not produce a crash. I think it's a PHP error and had started a PHP error thread in 2.5 development.

[bmeeks]

@NollipfSense said in Sort 4 Not Downloading VRT Rules:

@bmeeks Okay, I disabled Squid, rebooted pfSense, then update Snort but response...then, I force an update and still no response; however, it did not produce a crash. I think it's a PHP error and had started a PHP error thread in 2.5 development.

Refer to my post in your 2.5-DEVEL thread post. Your RAM disk for tmp is too small. I've told Snort and Suricata users repeatedly on this forum DO NOT use RAM disks with either of these packages. And the reason is because of exactly what is happening to you. Things fail mysteriously because the RAM disks are not large enough.

[NollipfSense]

@bmeeks said in Sort 4 Not Downloading VRT Rules:

@NollipfSense said in Sort 4 Not Downloading VRT Rules:

@bmeeks Okay, I disabled Squid, rebooted pfSense, then update Snort but response...then, I force an update and still no response; however, it did not produce a crash. I think it's a PHP error and had started a PHP error thread in 2.5 development.

Refer to my post in your 2.5-DEVEL thread post. Your RAM disk for tmp is too small. I've told Snort and Suricata users repeatedly on this forum DO NOT use RAM disks with either of these packages. And the reason is because of exactly what is happening to you. Things fail mysteriously because the RAM disks are not large enough.

Actually, I had it (tmp) much larger; however, I was only using one or two percent of it (350Mb), so I decreased it to what's showed in the earlier post with the image. I was trying to use more RAM why I had shift to RAM disk and on average still end up using 18% of the 16GB RAM.

Since I have enough RAM, I'll put it back to the 350MB or 400MB to see whether that resolve the issue.

[bmeeks]

@NollipfSense said in Sort 4 Not Downloading VRT Rules:

@bmeeks said in Sort 4 Not Downloading VRT Rules:

@NollipfSense said in Sort 4 Not Downloading VRT Rules:

@bmeeks Okay, I disabled Squid, rebooted pfSense, then update Snort but response...then, I force an update and still no response; however, it did not produce a crash. I think it's a PHP error and had started a PHP error thread in 2.5 development.

Refer to my post in your 2.5-DEVEL thread post. Your RAM disk for tmp is too small. I've told Snort and Suricata users repeatedly on this forum DO NOT use RAM disks with either of these packages. And the reason is because of exactly what is happening to you. Things fail mysteriously because the RAM disks are not large enough.

Actually, I had it (tmp) much larger; however, I was only using one or two percent of it (350Mb), so I decreased it to what's showed in the earlier post with the image. I was trying to use more RAM why I had shift to RAM disk and on average still end up using 18% of the 16GB RAM.

Since I have enough RAM, I'll put it back to the 350MB or 400MB to see whether that resolve the issue.

What you see on the Dashboard widget in terms of RAM disk usage is not a true picture of actual use during various tasks. For example, that can't show the amount of disk usage during rules update because odds are the update is not happening at the instant you are viewing the widget. Plus you would need to continually refresh the view to get an accurate number even then. Also remember what I said in your other PHP error thread in the 2.5 Snapshots forum -- Snort cleans up after the download attempt, so the amount of used space will shrink again even after the download fails.

Snort and Suricata download the rules archives, extracts their contents and then copies the various files within to the correct directories on pfSense. It takes a fair amount of disk space to hold the downloaded gzip archives AND their extracted content at the same time. That's why I said your RAM disk has to be very large. And "how large' is dependent on the types of rules packages you have enabled (just Snort Subscriber, ET and Snort, Snort GPLv2, OpenAppID, etc.).

So one more time, DO NOT use RAM disks with Snort and Suricata. If you insist on doing that against my advice (the package maintainer/creator), then don't post here about rules download issues caused by you not following the advice of the programmer ... .

[NollipfSense]

@bmeeks Okay Bill, lesson learned...Snort and Suricata don't like RAM Disk period!

[bmeeks]

@NollipfSense: other packages also are not fond of RAM disks. Pretty much any package that needs to download temporary files in /tmp or that wants to store logs or other "persistent" files on /var will not be happy with a RAM disk. This is because admins rarely truly understand how much disk space some packages need for certain operations, and so they grossly undersize the RAM disk based on the utilization number they see on the Dashboard Widget. Remember that widget is not really giving you a true picture of actual disk utilization at all times. It's just a snapshot at the instant you are looking at the widget.

For example, Snort and Suricata both store all their log files in directories under /var. You don't have very much space allocated there on your system for logs. pfBlockerNG stores its IP lists under /var, and when you run a RAM disk that partition is wiped clean with every reboot. That means all those files have to be downloaded again. Also means any existing Snort or Suricata log files that were there are wiped out on the reboot.

[NollipfSense]

@bmeeks I was thinking to set RAM disk for /tmp and /var to 2GB each just to utilize more RAM...then realize when pfSense 2.5 released, I would not be rebooting on a daily basis like how I am doing now with the nightly snapshots. So my strategy to utilize more RAM cannot happen this way since eventually, it will get full.

[bmeeks]

@NollipfSense said in Sort 4 Not Downloading VRT Rules:

@bmeeks I was thinking to set RAM disk for /tmp and /var to 2GB each just to utilize more RAM...then realize when pfSense 2.5 released, I would not be rebooting on a daily basis like how I am doing now with the nightly snapshots. So my strategy to utilize more RAM cannot happen this way since eventually, it will get full.

Why not just forget about RAM disks entirely? They really serve no purpose with most of today's hardware (solidstate disk drives and other forms of modern non-volatile memory). Just use your actual disk, then you have essentially no worries about space (within reason, of course).

[NollipfSense]

@bmeeks Well Bill, I was intrigued with Kiokoman's result so since I have Corona time on my hands, I set RAM disk to 2GB for each (/tmp & /var) just to experiment...all was good. I watch this for a few days...note to others, I am not advocating this setup...it's just an experiment...the developer and the maintainer does not approved

[NollipfSense]

@bmeeks said in Sort 4 Not Downloading VRT Rules:

They really serve no purpose with most of today's hardware (solidstate disk drives and other forms of modern non-volatile memory

I must say this point is well-taken.