Suricata Rule Update - 404 Error

[ccb056]

Was getting 404 error's when updating the ET rules on suricata - removed the package and reinstalled, still getting 404 errors

See below:

[bmeeks]

There is nothing wrong with the package. It's either of two things: (1) you have something else in your system blocking access to the URL (the two most common villians there are pfBlockerNG-devel IP lists or problems in Squid or Squidguard; or (2) there is a temporary connectivity issue between you and the Amazon Web Services infrastructure the Emerging Threats folks use to host their rules tarballs.

If you think it might be reason #2 above, then wait say an hour and try again. If the problem persists, see if you can browse to this URL and see the files: https://rules.emergingthreats.net/open/suricata-5.0/.

[ccb056]

Bill,

I don't believe I have any packages that would cause this issue, see below:

I also don't believe it to be a network connectivity issue - as I can access the URL you provided

I was able to get this running by populating the URL into the config age, see below:

Note - when I click "save" at the bottom of this config page - a blank page loads:

[bmeeks]

You are not supposed to put anything in that box UNLESS you host your own local and private web site for the rules. Notice the name of that box -- "Custom Rule Download URL". That's "custom" as in "not default", I want to host my own rules in some special place, etc. ... . Also read the help/hint text directly above the box.

Remove everything from that box and then save the configuration again. And uncheck the "Use Custom URL" checkbox as well.

[ccb056]

I've removed the string, and unchecked the box - the update now fails with a 404 error:

[bmeeks]

What version of Suricata and on what type of hardware are running it? Is it a Netgate appliance, and if so, which model? What is your pfSense version?

[ccb056]

Bill,

This recently started occurring on two dell optiplex machines I have in different locations

Oddly enough - I have a third machine (not a dell) in china, and I am not having this issue there:

All three machines were using the same config, and all use the same version of suricata

[bmeeks]

Just tested in a pfSense-2.4.5 virtual machine. Installed the Suricata package and everything went fine including downloading and installing the current Emerging Threats Open rules package.

Here is the Rules Update Log from that test just now --

Starting rules update...  Time: 2020-04-02 14:25:41
	Downloading Emerging Threats Open rules md5 file...
	Checking Emerging Threats Open rules md5 file...
	There is a new set of Emerging Threats Open rules posted.
	Downloading file 'emerging.rules.tar.gz'...
	Done downloading rules file.
	Downloading Snort VRT rules md5 file...
	Checking Snort VRT rules md5 file...
	There is a new set of Snort rules posted.
	Downloading file 'snortrules-snapshot-2983.tar.gz'...
	Done downloading rules file.
	Downloading Snort GPLv2 Community Rules md5 file...
	Checking Snort GPLv2 Community Rules md5 file...
	There is a new set of Snort GPLv2 Community Rules posted.
	Downloading file 'community-rules.tar.gz'...
	Done downloading rules file.
	Extracting and installing Emerging Threats Open rules...
	Installation of Emerging Threats Open rules completed.
	Extracting and installing Snort rules...
	Installation of Snort rules completed.
	Extracting and installing Snort GPLv2 Community Rules...
	Installation of Snort GPLv2 Community Rules completed.
	Copying new config and map files...
	Updating rules configuration for: WAN ...
	Updating rules configuration for: OPT1 ...
	Updating rules configuration for: LAN ...
The Rules update has finished.  Time: 2020-04-02 14:26:08

I also happen to have the Snort and Snort GPLv2 rules on this particular VM as I use it frequently to test both Snort and Suricata packages.

And just be sure it works on a routine update, here is a manual update check:

Starting rules update...  Time: 2020-04-02 14:29:09
	Downloading Emerging Threats Open rules md5 file...
	Checking Emerging Threats Open rules md5 file...
	Emerging Threats Open rules are up to date.
	Downloading Snort VRT rules md5 file...
	Checking Snort VRT rules md5 file...
	Snort VRT rules are up to date.
	Downloading Snort GPLv2 Community Rules md5 file...
	Checking Snort GPLv2 Community Rules md5 file...
	Snort GPLv2 Community Rules are up to date.
The Rules update has finished.  Time: 2020-04-02 14:29:10

It's working fine. You have something weird going on in your firewall configuration.

[ccb056]

The only difference I can think of between the two machines that do not work and the one machine that does:

I updated the packages on the two non-working machines prior to updating to 2.4.5

I updated the packages on the working machine after updating to 2.4.5

[bmeeks]

@ccb056 said in Suricata Rule Update - 404 Error:

The only difference I can think of between the two machines that do not work and the one machine that does:

I updated the packages on the two non-working machines prior to updating to 2.4.5

I updated the packages on the working machine after updating to 2.4.5

Bingo! The pfSense upgrade docs clearly recommend that you ALWAYS update pfSense first when a new version is available. Only after that should you upgrade any packages.

You need to remove the Suricata package from the non-working machines and try installing it again.

[ccb056]

I have removed and re-installed the packages, multiple times, with this option explicitly un-checked

The problem persists

Are there files/directories I need to delete on the machine after running through the un-install in the webgui?

[bmeeks]

Remove the package, and then open a command-line session to the firewall.

Clean up any Suricata directories and files you find in these locations:

/usr/local/etc/
/usr/local/pkg/
/usr/local/bin/

Change into each of those sub-directories and execute this command:

rm -rf suricata

Then reinstall the package. That should do it. If it does not, then I'm out of ideas. The PHP package determines which ET-Open rule set to download based on the version of Suricata binary existing on your system.

[ccb056]

Unfortunately its still not working

I think I will try backing up the pfsense config, and re-staging the firewalls

Thanks for your help Bill

[bmeeks]

@ccb056 said in Suricata Rule Update - 404 Error:

Unfortunately its still not working

I think I will try backing up the pfsense config, and re-staging the firewalls

Thanks for your help Bill

The last thing you could try, short of a full reinstall is this: https://docs.netgate.com/pfsense/en/latest/install/upgrade-troubleshooting.html#forced-pkg-reinstall. This worked for some users in another thread having Suricata issues. However, their problem was a failure to start due to missing libraries.

However, as that link states, a full reinstall from media is usually the best solution. What has happened is the update of the packages prior to update of the base OS left things in a confused state for the pkg utility.

[ccb056]

@bmeeks said in Suricata Rule Update - 404 Error:

@ccb056 said in Suricata Rule Update - 404 Error:

Unfortunately its still not working

I think I will try backing up the pfsense config, and re-staging the firewalls

Thanks for your help Bill

The last thing you could try, short of a full reinstall is this: https://docs.netgate.com/pfsense/en/latest/install/upgrade-troubleshooting.html#forced-pkg-reinstall. This worked for some users in another thread having Suricata issues. However, their problem was a failure to start due to missing libraries.

However, as that link states, a full reinstall from media is usually the best solution. What has happened is the update of the packages prior to update of the base OS left things in a confused state for the pkg utility.

Bill - Perfect ! I ran through the forced pkg reinstall and my issue is now resolved.

Thanks again!