Snort Won't Start, Failed to load file-other.so

jasonsansone

After a recent rule set update Snort will no longer start. The error code in the log is:

FATAL ERROR: Failed to load /usr/local/lib/snort_dynamicrules/file-other.so: /usr/local/lib/snort_dynamicrules/file-other.so: invalid file format

I submitted a bug report to Snort. Their response was "This may be a problem with how pfsense parses the ruleset directory. Please contact pfsense." #superhelpful

PfSense version is 2.4.4-RELEASE-p3. Snort package is 3.2.9.8_6. I have reinstalled the package, tried rebooting, tried to force update the rules. This has occurred on two different systems, both of which have otherwise hummed along fine for years.

Any one have any ideas?

bmeeks

I don't currently use that particular rule category, so I will have to load up a VM and test it there. At first glance, based on the error message, it looks like a potentially corrupt file in the tarball your update downloaded.

Where are you located in the world? That might help me determine if perhaps your update tarball was fetched from a different geographical server. My last rules update is showing for this past Saturday, June 22nd. What does your Update Log show as the most recent update of the Snort Subscriber Rules?

jasonsansone

Both servers are located in Oklahoma City, Oklahoma, USA and have a WAN IP to match.

The IPS Policy is "Balanced". ET Open Rules enabled are "current_events", "mobile_malware", and "web_server". Snort Test Rules, Snort SO Rules, and OpenAPPID are not enabled.

I did a force update just now on box sites. MD5 Signature date is Monday, 24-Jun-19 16:06:16 CDT.

bmeeks

UPDATE:
Just tested in a Snort VM that I happened to have running for another purpose. Enabled the File-Other shared-object rules and started Snort without issue. Verified those rules were actually loaded.

My suggestion would be to force another update to overwrite that earlier download. Perhaps the file got corrupted either during download or during the install process. Also need to mention that if you are running with a RAMDISK (highly NOT recommended for Snort or Suricata!), then make sure you have at least 256 MB of free space on the RAMDISK to hold the rules tarball and its extracted contents.

To force an update, click the Force button on the UPDATES tab.

bmeeks

@jasonsansone said in Snort Won't Start, Failed to load file-other.so:

Both servers are located in Oklahoma City, Oklahoma, USA and have a WAN IP to match.

The IPS Policy is "Balanced". ET Open Rules enabled are "current_events", "mobile_malware", and "web_server". Snort Test Rules, Snort SO Rules, and OpenAPPID are not enabled.

I did a force update just now on box sites. MD5 Signature date is Monday, 24-Jun-19 16:06:16 CDT.

Our posts may have passed each other in the ether ...

I am located in the Southern US, so likely we get our updates from the same server farm. Snort rules are hosted on AWS intrastructure.

I can force an update of my rules in the VM and try it again. I just installed Snort on that VM this morning and it installed a fresh set of rules.

BTW, you are likely picking up the SO rules via that IPS Policy (which is fine). The policy will load up a number of rules for you from the Snort Subscriber selection.

Update:
Still working in my test VM with a new forced rules update.

jasonsansone

I am using ram disks. Each (both /tmp and /var) have been set to 256mb on each machine and a reboot was performed. Tmp utilization is 0% and /var is below 10% on each server. I just performed a reinstall of the Snort package on both. Snort will now start. Not sure where the problem was, but it automagically works again. Thank you!

bmeeks

@jasonsansone said in Snort Won't Start, Failed to load file-other.so:

I am using ram disks. Each (both /tmp and /var) have been set to 256mb on each machine and a reboot was performed. Tmp utilization is 0% and /var is below 10% on each server. I just performed a reinstall of the Snort package on both. Snort will now start. Not sure where the problem was, but it automagically works again. Thank you!

The RAMDISK ran out of free space for /tmp. I guarantee you. Have seen it happen here many times. You won't see the out-of-space condition after the fact because the Snort cron job cleans up behind itself by deleting the temp files it created (thus your RAMDISK space magically reappears). Look through the pfSense system log and I suspect you will find an out-of-space error message.

I really DO NOT recommend running a RAMDISK with either Snort or Suricata for exactly the reason you just experienced. It will eventually run out of space and then cause very weird issues. With the reliability of today's SSDs, there is really no valid reason for a RAMDISK in my humble opinion.

jasonsansone

Thank you. I will disable the RAM disks going forward.

Giraffe794

I had the exact same reason, and I figured out it was the RAM disks, but I believe that the code that handles the update should be improved.
It is not reliable to have the IPS/IDS stop working because something failed in its rules-update process.

These are the logs I got:

snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file download failed. Bad MD5 checksum...

FATAL ERROR: Failed to load /usr/local/lib/snort_dynamicrules/browser-chrome.so: /usr/local/lib/snort_dynamicrules/browser-chrome.so: invalid file format

snort_check_for_rule_updates.php: [Snort] Rules download error: Failed writing body (0 != 1399)

...plus another one regarding the /tmp being full.
I understand the update failing, but corrupting files and then failing to start is a bit too far from expected functionality.

bmeeks

@Giraffe794 said in Snort Won't Start, Failed to load file-other.so:

I had the exact same reason, and I figured out it was the RAM disks, but I believe that the code that handles the update should be improved.
It is not reliable to have the IPS/IDS stop working because something failed in its rules-update process.

These are the logs I got:
snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file download failed. Bad MD5 checksum...

FATAL ERROR: Failed to load /usr/local/lib/snort_dynamicrules/browser-chrome.so: /usr/local/lib/snort_dynamicrules/browser-chrome.so: invalid file format

snort_check_for_rule_updates.php: [Snort] Rules download error: Failed writing body (0 != 1399)
...plus another one regarding the /tmp being full.
I understand the update failing, but corrupting files and then failing to start is a bit too far from expected functionality.

When you run a RAM disk with the IDS/IPS packages against the recommendations of the package author/maintainer, unexpected failures should be anticipated. I have stated here numerous times over the years DO NOT run a RAM disk with the IDS/IPS packages, or if you do, then it is incumbent upon you to make sure you allocate enough space on /tmp.

pfSense itself will fail in mysterious ways if you run the system disk out of space.

If you have a better way to stucture the update code, then please submit a Pull Request with the code changes to Github here: https://github.com/bmeeks8/FreeBSD-ports/tree/devel/security/pfSense-pkg-snort. The code has no way of knowing what the extracted size of the rules tarball archives will be. It can't know until after extraction, but by then the space is used up and bad things have happened. It needs enough space to hold all the compressed rules tarballs it downloads AND enough room to hold the extracted and decompressed contents while it copies them to the various places in /usr/local/etc/snort where they need to go.

Giraffe794

I have created a pull request:
https://github.com/pfsense/FreeBSD-ports/pull/702

But since I have no idea how to test it, I guess some other person has to do that part.