Snort IFs won't start after 2.3.3-RELEASE-p1 upgrade



  • I have read a lot of the other "IF won't start" threads.  None of the solutions I've seen have worked for me.

    The first reboot after my -p1 upgrade, I noticed none of my Snort interfaces would come up.  This happens from time to time, but hadn't happened in months.  My troubleshooting is as follows:

    • Tried manually starting IFs.  Result was a "gateway timeout" (happens sometimes) but no interface status change.
    • Tried restarting Snort service, then restarting IFs.  Same result.
    • Tried stopping Snort.  Service won't stay stopped.
    • Tried force updating all rules, then restarting IFs.  "Gateway timeout" result with no status change.
    • Tried a package reinstall.  Lots of "Write to restore size failed" messages.  Restarted IFs, same result.
    • Tried a package uninstall/reinstall.  Same "write to" messages.  Restarted IFs, same result.
    • Unchecked "Keep snort settings after uninstall".  Uninstalled/reinstalled.  All settings remained but (most) rules flushed.  Restarted IFs, same result.
    • Unchecked most - then all - rules from interface.  Restarted IFs after each, same result.
    • Uninstalled, then manually removed all "snort" directories and rules left behind.  Reinstalled.  Most settings remained.  Restarted IFs, same result.
    • Re-downloaded rules.  Removed one interface.  Created a new one with all the same rules.  Restarted IF.  Same result.
    • Removed ALL interfaces.  Created a WAN interface.  Used minimal rules.  Restarted IF.  Same result.

    I'm literally out of ideas aside from nuking the box and starting over, and I really don't want to go there.

    Every failed attempt, this is the log result:

    Mar 25 19:08:27 	check_reload_status 		Syncing firewall
    Mar 25 19:08:27 	php-fpm 	78297 	/snort/snort_rulesets.php: [Snort] Updating rules configuration for: WAN ...
    Mar 25 19:08:31 	php-fpm 	78297 	/snort/snort_rulesets.php: [Snort] Enabling any flowbit-required rules for: WAN...
    Mar 25 19:08:31 	php-fpm 	78297 	/snort/snort_rulesets.php: [Snort] Building new sid-msg.map file for WAN...
    Mar 25 19:08:37 	php-fpm 	80994 	/snort/snort_interfaces.php: [Snort] Updating rules configuration for: WAN ...
    Mar 25 19:08:41 	php-fpm 	80994 	/snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: WAN...
    Mar 25 19:08:42 	php-fpm 	80994 	/snort/snort_interfaces.php: [Snort] Building new sid-msg.map file for WAN...
    Mar 25 19:08:42 	php-fpm 	80994 	/snort/snort_interfaces.php: Starting Snort on WAN(igb1) per user request...
    Mar 25 19:08:42 	php-fpm 	80994 	/snort/snort_interfaces.php: [Snort] Snort START for WAN(igb1)...
    Mar 25 19:11:37 	x.xx.xxx.org 		nginx: 2017/03/25 19:11:37 [error] 41585#100172: *774 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.x.0.50, server: , request: "POST /snort/snort_interfaces.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.x.0.1", referrer: "https://10.x.0.1/snort/snort_interfaces.php" 
    

    /var/log/snort is empty (empty app-stats.log and barnyard2 dir), except for the rule update logs, which show nothing odd.

    I have had occasional instances in the past where a gateway timeout would occur, but simply refreshing would bring me back to the page with a running IF with a green status icon.  The gateway timeout every try with the IF never coming up is new.

    pfSense ver is 2.3.3.RELEASE-p1. pfSense snort package ver is 3.2.9.2_16. Snort ver is 2.9.8.3. Barnyard2 ver is 1.13_1.

    I have no idea what to try next.  Any advice would be greatly appreciated.

    Thanks.



  • Did you restart php-fpm (option 16)? Do you have pfblockerng package? If so have you tried disabling it.



  • I have tried Option 16, no difference.  Still a gateway timeout, and the interface comes up as stopped.

    I used to have pfblockerng installed, but it had been disabled for about a month and a few days ago I uninstalled it prior to the upgrade.

    I'm wondering if I have some stale config somewhere… but I can't even find a log entry anywhere that shows where this is choking.  All I know is it's choking on every interface, every time.  There have been times where the web gateway froze indefinitely, and option 11 wouldn't even recover it.  I had to do an option 5 reboot to get back in.



  • Additional steps I've taken…

    I found where the settings are retained in this post: https://forum.pfsense.org/index.php?topic=80365.msg438860#msg438860

    I uninstalled, removed all settings, and reinstalled.  Finally, a fresh install.  However still no luck.

    I enabled detailed startup logging, and I'm starting to see something.  On every boot attempt and on every interface refresh, I'm noticing it dies in the same place - while parsing "file-executable.so".  Here's the last couple of lines from the log:

    Mar 25 22:26:01 	snort 	30401 	PortVar 'DCERPC_NCACN_IP_LONG' defined :
    Mar 25 22:26:01 	snort 	30401 	[ 135 139 445 593 1024:65535 ]
    Mar 25 22:26:01 	snort 	30401 	PortVar 'DCERPC_NCACN_UDP_LONG' defined :
    Mar 25 22:26:01 	snort 	30401 	[ 135 1024:65535 ]
    Mar 25 22:26:01 	snort 	30401 	PortVar 'DCERPC_NCACN_UDP_SHORT' defined :
    Mar 25 22:26:01 	snort 	30401 	[ 135 593 1024:65535 ]
    Mar 25 22:26:01 	snort 	30401 	PortVar 'DCERPC_NCACN_TCP' defined :
    Mar 25 22:26:01 	snort 	30401 	[ 2103 2105 2107 ]
    Mar 25 22:26:01 	snort 	30401 	PortVar 'DCERPC_BRIGHTSTORE' defined :
    Mar 25 22:26:01 	snort 	30401 	[ 6503:6504 ]
    Mar 25 22:26:01 	snort 	30401 	PortVar 'DNP3_PORTS' defined :
    Mar 25 22:26:01 	snort 	30401 	[ 20000 ]
    Mar 25 22:26:01 	snort 	30401 	PortVar 'MODBUS_PORTS' defined :
    Mar 25 22:26:01 	snort 	30401 	[ 502 ]
    Mar 25 22:26:01 	snort 	30401 	PortVar 'GTP_PORTS' defined :
    Mar 25 22:26:01 	snort 	30401 	[ 2123 2152 3386 ]
    Mar 25 22:26:01 	snort 	30401 	Detection:
    Mar 25 22:26:01 	snort 	30401 	Search-Method = AC-BNFA-Q
    Mar 25 22:26:01 	snort 	30401 	Maximum pattern length = 20
    Mar 25 22:26:01 	snort 	30401 	Search-Method-Optimizations = enabled
    Mar 25 22:26:01 	snort 	30401 	Found pid path directive (/var/run)
    Mar 25 22:26:01 	snort 	30401 	Tagged Packet Limit: 256
    Mar 25 22:26:01 	snort 	30401 	Loading all dynamic engine libs from /usr/local/lib/snort_dynamicengine...
    Mar 25 22:26:01 	snort 	30401 	Loading dynamic engine /usr/local/lib/snort_dynamicengine/libsf_engine.so...
    Mar 25 22:26:01 	snort 	30401 	done
    Mar 25 22:26:01 	snort 	30401 	Finished Loading all dynamic engine libs from /usr/local/lib/snort_dynamicengine
    Mar 25 22:26:01 	snort 	30401 	Loading all dynamic detection libs from /usr/local/lib/snort_dynamicrules...
    Mar 25 22:26:01 	snort 	30401 	Loading dynamic detection library /usr/local/lib/snort_dynamicrules/browser-ie.so...
    Mar 25 22:26:01 	snort 	30401 	done
    Mar 25 22:26:01 	snort 	30401 	Loading dynamic detection library /usr/local/lib/snort_dynamicrules/file-executable.so... 
    

    The plot thickens.

    I went into the directory in the shell, moved file-executable.so out of the directory, and now my WAN interface comes up.  Though I'm sure it will choke on the next rules update.

    Thoughts?