Snort IFs won't start after 2.3.3-RELEASE-p1 upgrade
-
I have read a lot of the other "IF won't start" threads. None of the solutions I've seen have worked for me.
The first reboot after my -p1 upgrade, I noticed none of my Snort interfaces would come up. This happens from time to time, but hadn't happened in months. My troubleshooting is as follows:
- Tried manually starting IFs. Result was a "gateway timeout" (happens sometimes) but no interface status change.
- Tried restarting Snort service, then restarting IFs. Same result.
- Tried stopping Snort. Service won't stay stopped.
- Tried force updating all rules, then restarting IFs. "Gateway timeout" result with no status change.
- Tried a package reinstall. Lots of "Write to restore size failed" messages. Restarted IFs, same result.
- Tried a package uninstall/reinstall. Same "write to" messages. Restarted IFs, same result.
- Unchecked "Keep snort settings after uninstall". Uninstalled/reinstalled. All settings remained but (most) rules flushed. Restarted IFs, same result.
- Unchecked most - then all - rules from interface. Restarted IFs after each, same result.
- Uninstalled, then manually removed all "snort" directories and rules left behind. Reinstalled. Most settings remained. Restarted IFs, same result.
- Re-downloaded rules. Removed one interface. Created a new one with all the same rules. Restarted IF. Same result.
- Removed ALL interfaces. Created a WAN interface. Used minimal rules. Restarted IF. Same result.
I'm literally out of ideas aside from nuking the box and starting over, and I really don't want to go there.
Every failed attempt, this is the log result:
Mar 25 19:08:27 check_reload_status Syncing firewall Mar 25 19:08:27 php-fpm 78297 /snort/snort_rulesets.php: [Snort] Updating rules configuration for: WAN ... Mar 25 19:08:31 php-fpm 78297 /snort/snort_rulesets.php: [Snort] Enabling any flowbit-required rules for: WAN... Mar 25 19:08:31 php-fpm 78297 /snort/snort_rulesets.php: [Snort] Building new sid-msg.map file for WAN... Mar 25 19:08:37 php-fpm 80994 /snort/snort_interfaces.php: [Snort] Updating rules configuration for: WAN ... Mar 25 19:08:41 php-fpm 80994 /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: WAN... Mar 25 19:08:42 php-fpm 80994 /snort/snort_interfaces.php: [Snort] Building new sid-msg.map file for WAN... Mar 25 19:08:42 php-fpm 80994 /snort/snort_interfaces.php: Starting Snort on WAN(igb1) per user request... Mar 25 19:08:42 php-fpm 80994 /snort/snort_interfaces.php: [Snort] Snort START for WAN(igb1)... Mar 25 19:11:37 x.xx.xxx.org nginx: 2017/03/25 19:11:37 [error] 41585#100172: *774 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.x.0.50, server: , request: "POST /snort/snort_interfaces.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.x.0.1", referrer: "https://10.x.0.1/snort/snort_interfaces.php"/var/log/snort is empty (empty app-stats.log and barnyard2 dir), except for the rule update logs, which show nothing odd.
I have had occasional instances in the past where a gateway timeout would occur, but simply refreshing would bring me back to the page with a running IF with a green status icon. The gateway timeout every try with the IF never coming up is new.
pfSense ver is 2.3.3.RELEASE-p1. pfSense snort package ver is 3.2.9.2_16. Snort ver is 2.9.8.3. Barnyard2 ver is 1.13_1.
I have no idea what to try next. Any advice would be greatly appreciated.
Thanks.
-
Did you restart php-fpm (option 16)? Do you have pfblockerng package? If so have you tried disabling it.
-
I have tried Option 16, no difference. Still a gateway timeout, and the interface comes up as stopped.
I used to have pfblockerng installed, but it had been disabled for about a month and a few days ago I uninstalled it prior to the upgrade.
I'm wondering if I have some stale config somewhere… but I can't even find a log entry anywhere that shows where this is choking. All I know is it's choking on every interface, every time. There have been times where the web gateway froze indefinitely, and option 11 wouldn't even recover it. I had to do an option 5 reboot to get back in.
-
Additional steps I've taken…
I found where the settings are retained in this post: https://forum.pfsense.org/index.php?topic=80365.msg438860#msg438860
I uninstalled, removed all settings, and reinstalled. Finally, a fresh install. However still no luck.
I enabled detailed startup logging, and I'm starting to see something. On every boot attempt and on every interface refresh, I'm noticing it dies in the same place - while parsing "file-executable.so". Here's the last couple of lines from the log:
Mar 25 22:26:01 snort 30401 PortVar 'DCERPC_NCACN_IP_LONG' defined : Mar 25 22:26:01 snort 30401 [ 135 139 445 593 1024:65535 ] Mar 25 22:26:01 snort 30401 PortVar 'DCERPC_NCACN_UDP_LONG' defined : Mar 25 22:26:01 snort 30401 [ 135 1024:65535 ] Mar 25 22:26:01 snort 30401 PortVar 'DCERPC_NCACN_UDP_SHORT' defined : Mar 25 22:26:01 snort 30401 [ 135 593 1024:65535 ] Mar 25 22:26:01 snort 30401 PortVar 'DCERPC_NCACN_TCP' defined : Mar 25 22:26:01 snort 30401 [ 2103 2105 2107 ] Mar 25 22:26:01 snort 30401 PortVar 'DCERPC_BRIGHTSTORE' defined : Mar 25 22:26:01 snort 30401 [ 6503:6504 ] Mar 25 22:26:01 snort 30401 PortVar 'DNP3_PORTS' defined : Mar 25 22:26:01 snort 30401 [ 20000 ] Mar 25 22:26:01 snort 30401 PortVar 'MODBUS_PORTS' defined : Mar 25 22:26:01 snort 30401 [ 502 ] Mar 25 22:26:01 snort 30401 PortVar 'GTP_PORTS' defined : Mar 25 22:26:01 snort 30401 [ 2123 2152 3386 ] Mar 25 22:26:01 snort 30401 Detection: Mar 25 22:26:01 snort 30401 Search-Method = AC-BNFA-Q Mar 25 22:26:01 snort 30401 Maximum pattern length = 20 Mar 25 22:26:01 snort 30401 Search-Method-Optimizations = enabled Mar 25 22:26:01 snort 30401 Found pid path directive (/var/run) Mar 25 22:26:01 snort 30401 Tagged Packet Limit: 256 Mar 25 22:26:01 snort 30401 Loading all dynamic engine libs from /usr/local/lib/snort_dynamicengine... Mar 25 22:26:01 snort 30401 Loading dynamic engine /usr/local/lib/snort_dynamicengine/libsf_engine.so... Mar 25 22:26:01 snort 30401 done Mar 25 22:26:01 snort 30401 Finished Loading all dynamic engine libs from /usr/local/lib/snort_dynamicengine Mar 25 22:26:01 snort 30401 Loading all dynamic detection libs from /usr/local/lib/snort_dynamicrules... Mar 25 22:26:01 snort 30401 Loading dynamic detection library /usr/local/lib/snort_dynamicrules/browser-ie.so... Mar 25 22:26:01 snort 30401 done Mar 25 22:26:01 snort 30401 Loading dynamic detection library /usr/local/lib/snort_dynamicrules/file-executable.so...The plot thickens.
I went into the directory in the shell, moved file-executable.so out of the directory, and now my WAN interface comes up. Though I'm sure it will choke on the next rules update.
Thoughts?