Suricata with custom memory_limit crashing on upgrade
I attempted an upgrade from Suricata 6.0.8 to 6.0.10 today. I had to use a larger memory limit (1024M from 512M) in the /usr/local/pkg/suricata/suricata.inc as described in:
The problem is, when the new suricata package is installed, the new suricata.inc is going back to the default 512M, which isn't large enough to rebuild the yaml files. I've tried to run a sed as soon as the suricata.inc is written, but it isn't working.
I keep getting this error:
Cleaning up after rules extraction... done. The Rules update has finished. Generating suricata.yaml configuration file from saved settings. Generating YAML configuration file for 00_WAN... done. Generating YAML configuration file for 05_LAN... Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 51232048 bytes) in /usr/local/pkg/suricata/suricata.inc on line 1066 PHP ERROR: Type: 1, File: /usr/local/pkg/suricata/suricata.inc, Line: 1066, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 51232048 bytes)pkg-static: POST-INSTALL script failed >>> Cleaning up cache... done.
Are there any other options for me to get Suricata running again? The system is crashing before getting to where I can get in to the Suricata GUI.
I really wish the memory_limit was a tunable feature that wouldn't be overwritten.
Ok, well I think I found a solution. I was able to manually uninstall suricata, then remove but save the <suricata></suricata> XML from the config.xml file.
When I reinstalled suricata with the default settings, I then changed the memory from 512 back to 1024, and replaced the generated <suricata></suricata> XML from the config file with the original settings. I wasn't able to start suricata entirely, but it let me start each interface one at a time.
It looks like Suricata is back on track, so I'll monitor for anything out of the ordinary.
While altering the allocated PHP memory is a workaround, you should address the root cause of the problem instead.
Sounds like you need to rethink what you are doing in your SID MGMT settings. Exhausting the PHP memory should not be happening there unless something really strange is going on with what you have configured there.
How about posting your SID MGMT conf files and let me have a look?
@bmeeks Thanks for the input. I like having the additional rules, which don't seem to affect Suricata at all other than initially building the yaml files, but I know you are correct in finding a better way to optimize my rules. I'll have to go through and audit the ruleset vs what I have in the SID management. I basically have a SID drop file and SID disable, with default enable on the rulesets. Here are the SID drop/disable rules for WAN.
I have similar SID files for 5 other interfaces that I manage/monitor with Suricata separately. I'm sure if I go through and find what rules I need on WAN, then reduce the rules even more for what I have on the internal interfaces, it would be a lot better. I won't need some of the WAN rules on internal interfaces, so only would then need rules appropriate for local interface to local interface, since the WAN rule will get anything in/outbound, if I'm thinking about it correctly. So I'll focus on that first.
I'm not sure really where the memory usage is the highest when building the yaml files. Does the bulk of the memory usage come from loading the various rules, and then more memory is used when adding in the SID management files? If I know how the memory is consumed, I can try and optimize focusing on that, too.
Here is the WAN disable SID:
# WAN Disable Ruleset # ET Pro Rules etpro-chat,etpro-dns,etpro-games,etpro-icmp,etpro-icmp_info,etpro-inappropriate,etpro-info,etpro-p2p etpro-policy,etpro-tor # SNORT Rules snort_app-detect,snort_chat,snort_content-replace,snort_dns,snort_icmp-info,snort_icmp,snort_info snort_multimedia,snort_p2p,snort_policy,snort_x11 # Individual Disabled Ruleset # FIOS Guide 1:2840787 # ETPRO HUNTING Request for config.json # Suricata Stream 1:2210008 # SURICATA STREAM 3way handshake SYN resend different seq on SYN recv 1:2210016 # SURICATA STREAM CLOSEWAIT FIN out of window 1:2210029 # SURICATA STREAM ESTABLISHED invalid ack 1:2210038 # SURICATA STREAM FIN out of window 1:2210042 # SURICATA STREAM TIMEWAIT ACK with wrong seq 1:2210044 # SURICATA STREAM Packet with invalid timestamp 1:2210045 # SURICATA STREAM Packet with invalid ack 1:2210050 # SURICATA STREAM reassembly overlap with different data 1:2210054 # SURICATA STREAM excessive retransmissions # Breaks webpage/NEST 1:2221010 # SURICATA HTTP unable to match response to request # Breaks NEST 1:2018383 # ET EXPLOIT Possible OpenSSL HeartBleed Large HeartBeat Response from Common SSL Port (Outbound from Client) # Generated from LTE_Extender 1:2221045 # SURICATA HTTP Unexpected Request body # Weatherflow 1:2229001 # SURICATA MQTT PUBLISH not seen before PUBACK/PUBREL/PUBREC/PUBCOMP # Noisy / Misc 1:2023883 # ET DNS Query to a *.top domain - Likely Hostile 1:2027390 # ET USER_AGENTS Microsoft Device Metadata Retrieval Client User-Agent 1:2027757 # ET DNS Query for .to TLD 1:2027758 # ET DNS Query for .cc TLD 1:2200036 # SURICATA TCP option invalid length 1:2200070 # SURICATA FRAG IPv4 Fragmentation overlap 1:2200073 # SURICATA IPv4 invalid checksum 1:2200075 # SURICATA UDPv4 invalid checksum 1:2200076 # SURICATA ICMPv4 invalid checksum 1:2210002 # SURICATA STREAM 3way handshake right seq wrong ack evasion 1:2210004 # SURICATA STREAM 3way handshake SYNACK resend with different ack 1:2210010 # SURICATA STREAM 3way handshake wrong seq wrong ack 1:2210015 # SURICATA STREAM CLOSEWAIT ACK out of window 1:2210020 # SURICATA STREAM ESTABLISHED packet out of window 1:2210023 # SURICATA STREAM ESTABLISHED SYNACK resend with different ACK 1:2210024 # SURICATA STREAM ESTABLISHED SYNACK resend with different seq 1:2210026 # SURICATA STREAM ESTABLISHED SYN resend 1:2210030 # SURICATA STREAM FIN invalid ack 1:2210035 # SURICATA STREAM FIN2 FIN with wrong seq 1:2210036 # SURICATA STREAM FIN2 invalid ack 1:2210046 # SURICATA STREAM SHUTDOWN RST invalid ack 1:2221014 # SURICATA HTTP missing Host header 1:2221017 # SURICATA HTTP invalid response field folding 1:2221021 # SURICATA HTTP response header invalid 1:2224003 # SURICATA IKEv2 weak cryptographic parameters (PRF) 1:2224004 # SURICATA IKEv2 weak cryptographic parameters (Auth) 1:2224005 # SURICATA IKEv2 weak cryptographic parameters (Diffie-Hellman) 1:2229002 # SURICATA MQTT SUBSCRIBE not seen before SUBACK 1:2229005 # SURICATA MQTT message seen before CONNECT/CONNACK completion 1:2230003 # SURICATA TLS invalid handshake message 1:2230010 # SURICATA TLS invalid record/traffic 1:2260000 # SURICATA Applayer Mismatch protocol both directions 1:2260002 # SURICATA Applayer Detect protocol only one direction
Here is the WAN Drop Sid:
# WAN Drop Ruleset # Snort GPLv2 Community Rules Drop GPLv2_community # Feodo Tracker Botnet C2 Rules feodotracker # Abuse.ch SSL Blacklist Rules sslblacklist_tls_cert # ET Pro Rules etpro-activex,etpro-adware_pup,etpro-attack_response,etpro-botcc,etpro-ciarmy,etpro-coinminer etpro-compromised,etpro-current_events,etpro-dos,etpro-drop,etpro-dshield etpro-exploit,etpro-exploit_kit,etpro-ftp,etpro-hunting,etpro-imap,etpro-ja3,etpro-malware,etpro-misc etpro-mobile_malware,etpro-netbios,etpro-phishing,etpro-pop3,etpro-rpc etpro-scan,etpro-shellcode,etpro-smtp,etpro-sql,etpro-telnet,etpro-tftp,etpro-threatview_CS_c2 etpro-trojan,etpro-user_agents,etpro-web,etpro-worm # Snort Ruleset snort_attack-response,snort_backdoor,snort_bad-traffic,snort_blacklist,snort_botnet-cnc,snort_browser snort_ddos,snort_deleted,snort_dos,snort_experimental,snort_exploit-kit,snort_file,snort_finger snort_ftp,snort_indicator,snort_local,snort_malware,snort_misc,snort_mysql,snort_netbios,snort_os snort_other-ida,snort_phishing-spam,snort_pop,snort_protocol,snort_pua,snort_rpc,snort_rservices snort_scan,snort_server,snort_shellcode,snort_smtp,snort_snmp,snort_specific-threats snort_spyware-put,snort_sql,snort_telnet,snort_tftp,snort_virus,snort_voip,snort_web # Individual Ruleset 1:2210008 # SURICATA STREAM 3way handshake SYNACK resend with different ack 1:2210026 # SURICATA STREAM ESTABLISHED SYN resend