Suricata crashing during Windows Server backup to backuppc by SMB
-
We make our backup to a backuppc server outside our infrastructure. Since a few months we have a out of memory problem with suricata when a full backup of one of our windows server is made. This occurs only for the full backups. The suricata memory usage starts increasing until the service crashed with an out of memory.
I already tried to put both windows server and backuppc in whitelist, passlist, ... without success. I just want that the trafic between these two servers won't be treated by suricata but I'm out of ideas to configure it.
Could someone help me ?
Some infos :
pfsense 2.4.4-RELEASE-p3 (amd64)
suricata 4.1.5_1 used in legacy ModeThanks a lot for your help
-
The only thing I can think of maybe tried using Suricata's inline mode if your NIC supports. It appears that you might have lots of traffic to that Windows server to use Legacy mode that makes a double copy of every traffic.
-
I'm not sure that the inline mode will work and I cannot test it as it is a production environment.
I'm quite sure that there are very few traffic on the windows server. The problem happens only during the full backup to backuppc by using the SMB transfer method. The firewall has 8Gb of memory and doesn't have any problem with 30 webservers. It only crash during these kinds of backups. No problem with linux backups using rsync method.
-
@pitouyak Lets wait for Bill to respond and guide you correctly.
-
@NollipfSense ok thanks for your help.
-
There was at least one bug reported on the Suricata Redmine site about SMB inspection. I think it was specifically reported for Suricata 5.0.0, but that does not mean the bug could also be in Suricata 4.x code.
Using a Pass List will not help in this case because all that does is prevent blocks. It does not prevent traffic from being inspected by the Suricata engine. You will need to create a custom PASS rule that contains the IP addresses of the two servers. A PASS rule will allow matching traffic to bypass the Suricata inspection engine. Hopefully that PASS rule check happens in the binary code before the SMB inspector is called.
-
@bmeeks It should be a regression because I didn't had these problems before. But it took me a long time to discover the source so I don't even know when it started and with which update.
I'm going to try a custom pass rule. Thanks for your suggestion.
-
Could you confirm that this rule should make the job ?
pass tcp windows_server_ip [139,445] <> backup_server_ip [139,445] (msg:"Backuppc through SAMBA"; sid:100000; rev:1; )
Thanks a lot. I habe been awake tonight again and it will be so good to "solve" this problem.
-
@pitouyak said in Suricata crashing during Windows Server backup to backuppc by SMB:
Could you confirm that this rule should make the job ?
pass tcp windows_server_ip [139,445] <> backup_server_ip [139,445] (msg:"Backuppc through SAMBA"; sid:100000; rev:1; )
Thanks a lot. I habe been awake tonight again and it will be so good to "solve" this problem.
I haven't looked at logged SMB traffic in a long, long time. That rule should work, but you might find you need to change the SOURCE PORT to "any". I can't recall if SMB and CIFS uses an ephemeral port on the initiating end or not. If so, then you would need two PASS rules, each with one of the host IP addresses and "any" as the SOURCE. Something like this:
pass tcp windows_server_ip any -> backup_server_ip [139,445] (msg:"Backuppc through SAMBA"; sid:100000; rev:1; ) pass tcp backup_server_ip any -> windows_server_ip [139,445] (msg:"Backuppc through SAMBA"; sid:100000; rev:1; )You can tell by turning on logging in the firewall for that traffic (or capturing some of the traffic using the option under the DIAGNOSTICS menu) and seeing what is being used for source ports.
-
@bmeeks Thanks you for your time and your help. I'm going to try this.
Do I have to do something to submit this as a bug or the one on suricata 5 is enough ?
-
@pitouyak said in Suricata crashing during Windows Server backup to backuppc by SMB:
@bmeeks Thanks you for your time and your help. I'm going to try this.
Do I have to do something to submit this as a bug or the one on suricata 5 is enough ?
They already have a bug on the Redmine site. If the PASS rule fixes it for you, you can certainly comment on the Redmine issue thread that it impacts version 4.x as well. Test the PASS rule and lets see if that helps first.
Upon further research at the Suricata Redmine site, I found this bug from 2018 which is reported as fixed, but maybe not all situations where the bug could surface were fixed ???
https://redmine.openinfosecfoundation.org/issues/2515
That version 5.0.0 bug is causing a Suricata crash. It's not a memory leak which is what your error appears to be.
-
Hello
No more crash this weekend. I have launched a manual backup on friday to test and the memory usage hasn't increased. I will still wait for a week but I think the solution is good.
Thanks a lot to @bmeeks for the help.
