dpinger stops (crashes?) after update to 2.6.0
-
@reberhar Yeah same here. My pfblocker and surricata updates at around 3AM and after that dpinger stops. That's the regular one I see daily. The other random ones were probably because of my WAN cable issue. I'll check again after fixing the cable. On my SG1100 that is not running surricata obviously I can see the same thing happen on pfblocker updates.
-
That was really quick. It is a stock install and I have not changed anything except for the upgrade to 2.6.0. However I will investigate that problem which is easy to do.
On the one site that is losing the pinging on one head we are using the lan for a gateway as well as a lan. There is only one wire available.
-
@stephenw10 No gateway defined
-
Is it DHCP then? Like running in AWS/Azure maybe?
-
@stephenw10 No not DHCP. It is Saturday and I have already been at this since before 6, but I will research it.
-
@reberhar said in dpinger stops (crashes?) after update to 2.6.0:
@stephenw10 No not DHCP. It is Saturday and I have already been at this since before 6, but I will research it.
I see you are running Suricata. From the log snippet you posted, I'm guessing Suricata is using Inline IPS Mode. If true, that mode uses the netmap kernel device. And each time Suricata stops and restarts, it closes and then reopens a netmap connection for the interface where Inline IPS Mode is used. When netmap closes and opens a connection, it will literally bounce the interface down and back up in FreeBSD the same as if you did the equivalent of an
ifupandifdownin Linux from the command line. Some daemons don't like the physical interface they are running on disappearing out from underneath them.If you suspect this might be your case, you can go to the GLOBAL SETTINGS tab in Suricata and enable the "Live Rule Swap on Update" option. That will tell Suricata to update its internal rules tables without restarting the deamon (and thus also netmap). The only downside of that option is a temporary increase in RAM usage during the rule swap process as two copies of the enabled rules are held in memory until the swap is complete. I probably should change the code to automatically enable the live swap option when Inline IPS is chosen for the blocking mode.
-
@reberhar said in dpinger stops (crashes?) after update to 2.6.0:
Feb 19 00:32:43 dpinger 89107 WAN_DHCP 192.168.0.1: sendto error: 65
Since I set this the "sento error 65" stops: kern.ipc.maxsockbuf Maximum socket buffer size 1000000
-
@bmeeks Hi bmeeks, Actually I saw the post on inline as opposed to legacy mode. I am using legacy mode. However I will try your suggestion. I am using a 2440 box at that location so memory is an issue. So I have adequate swap space however.
Roy
-
@nollipfsense Thanks for reminding me about the buffer change. I had forgotten that. I was over focused on the other problem. I have a 4 head box and had to do that with that install.
-
@reberhar I had the guy on site change the cable for the sendto error. First the simple stuff. I will be watching the site. One of the dpinger offlines stopped with a reinstall of Suricata. Now I am waiting to see if the Live Rule Swap on Update option helps on the other two.
-
@reberhar said in dpinger stops (crashes?) after update to 2.6.0:
@bmeeks Hi bmeeks, Actually I saw the post on inline as opposed to legacy mode. I am using legacy mode. However I will try your suggestion. I am using a 2440 box at that location so memory is an issue. So I have adequate swap space however.
Roy
I saw the physical interface "link state changed" messages, that's why I wondered about Suricata Inline IPS mode. Those messages occur when that mode is used. They should not happen in Legacy Mode (or at least they did not in previous FreeBSD versions).
-
@stephenw10 said in dpinger stops (crashes?) after update to 2.6.0:
Mmm, I would expect dpinger to stop if the WAN loses link and that's the only gateway. But it should start again when you reconnect it. Without needing the watchdog package.
Once successfully started, dpinger itself will not stop/exit for any reason other than a TERM signal. Send and recv errors are logged, but they do not cause the program to exit.
-
Hmm, I wonder what's killing it then...
-
@stephenw10 I tried the Live Reload on 2440 box and that worked even though I am using Legacy mode. I am going to try it on the 4 head system next.
-
@reberhar said in dpinger stops (crashes?) after update to 2.6.0:
@stephenw10 I tried the Live Reload on 2440 box and that worked even though I am using Legacy mode. I am going to try it on the 4 head system next.
That's interesting. The only thing Legacy Mode does with an interface is enable an instance of the PCAP library on it to capture traffic. Well that, and also enabling promiscuous mode. The Suricata binary does this using FreeBSD system calls. In the past, that never resulted in the physical interface bouncing.
-
@stephenw10 To help with this kind of thing in the future, I added logging of the signal number on exit to dpinger. It's pushed as release v3.2. I sent a note to Renato asking him to pull it in when he has time.
-
Thanks for that. Should be in soon.
-
@dennypage Yes thanks for that. That should help.
-
@bmeeks My multihead site responded to changing to the Live Rule Swap option as well. The one that I thought was fixed by a reinstall of Suricata failed again so I have turned on Live Rule Swap there as well. Suricata reloads the rules even if you have blocking turned off and the same problem occurs.
-
@reberhar said in dpinger stops (crashes?) after update to 2.6.0:
@bmeeks My multihead site responded to changing to the Live Rule Swap option as well. The one that I thought was fixed by a reinstall of Suricata failed again so I have turned on Live Rule Swap there as well. Suricata reloads the rules even if you have blocking turned off and the same problem occurs.
In order to monitor traffic, Suricata generally must put the interface in promiscuous mode. That happens whether blocking is enabled or not. And even when run in IDS mode (no blocking), Suricata still must update its rules.