Still seeing suricata stop an interface due to .pid error
-
@boobletins said in Still seeing suricata stop an interface due to .pid error:
grep -v SC_ERR_UNDEFINED_VAR |
grep -v SC_ERR_EVENT_ENGINE |All your listed errors except these two are due to running Snort rules with Suricata. I would like to have the entire line of error text from the log for these two errors:
grep -v SC_ERR_UNDEFINED_VAR | grep -v SC_ERR_EVENT_ENGINE |While I don't think it's going to magically fix your problem, the lastest Snort rules snapshot is snortrules-snapshot-29120.tar.gz. That's the file I would suggest running.
I don't think this is a widespread issue on pfSense. So far you and the OP are the only reports I have out of quite a few Suricata users on pfSense. If you are also seeing the "stale PID error", that indicates Suricata is crashing (likely either a Signal 11 or Signal 10) and thus not cleaning up after itself. During an orderly shutdown, Suricata itself deletes that PID file it is complaining about. Upon startup, it expects that file to be gone as the previousj orderly shutdown should have resulted in it being deleted. If Suricata crashes and is terminated by the OS, then the file is not deleted and thus the error is thrown at the next startup attempt.
Search your pfSense system log to see if there are any Signal 11 or Signal 10 errors reported for Suricata. Also let me know what kind of WAN interface you have. Is it PPPoE or a standard DHCP or static IP mapping?
-
Thanks for the reply!
The undefined var errors are uniformly:
SC_ERR_UNDEFINED_VAR(101)] - Variable "SIP_PORTS" is not defined in configuration file
The event engine error does not appear on the LAN interface and is uniformly:
[ERRCODE: SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2200075, gid 1: unknown ruleBoth interfaces are provisioned using DHCP.
I assume the errors above could safely be ignored?
system.log is maybe more interesting--it contains the following signal 11 notifications:
Nov 18 05:47:56 XXXX kernel: pid 72925 (suricata), uid 0: exited on signal 11 (core dumped)
Nov 18 05:50:21 XXXX kernel: pid 31920 (suricata), uid 0: exited on signal 11 (core dumped)Here's the nearby log with some redaction:
Nov 18 05:47:17 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:47:20 XXXX suricata[72925]: [Drop] [1:2402000:5002] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 198.108.67.36:35236 -> X.X.X.X:9955 Nov 18 05:47:32 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:47:38 XXXX rc.gateway_alarm[21524]: >>> Gateway alarm: WAN_DHCP6 (Addr:IPV6-1%em0 Alarm:1 RTT:132.089ms RTTsd:8.936ms Loss:22%) Nov 18 05:47:38 XXXX check_reload_status: updating dyndns WAN_DHCP6 Nov 18 05:47:38 XXXX check_reload_status: Restarting ipsec tunnels Nov 18 05:47:38 XXXX check_reload_status: Restarting OpenVPN tunnels/interfaces Nov 18 05:47:38 XXXX check_reload_status: Reloading filter Nov 18 05:47:38 XXXX rc.gateway_alarm[23488]: >>> Gateway alarm: WAN_DHCP (Addr:X.X.X.1 Alarm:1 RTT:132.204ms RTTsd:9.258ms Loss:22%) Nov 18 05:47:38 XXXX check_reload_status: updating dyndns WAN_DHCP Nov 18 05:47:38 XXXX check_reload_status: Restarting ipsec tunnels Nov 18 05:47:38 XXXX check_reload_status: Restarting OpenVPN tunnels/interfaces Nov 18 05:47:38 XXXX check_reload_status: Reloading filter Nov 18 05:47:39 XXXX php-fpm[33153]: /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP' Nov 18 05:47:39 XXXX php-fpm[33153]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Nov 18 05:47:39 XXXX php-fpm[37754]: /rc.dyndns.update: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:47:39 XXXX php-fpm[88455]: /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP' Nov 18 05:47:39 XXXX php-fpm[88455]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Nov 18 05:47:39 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:47:39 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:47:39 XXXX xinetd[88961]: Swapping defaults Nov 18 05:47:39 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:47:39 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:47:40 XXXX php-fpm[37754]: /rc.dyndns.update: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:47:40 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:47:40 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:47:40 XXXX xinetd[88961]: Swapping defaults Nov 18 05:47:40 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:47:40 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:47:50 XXXX kernel: 670.657772 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 99 m 0xfffff80128ee6a00 Nov 18 05:47:50 XXXX kernel: 670.697930 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 107 m 0xfffff802164f6900 Nov 18 05:47:50 XXXX kernel: 670.712976 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 108 m 0xfffff802b8d49d00 Nov 18 05:47:50 XXXX kernel: 670.733007 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 85 m 0xfffff802164f6900 Nov 18 05:47:50 XXXX kernel: 670.738608 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802164f6900 Nov 18 05:47:50 XXXX kernel: 670.738924 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802b8d49d00 Nov 18 05:47:50 XXXX kernel: 670.738961 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 98 m 0xfffff80128ee6a00 Nov 18 05:47:50 XXXX kernel: 670.739752 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 98 m 0xfffff802b8d49d00 Nov 18 05:47:50 XXXX kernel: 670.740041 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802b8d49d00 Nov 18 05:47:50 XXXX kernel: 670.741516 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 86 m 0xfffff802164f6900 Nov 18 05:47:51 XXXX kernel: 671.032906 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 42 m 0xfffff801a05e6d00 Nov 18 05:47:51 XXXX kernel: 671.038964 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 107 m 0xfffff8000d278b00 Nov 18 05:47:51 XXXX kernel: 671.043875 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 95 m 0xfffff801a05e6700 Nov 18 05:47:51 XXXX kernel: 671.095031 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 108 m 0xfffff8000d881900 Nov 18 05:47:51 XXXX kernel: 671.125010 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 85 m 0xfffff801a05e6d00 Nov 18 05:47:51 XXXX kernel: 671.125037 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 86 m 0xfffff801a05e6d00 Nov 18 05:47:51 XXXX kernel: 671.125073 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 111 m 0xfffff801a05e6d00 Nov 18 05:47:51 XXXX kernel: 671.140838 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 107 m 0xfffff801a05e6700 Nov 18 05:47:51 XXXX kernel: 671.155689 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 87 m 0xfffff8000d278b00 Nov 18 05:47:51 XXXX kernel: 671.158230 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 107 m 0xfffff8000d881900 Nov 18 05:47:52 XXXX kernel: 672.036022 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 96 m 0xfffff802b8c36200 Nov 18 05:47:52 XXXX kernel: 672.043806 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff801a05e6700 Nov 18 05:47:52 XXXX kernel: 672.045644 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 42 m 0xfffff802b8c36200 Nov 18 05:47:52 XXXX kernel: 672.075390 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff8000d978700 Nov 18 05:47:52 XXXX kernel: 672.075414 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 98 m 0xfffff8000d8e5200 Nov 18 05:47:52 XXXX kernel: 672.075432 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 82 m 0xfffff80129007300 Nov 18 05:47:52 XXXX kernel: 672.075451 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff801a0473700 Nov 18 05:47:52 XXXX kernel: 672.075895 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff8000d8e5200 Nov 18 05:47:52 XXXX kernel: 672.080115 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 119 m 0xfffff8000d978700 Nov 18 05:47:52 XXXX kernel: 672.093778 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff8000d978700 Nov 18 05:47:53 XXXX kernel: 673.012356 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 111 m 0xfffff802b8b2b200 Nov 18 05:47:53 XXXX kernel: 673.034875 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 87 m 0xfffff801a098b300 Nov 18 05:47:53 XXXX kernel: 673.039092 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 87 m 0xfffff801a098b300 Nov 18 05:47:53 XXXX kernel: 673.050711 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 42 m 0xfffff8027f059600 Nov 18 05:47:53 XXXX kernel: 673.065905 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802b8984200 Nov 18 05:47:53 XXXX kernel: 673.065925 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff8000d86f700 Nov 18 05:47:53 XXXX kernel: 673.065941 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802b8984200 Nov 18 05:47:53 XXXX kernel: 673.065958 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff801a098b300 Nov 18 05:47:53 XXXX kernel: 673.070275 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 105 m 0xfffff8027f059600 Nov 18 05:47:53 XXXX kernel: 673.080830 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff8027f059600 Nov 18 05:47:54 XXXX kernel: 674.013828 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 98 m 0xfffff802b8af4b00 Nov 18 05:47:54 XXXX kernel: 674.015016 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 102 m 0xfffff80128921800 Nov 18 05:47:54 XXXX kernel: 674.032843 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 118 m 0xfffff802b8af4b00 Nov 18 05:47:54 XXXX kernel: 674.056702 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 42 m 0xfffff801a098b300 Nov 18 05:47:54 XXXX kernel: 674.062713 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 74 m 0xfffff8000d863c00 Nov 18 05:47:54 XXXX kernel: 674.068904 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff80128dbf800 Nov 18 05:47:54 XXXX kernel: 674.107820 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 88 m 0xfffff802b8af4b00 Nov 18 05:47:54 XXXX kernel: 674.112807 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 95 m 0xfffff802b8af4b00 Nov 18 05:47:54 XXXX kernel: 674.128842 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 118 m 0xfffff802b8af4b00 Nov 18 05:47:54 XXXX kernel: 674.130808 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 115 m 0xfffff802b8af4b00 Nov 18 05:47:55 XXXX kernel: 675.056754 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff802334fac00 Nov 18 05:47:55 XXXX kernel: 675.058729 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 42 m 0xfffff802334fac00 Nov 18 05:47:55 XXXX kernel: 675.059914 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802334fac00 Nov 18 05:47:55 XXXX kernel: 675.059955 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff802334fac00 Nov 18 05:47:55 XXXX kernel: 675.061751 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 98 m 0xfffff80128e56100 Nov 18 05:47:55 XXXX kernel: 675.067740 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff80128e56100 Nov 18 05:47:55 XXXX kernel: 675.074865 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 82 m 0xfffff801cb0f9300 Nov 18 05:47:55 XXXX kernel: 675.082237 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 106 m 0xfffff80262178d00 Nov 18 05:47:55 XXXX kernel: 675.082756 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 86 m 0xfffff80262178d00 Nov 18 05:47:55 XXXX kernel: 675.099097 [2925] netmap_transmit em0 full hwcur 84 hwtail 84 qlen 1023 len 78 m 0xfffff8000d84ce00 Nov 18 05:47:56 XXXX kernel: pid 72925 (suricata), uid 0: exited on signal 11 (core dumped) Nov 18 05:47:56 XXXX kernel: em0: link state changed to DOWN Nov 18 05:47:56 XXXX check_reload_status: Linkup starting em0 Nov 18 05:47:57 XXXX php-fpm[60994]: /rc.linkup: DEVD Ethernet detached event for wan Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:58 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:59 XXXX kernel: em0: link state changed to UP Nov 18 05:47:59 XXXX check_reload_status: Linkup starting em0 Nov 18 05:47:59 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:59 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:59 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:59 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:47:59 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:48:00 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:48:00 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:48:00 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:48:00 XXXX kernel: cannot forward src fe80:1::3af7:3dff:fe40:9e6b, dst 2600:6c40:500:8ac:230:18ff:fece:19d0, nxt 58, rcvif igb0, outif em0 Nov 18 05:48:01 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:48:01 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:48:01 XXXX kernel: arpresolve: can't allocate llinfo for X.X.X.1 on em0 Nov 18 05:48:02 XXXX php-fpm[60994]: /rc.linkup: Shutting down Router Advertisment daemon cleanly Nov 18 05:48:02 XXXX check_reload_status: Reloading filter Nov 18 05:48:02 XXXX php-fpm[33153]: /rc.linkup: DEVD Ethernet attached event for wan Nov 18 05:48:02 XXXX php-fpm[33153]: /rc.linkup: HOTPLUG: Configuring interface wan Nov 18 05:48:02 XXXX check_reload_status: rc.newwanip starting em0 Nov 18 05:48:02 XXXX php-fpm[33153]: /rc.linkup: calling interface_dhcpv6_configure. Nov 18 05:48:02 XXXX php-fpm[33153]: /rc.linkup: Accept router advertisements on interface em0 Nov 18 05:48:02 XXXX php-fpm[33153]: /rc.linkup: Starting rtsold process Nov 18 05:48:03 XXXX php-fpm[27412]: /rc.newwanip: rc.newwanip: Info: starting on em0. Nov 18 05:48:03 XXXX php-fpm[27412]: /rc.newwanip: rc.newwanip: on (IP address: X.X.X.X) (interface: WAN[wan]) (real interface: em0). Nov 18 05:48:03 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:48:03 XXXX xinetd[88961]: Swapping defaults Nov 18 05:48:03 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:48:03 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:48:04 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} 192.168.1.90:34325 -> 172.217.4.227:80 Nov 18 05:48:04 XXXX php-fpm[33153]: /rc.linkup: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP' Nov 18 05:48:04 XXXX php-fpm[33153]: /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6' Nov 18 05:48:04 XXXX check_reload_status: Restarting ipsec tunnels Nov 18 05:48:04 XXXX rtsold: Received RA specifying route IPV6-1 for interface wan(em0) Nov 18 05:48:04 XXXX rtsold: Starting dhcp6 client for interface wan(em0) Nov 18 05:48:04 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:48:04 XXXX xinetd[88961]: Swapping defaults Nov 18 05:48:04 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:48:04 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:48:05 XXXX php-fpm[27412]: /rc.newwanip: Removing static route for monitor IPV6-1 and adding a new route through IPV6-1%em0 Nov 18 05:48:06 XXXX php-fpm[13145]: /rc.newwanipv6: rc.newwanipv6: Info: starting on em0. Nov 18 05:48:06 XXXX php-fpm[13145]: /rc.newwanipv6: rc.newwanipv6: on (IP address: IPV6.2) (interface: wan) (real interface: em0). Nov 18 05:48:07 XXXX php-fpm[13145]: /rc.newwanipv6: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1542541687] unbound[798:0] error: bind: address already in use [1542541687] unbound[798:0] fatal error: could not open ports' Nov 18 05:48:07 XXXX php-fpm[33153]: /rc.linkup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1542541687] unbound[16614:0] error: bind: address already in use [1542541687] unbound[16614:0] fatal error: could not open ports' Nov 18 05:48:09 XXXX php-fpm[37754]: /rc.dyndns.update: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'couldn't get address for 'update.dyndns.com': failure syntax error' Nov 18 05:48:09 XXXX php-fpm[37754]: /rc.dyndns.update: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X) Nov 18 05:48:09 XXXX php-fpm[97376]: /rc.dyndns.update: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'couldn't get address for 'update.dyndns.com': failure syntax error' Nov 18 05:48:09 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X) Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'dns_request_getresponse: expected a TSIG or SIG(0)' Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X) Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:48:10 XXXX php-fpm[13145]: /rc.newwanipv6: Removing static route for monitor IPV6-1 and adding a new route through IPV6-1%em0 Nov 18 05:48:11 XXXX check_reload_status: Reloading filter Nov 18 05:48:11 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:48:12 XXXX php-fpm[27412]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN. Nov 18 05:48:12 XXXX php-fpm[27412]: /rc.newwanip: Creating rrd update script Nov 18 05:48:12 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:48:12 XXXX xinetd[88961]: Swapping defaults Nov 18 05:48:12 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:48:12 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:48:12 XXXX check_reload_status: updating dyndns wan Nov 18 05:48:12 XXXX check_reload_status: Reloading filter Nov 18 05:48:14 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:48:14 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:48:14 XXXX xinetd[88961]: Swapping defaults Nov 18 05:48:14 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:48:14 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:48:14 XXXX php-fpm[27412]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - X.X.X.X -> X.X.X.X - Restarting packages. Nov 18 05:48:14 XXXX check_reload_status: Starting packages Nov 18 05:48:15 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:48:15 XXXX php-fpm[88455]: /rc.start_packages: Restarting/Starting all packages. Nov 18 05:48:15 XXXX php-fpm[88455]: [pfBlockerNG] Starting cron process. Nov 18 05:48:15 XXXX SuricataStartup[3518]: Suricata START for WAN(34205_em0)... Nov 18 05:48:15 XXXX check_reload_status: Syncing firewall Nov 18 05:48:15 XXXX check_reload_status: Reloading filter Nov 18 05:48:16 XXXX php-fpm[97376]: /rc.dyndns.update: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'dns_request_getresponse: expected a TSIG or SIG(0)' Nov 18 05:48:16 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X) Nov 18 05:48:17 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:48:17 XXXX xinetd[88961]: Swapping defaults Nov 18 05:48:17 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:48:17 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:48:17 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] - squid_resync function call pr:1 bp: rpc:no Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Adding cronjobs ... Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Antivirus features disabled. Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Removing freshclam cronjob. Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Stopping any running proxy monitors Nov 18 05:48:23 XXXX php-fpm[88455]: /rc.start_packages: [squid] Reloading for configuration sync... Nov 18 05:48:23 XXXX php-fpm[88455]: /rc.start_packages: [squid] Starting a proxy monitor script Nov 18 05:48:24 XXXX check_reload_status: Reloading filter Nov 18 05:48:09 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X) Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'dns_request_getresponse: expected a TSIG or SIG(0)' Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X) Nov 18 05:48:10 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:48:10 XXXX php-fpm[13145]: /rc.newwanipv6: Removing static route for monitor IPV6-1 and adding a new route through IPV6-1%em0 Nov 18 05:48:11 XXXX check_reload_status: Reloading filter Nov 18 05:48:11 XXXX php-fpm[27412]: /rc.newwanip: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:48:12 XXXX php-fpm[27412]: /rc.newwanip: Resyncing OpenVPN instances for interface WAN. Nov 18 05:48:12 XXXX php-fpm[27412]: /rc.newwanip: Creating rrd update script Nov 18 05:48:12 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:48:12 XXXX xinetd[88961]: Swapping defaults Nov 18 05:48:12 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:48:12 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:48:12 XXXX check_reload_status: updating dyndns wan Nov 18 05:48:12 XXXX check_reload_status: Reloading filter Nov 18 05:48:14 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (@.X.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:48:14 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:48:14 XXXX xinetd[88961]: Swapping defaults Nov 18 05:48:14 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:48:14 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:48:14 XXXX php-fpm[27412]: /rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - X.X.X.X -> X.X.X.X - Restarting packages. Nov 18 05:48:14 XXXX check_reload_status: Starting packages Nov 18 05:48:15 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS (X.X.X): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry. Nov 18 05:48:15 XXXX php-fpm[88455]: /rc.start_packages: Restarting/Starting all packages. Nov 18 05:48:15 XXXX php-fpm[88455]: [pfBlockerNG] Starting cron process. Nov 18 05:48:15 XXXX SuricataStartup[3518]: Suricata START for WAN(34205_em0)... Nov 18 05:48:15 XXXX check_reload_status: Syncing firewall Nov 18 05:48:15 XXXX check_reload_status: Reloading filter Nov 18 05:48:16 XXXX php-fpm[97376]: /rc.dyndns.update: The command '/usr/local/bin/nsupdate -k /var/etc/nsupdatekey0 /var/etc/nsupdatecmds0' returned exit code '1', the output was 'dns_request_getresponse: expected a TSIG or SIG(0)' Nov 18 05:48:16 XXXX php-fpm[97376]: /rc.dyndns.update: phpDynDNS: ERROR while updating IP Address (A) for X.X.X (X.X.X.X) Nov 18 05:48:17 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:48:17 XXXX xinetd[88961]: Swapping defaults Nov 18 05:48:17 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:48:17 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:48:17 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] - squid_resync function call pr:1 bp: rpc:no Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Adding cronjobs ... Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Antivirus features disabled. Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Removing freshclam cronjob. Nov 18 05:48:22 XXXX php-fpm[88455]: /rc.start_packages: [squid] Stopping any running proxy monitors Nov 18 05:48:23 XXXX php-fpm[88455]: /rc.start_packages: [squid] Reloading for configuration sync... Nov 18 05:48:23 XXXX php-fpm[88455]: /rc.start_packages: [squid] Starting a proxy monitor script Nov 18 05:48:24 XXXX check_reload_status: Reloading filter Nov 18 05:48:24 XXXX php-cgi: haproxy: reload old pid:64450 Nov 18 05:48:24 XXXX php-cgi: haproxy: started new pid:45715 Nov 18 05:48:26 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:48:26 XXXX xinetd[88961]: Swapping defaults Nov 18 05:48:26 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:48:26 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:48:32 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:48:47 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:49:02 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:49:17 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:49:32 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:49:47 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:50:02 XXXX suricata[31920]: [1:2024897:1] ET USER_AGENTS Go HTTP Client User-Agent [Classification: Misc activity] [Priority: 3] {TCP} IPV6-2:57991 -> 2607:f8b0:4009:0805:0000:0000:0000:2003:80 Nov 18 05:50:21 XXXX check_reload_status: Linkup starting igb0 Nov 18 05:50:21 XXXX kernel: pid 31920 (suricata), uid 0: exited on signal 11 (core dumped) Nov 18 05:50:21 XXXX kernel: igb0: link state changed to DOWN Nov 18 05:50:22 XXXX php-fpm[37754]: /rc.linkup: DEVD Ethernet detached event for lan Nov 18 05:50:22 XXXX check_reload_status: Reloading filter Nov 18 05:50:23 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:50:23 XXXX xinetd[88961]: Swapping defaults Nov 18 05:50:23 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:50:23 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:50:25 XXXX kernel: igb0: link state changed to UP Nov 18 05:50:25 XXXX check_reload_status: Linkup starting igb0 Nov 18 05:50:26 XXXX php-fpm[27412]: /rc.linkup: DEVD Ethernet attached event for lan Nov 18 05:50:26 XXXX php-fpm[27412]: /rc.linkup: HOTPLUG: Configuring interface lan Nov 18 05:50:26 XXXX check_reload_status: Restarting ipsec tunnels Nov 18 05:50:26 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: Info: starting on em0. Nov 18 05:50:26 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: No IPv6 address found for interface WAN [wan]. Nov 18 05:50:27 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: Info: starting on em0. Nov 18 05:50:27 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: No IPv6 address found for interface WAN [wan]. Nov 18 05:50:27 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: Info: starting on em0. Nov 18 05:50:27 XXXX php-fpm[60994]: /rc.newwanipv6: rc.newwanipv6: on (IP address: IPV6.2) (interface: wan) (real interface: em0). Nov 18 05:50:29 XXXX php-fpm[27412]: /rc.linkup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1542541829] unbound[2124:0] error: bind: address already in use [1542541829] unbound[2124:0] fatal error: could not open ports' Nov 18 05:50:30 XXXX php-fpm[60994]: /rc.newwanipv6: Removing static route for monitor IPV6-1 and adding a new route through IPV6-1%em0 Nov 18 05:50:30 XXXX check_reload_status: Reloading filter Nov 18 05:50:32 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:50:32 XXXX xinetd[88961]: Swapping defaults Nov 18 05:50:32 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:50:32 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 05:50:32 XXXX check_reload_status: updating dyndns lan Nov 18 05:50:32 XXXX check_reload_status: Reloading filter Nov 18 05:50:33 XXXX xinetd[88961]: Starting reconfiguration Nov 18 05:50:33 XXXX xinetd[88961]: Swapping defaults Nov 18 05:50:33 XXXX xinetd[88961]: readjusting service 19000-tcp Nov 18 05:50:33 XXXX xinetd[88961]: Reconfigured: new=0 old=1 dropped=0 (services) Nov 18 06:05:04 XXXX php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort Subscriber rules are up to date... Nov 18 06:05:16 XXXX php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules are up to date... Nov 18 06:05:16 XXXX php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished. Nov 18 06:05:16 XXXX check_reload_status: Syncing firewall Nov 18 06:15:07 XXXX php: [pfBlockerNG] Starting cron process.It looks like both interfaces are going down for new IPs and Suricata is crashing shortly thereafter as the netmap queue fills up? Maybe? heh
Or the interface is failing with high packet loss, resetting, and Suricata crashes?
-
You are running Snort and Suricata on the same box? That is certainly not recommended. They will potentially step on each other, especially when using Legacy Mode. I do see you appear to be using Inline IPS mode for Suricata, but running both Snort and Suricata on the same box is not a good decision. Choose only one of the IDS/IPS systems to use. Do not use both on the same box.
Your second issue is something appears to be causing your interface to flap. Each time that happens the internal pfSense system will restart all the packages. That's how Suricata (and Snort) can collide with multiple copies of themselves, especially if one startup is still happening and then pfSense comes along and issues another closely spaced "restart all packages" command in response to an interface down/up cycle. I'm not positive that's what is happening in your case, but it appears from the log data your interface cycled at least twice in a relatively short period.
-
I started with Snort and switched to Suricata ~1 year ago. Snort has no configured interfaces, but it looks like it was still updating rules (I didn't notice they were still running until you pointed it out). I'll remove the pkg entirely if that might be causing an issue.
I agree it looks like the interfaces are cycling. It looks like both LAN igb0 and WAN em0 cycled around the same time. It looks to me like rc.gateway_alarm detects 22% packet loss and cycles the interfaces assuming something went wrong.
I'll try to investigate what's going on with the interfaces, but in the meantime, would it be an acceptable solution to modify rc.start_packages (or similar) to simply rm *.pid in the appropriate suricata directories? Or would that be dangerous in ways I don't understand?
-
@boobletins said in Still seeing suricata stop an interface due to .pid error:
I started with Snort and switched to Suricata ~1 year ago. Snort has no configured interfaces, but it looks like it was still updating rules (I didn't notice they were still running until you pointed it out). I'll remove the pkg entirely if that might be causing an issue.
I agree it looks like the interfaces are cycling. It looks like both LAN igb0 and WAN em0 cycled around the same time. It looks to me like rc.gateway_alarm detects 22% packet loss and cycles the interfaces assuming something went wrong.
I'll try to investigate what's going on with the interfaces, but in the meantime, would it be an acceptable solution to modify rc.start_packages (or similar) to simply rm *.pid in the appropriate suricata directories? Or would that be dangerous in ways I don't understand?
No, no danger at all in deleting the file. But that is just going to be masking the problem. The file being there is a symptom caused by something crashing Suricata.
I would try changing whatever host you are "pinging" for the gateway monitoring. Maybe that host is tardy responding to ICMP requests or even drops them when it gets busy.
-
-
@val said in Still seeing suricata stop an interface due to .pid error:
PM you the log file....it's way to big to post here.
Thanks bmeeks.
I looked through you log file. What version of the Snort Rules Snapshot file are you using? You should be using only rules packages for Snort 2.9.x if you are running Snort rules with Suricata. Your file name should be snortrules-snapshot-29120.tar.gz. Do not use the Snort3 rules (that means do not use any Snort rules file with 3 in the name). You should not be seeing those "unknown reference" error messages. The only time I've noticed those is when the user has downloaded the rules meant for use only with the new Snort3 beta package from the Snort team.
-
The version I am using and file name it's:-
snortrules-snapshot-29111.tar.gzThanks for that info I will change it to see if process still kill it self.
-
Just want to add I have been having the same issue with the Suricata .pid file becoming stale, and the engine failing to restart because of this after a crash. I am using SG-3100. I also run pfBlocker. I notice when I am tweaking setting within here, most of the time, that's when Suricata crashes. Having to manually rm the .pid file to start Suricata.
-
@bhjitsense said in Still seeing suricata stop an interface due to .pid error:
Just want to add I have been having the same issue with the Suricata .pid file becoming stale, and the engine failing to restart because of this after a crash. I am using SG-3100. I also run pfBlocker. I notice when I am tweaking setting within here, most of the time, that's when Suricata crashes. Having to manually rm the .pid file to start Suricata.
The SG-3100 crash is due to a compiler optimization problem for armv6 and armv7 CPUs (like those used in the SG-1000 and SG-3100 appliances). I've been in contact with the pfSense team about this, but so far there is no resolution posted. The only fix for now is to NOT run Suricata on SG-3100 hardware. If you do, it will continue to randomly crash with the Signal 10 Bus Error. The Signal 10 crash leaves the PID file in place, so the next time you attempt to start Suricata it will see the file remaining from the previously crashed instance and complain. The stale PID file is a symptom and not a cause in this case.
You can research Google for what Signal 10 Bus Errors are and what causes them. It is due to the clang/llvm compiler generating machine opcodes that do not support unaligned memory access on arm processors.
-
@bmeeks said in Still seeing suricata stop an interface due to .pid error:
The only fix for now is to NOT run Suricata on SG-3100 hardware.
Hmm, interesting. I checked the router I had posted about a week ago, and Suricata is still running. Is the crash random or only when making changes as @bhjitsense suggested might be the case? (I have made no changes in the past week)
-
@teamits said in Still seeing suricata stop an interface due to .pid error:
@bmeeks said in Still seeing suricata stop an interface due to .pid error:
The only fix for now is to NOT run Suricata on SG-3100 hardware.
Hmm, interesting. I checked the router I had posted about a week ago, and Suricata is still running. Is the crash random or only when making changes as @bhjitsense suggested might be the case? (I have made no changes in the past week)
It will appear to be somewhat random, although in fact each and every time a particular piece of code is hit (the part with the opcodes I mentioned) the error will be thrown. The randomness comes in from the fact that the problem code might be part of an if-then statement in code where one piece of code is executed if a tested condition is true while a different section of code is executed if the condition tests as false. I have no idea where specifically in the Suricata code the problem lies. It very well could exist in several places.
This is not a problem on Intel hardware because all Intel CPUs will automatically fix-up and execute data loads or stores to unaligned addresses. Intel CPUs do this by default with the processor hardware. Since developers mostly target Intel hardware, they have all gotten quite complacent and sloppy with data access via structures and pointer casting in C programming. Intel hardware will cover for their sloppiness, but other hardware (like the armv6 and armv7 CPU) is not as forgiving. Of course part of the fault here also lies with the clang/llvm compiler used to produce the binary code for the arm hardware.
I also want to make clear this is an issue within the acutal Suricata binary code and has nothing at all to do with the PHP GUI code of pfSense. Remember all that the GUI packages for Suricata and Snort do on pfSense is provide a wrapper to let users easily create the text configuration files the underlying binaries need to run.
-
Thanks for the explanation. I meant to ask, is this issue with the latest release version of the Suricata package/binaries or prior versions also?
-
@teamits said in Still seeing suricata stop an interface due to .pid error:
Thanks for the explanation. I meant to ask, is this issue with the latest release version of the Suricata package/binaries or prior versions also?
It can very well exist in any previous version, but appears to have reared its head in the latest binary update. The explanation of what unaligned access is and how it happens in C programming code is a bit long-winded and requires a good bit of hardware understanding (things like memory bus widths, CPU register loads/stores and how memory access works at a hardware level) in order to fully grasp the concept. You can Google "unaligned memory access" and start your research if interested.
It could be that a simple code change that happened in the latest binary (maybe fixing some other bug) caused this issue on arm hardware. Key to understanding the impact of unaligned access issues is also understanding what role compilers play in producing the acutal binary instruction codes from the higher-level C programming code. On FreeBSD, the compiler used for this is clang/llvm. Linux traditionally uses gcc. Of course Windows uses Microsoft-supplied compilers. Each compiler will produce slightly different binary code from the exact same C high-level code. On FreeBSD, that compiler produces a poorly chosen set of opcodes for certain memory access operations. It chooses to use a pair of opcodes that the arm CPU cannot fix-up on the fly for unaligned access. There are other opcodes that can perform the same operation and the arm CPU can auto fix-up those memory accesses to prevent the unaligned access. Research LDM/STM and LDR/STR instruction opcodes on armv6 and armv7 microprocessors to see what I mean.
EDIT: up above, when I say "poorly chosen set of opcodes" I mean it chooses speed over compatibility. The opcodes it chooses to use (LDM/STM) do execute ever so slightly faster than their counterparts (LDR/STR), but the latter codes support unaligned memory access without crashing on the Signal 10 Bus Error. The former LDM/STM instructions will crash if the original Suricata binary source code programmer inadvertently asked the CPU to load or store a piece of data that causes an unaligned memory access.
There are also frequent battles of words between compiler developers and other C programmers. The compiler gurus say the C programmers should not be so sloppy and make sure to avoid unaligned access issues, but the C programmers retaliate with the old "well it works on Intel without issue" or "it works fine on gcc", etc. So that can mean a lot of posturing on each side with nothing useful really happening. There is reluctance for say the Suricata team to dive into this because it is only affecting Suricata running on arm hardware where Suricata was compiled by clang/llvm. That's a pretty small footprint of users compared to Linux land. Thus no incentive for the Suricata developers to spend hours trying to find where the issue is.
-
!@#%$(*!@#^%
I had a whole post written up describing how to solve this on my igb0 card and the forum thought it was spam and poof. Extremely frustrating.
IPV6 TX Checksums are not disabled via the GUI when they appear to be. Command line ifconfig can fix this issue (until a reboot). I can reliably reproduce this on my igb interface, haven't tested the em yet.
Hope this helps someone? If it lets me post..........
Edit: I give up, PMing you Bill... this is absurd.
-
@boobletins said in Still seeing suricata stop an interface due to .pid error:
!@#%$(*!@#^%
I had a whole post written up describing how to solve this on my igb0 card and the forum thought it was spam and poof. Extremely frustrating.
IPV6 TX Checksums are not disabled via the GUI when they appear to be. Command line ifconfig can fix this issue (until a reboot). I can reliably reproduce this on my igb interface, haven't tested the em yet.
Hope this helps someone? If it lets me post..........
Edit: I give up, PMing you Bill... this is absurd.
Replied to your PM.
-
@boobletins said in Still seeing suricata stop an interface due to .pid error:
!@#%$(*!@#^%
I had a whole post written up describing how to solve this on my igb0 card and the forum thought it was spam and poof. Extremely frustrating.
IPV6 TX Checksums are not disabled via the GUI when they appear to be. Command line ifconfig can fix this issue (until a reboot). I can reliably reproduce this on my igb interface, haven't tested the em yet.
Hope this helps someone? If it lets me post..........
Edit: I give up, PMing you Bill... this is absurd.
I'm not 100% positive this is the cause of the Signal 10 error. That error definitely indicates an unaligned memory access. Now it is possible that the checksum failures may be leading to that "particular section" of bad opcodes I mentioned in my earlier post getting executed and thus throwing the Signal 10 error.
-
@bmeeks said in Still seeing suricata stop an interface due to .pid error:
@val said in Still seeing suricata stop an interface due to .pid error:
PM you the log file....it's way to big to post here.
Thanks bmeeks.
I looked through you log file. What version of the Snort Rules Snapshot file are you using? You should be using only rules packages for Snort 2.9.x if you are running Snort rules with Suricata. Your file name should be snortrules-snapshot-29120.tar.gz. Do not use the Snort3 rules (that means do not use any Snort rules file with 3 in the name). You should not be seeing those "unknown reference" error messages. The only time I've noticed those is when the user has downloaded the rules meant for use only with the new Snort3 beta package from the Snort team.
Hi bmeeks
I have since moved away from suricata backon Snort for now, my internet connection it's through an PPPoE connection so from my understanding suricata doesn't play well with PPPoE.I have tried few difference thing all result the same suricata still kill it self and wouldn't start again til I delete the pid file.
Thanks for all the help.
