Snort not restart on interface

fireodo

@bmeeks said in Snort not restart on interface:

Okay. Give me some time to investigate in a virtual machine. I also have to look into a Suricata problem on aarch64 hardware for the pfSense team, so it may be a day or two. The only thing I can imagine at this point is maybe something is different in the rc.start_packages script and I need to make an adjustment to how the Snort package interracts with it. Just a guess at this point, though.

However, even should I find an issue there, I still think you having the cron task will eventually be problematic should its execution happen to coincide with a time when pfSense is responding to one of those loss-of-lcp signal issues.

The cron task is normally in the middle of the night (at 2:10) the probability to do the reconnection on a modem resync is like gaining in Lotto :-)
I hope not to put you on a wrong way. Thank you VERY MUCH for your time and pacience!

Regards,
fireodo

bmeeks

I took some time to have a quick look in a 2.4.5 virtual machine before starting on the Suricata issue. I can see nothing wrong with the interraction of Snort and the rc.start_packages script. Everything starts and stops fine in the virtual machine. I don't know what exactly it would be, but I now suspect something unique to your setup.

Examine your config.xml file for the firewall. It should have a section for <installedpackages><service>, and the <service> section for Snort should look like this --

	<service>
		<name>snort</name>
		<rcfile>snort.sh</rcfile>
		<executable>snort</executable>
		<description><![CDATA[Snort IDS/IPS Daemon]]></description>
	</service>

This is the section that the rc.start_packges script scans to find the shell script to execute when starting packages. This same script and section gets called when you boot up the firewall as well. Does Snort start by itself on a firewall reboot, or do you have to intervene manually?

One thing I suggest you add to your cron task is a command to shut down Snort and then later restart it again from inside the cron task. Use these commands:

/usr/local/etc/rc.d/snort.sh stop
sleep 10
/usr/local/etc/rc.d/snort.sh start

The sleep() command is only required if the stop and start are immediately adjacent to each other. I suggest stopping Snort BEFORE you cycle the PPPoE interface, and then starting Snort again only AFTER the PPPoE interface is back up.

fireodo

@bmeeks said in Snort not restart on interface:

I took some time to have a quick look in a 2.4.5 virtual machine before starting on the Suricata issue. I can see nothing wrong with the interraction of Snort and the rc.start_packages script. Everything starts and stops fine in the virtual machine. I don't know what exactly it would be, but I now suspect something unique to your setup.

Examine your config.xml file for the firewall. It should have a section for <installedpackages><service>, and the <service> section for Snort should look like this --
  <service>
  	<name>snort</name>
  	<rcfile>snort.sh</rcfile>
  	<executable>snort</executable>
  	<description><![CDATA[Snort IDS/IPS Daemon]]></description>
  </service>

Is exactly so present in my config.

This is the section that the rc.start_packges script scans to find the shell script to execute when starting packages. This same script and section gets called when you boot up the firewall as well. Does Snort start by itself on a firewall reboot, or do you have to intervene manually?

Snort starts on reboot by itself no need to intervene manually.

One thing I suggest you add to your cron task is a command to shut down Snort and then later restart it again from inside the cron task. Use these commands:
/usr/local/etc/rc.d/snort.sh stop
sleep 10
/usr/local/etc/rc.d/snort.sh start
The sleep() command is only required if the stop and start are immediately adjacent to each other. I suggest stopping Snort BEFORE you cycle the PPPoE interface, and then starting Snort again only AFTER the PPPoE interface is back up.

On normal cycling of pppoe connection snort starts correctly and works correctly - only in case of modem loose connection it makes problems.

I added your command to stop/start snort at the end of the /usr/local/sbin/ppp-linkup script and now snort stays functional even if the modem resyncs! I know thats only a workaraound but I can live with it! Good Night!

bmeeks

@fireodo said in Snort not restart on interface:

@bmeeks said in Snort not restart on interface:
I took some time to have a quick look in a 2.4.5 virtual machine before starting on the Suricata issue. I can see nothing wrong with the interraction of Snort and the rc.start_packages script. Everything starts and stops fine in the virtual machine. I don't know what exactly it would be, but I now suspect something unique to your setup.

Examine your config.xml file for the firewall. It should have a section for <installedpackages><service>, and the <service> section for Snort should look like this --
  <service>
  	<name>snort</name>
  	<rcfile>snort.sh</rcfile>
  	<executable>snort</executable>
  	<description><![CDATA[Snort IDS/IPS Daemon]]></description>
  </service>
Is exactly so present in my config.

This is the section that the rc.start_packges script scans to find the shell script to execute when starting packages. This same script and section gets called when you boot up the firewall as well. Does Snort start by itself on a firewall reboot, or do you have to intervene manually?

Snort starts on reboot by itself no need to intervene manually.
One thing I suggest you add to your cron task is a command to shut down Snort and then later restart it again from inside the cron task. Use these commands:
/usr/local/etc/rc.d/snort.sh stop
sleep 10
/usr/local/etc/rc.d/snort.sh start
The sleep() command is only required if the stop and start are immediately adjacent to each other. I suggest stopping Snort BEFORE you cycle the PPPoE interface, and then starting Snort again only AFTER the PPPoE interface is back up.
On normal cycling of pppoe connection snort starts correctly and works correctly - only in case of modem loose connection it makes problems.

Would it be a good idea to add this command you suggested to the /usr/local/sbin/ppp-linkup script?

I don't think it would hurt anything, but of course it would get wiped out with the next upgrade and you would have to remember to edit the script again.

I'm really not a PPPoE expert. I had DSL way, way back and dealt with PPPoE just a bit back then. But luckily with my carrier at the time, and with the DSL modem I had, I was able to switch over to PPPoA (PPP over ATM). I did that to take advantage of the full 1500-byte MTU instead of having to limit my MTU to 1492.

So you say your modem is in bridge mode, but I don't believe it actually is if you have a pppoe0 connection in pfsense. Or at least that's not how my DSL modem worked. I put the password and login name in the modem and it handled all the PPP stuff. What I got on the LAN side of the modem was a DHCP address (the public IP assigned by my ISP). Does your modem not support a mode of operation like that? It's been a long time, so maybe that was how the PPPoA worked for me and PPPoE would have been different.

Edit: the more I've tried to remember, perhaps I am remembering wrong for PPPoE. Maybe it did put a PPP interface on my pfSense and only the PPPoA mode was DHCP on the firewall side. That was 17 years ago and I can't recall the details now.

fireodo

@bmeeks said in Snort not restart on interface:

@fireodo said in Snort not restart on interface:
@bmeeks said in Snort not restart on interface:
I took some time to have a quick look in a 2.4.5 virtual machine before starting on the Suricata issue. I can see nothing wrong with the interraction of Snort and the rc.start_packages script. Everything starts and stops fine in the virtual machine. I don't know what exactly it would be, but I now suspect something unique to your setup.

Examine your config.xml file for the firewall. It should have a section for <installedpackages><service>, and the <service> section for Snort should look like this --
  <service>
  	<name>snort</name>
  	<rcfile>snort.sh</rcfile>
  	<executable>snort</executable>
  	<description><![CDATA[Snort IDS/IPS Daemon]]></description>
  </service>
Is exactly so present in my config.

This is the section that the rc.start_packges script scans to find the shell script to execute when starting packages. This same script and section gets called when you boot up the firewall as well. Does Snort start by itself on a firewall reboot, or do you have to intervene manually?

Snort starts on reboot by itself no need to intervene manually.
One thing I suggest you add to your cron task is a command to shut down Snort and then later restart it again from inside the cron task. Use these commands:
/usr/local/etc/rc.d/snort.sh stop
sleep 10
/usr/local/etc/rc.d/snort.sh start
The sleep() command is only required if the stop and start are immediately adjacent to each other. I suggest stopping Snort BEFORE you cycle the PPPoE interface, and then starting Snort again only AFTER the PPPoE interface is back up.
On normal cycling of pppoe connection snort starts correctly and works correctly - only in case of modem loose connection it makes problems.

Would it be a good idea to add this command you suggested to the /usr/local/sbin/ppp-linkup script?
I don't think it would hurt anything, but of course it would get wiped out with the next upgrade and you would have to remember to edit the script again.

That I have to do, thats no problem ... :-)

kiokoman

the more i think about it the more i find it strange

i put up a virtual machine and tested it myself

Mar 31 20:38:27	php-fpm	95633	/rc.start_packages: Restarting/Starting all packages.
Mar 31 20:38:26	check_reload_status	381	Starting packages
Mar 31 20:38:26	php-fpm	343	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.78.2 -> 192.168.78.2 - Restarting packages.
Mar 31 20:38:24	php-fpm	343	/rc.newwanip: Creating rrd update script
Mar 31 20:38:24	php-fpm	343	/rc.newwanip: Resyncing OpenVPN instances for interface OPT1.
Mar 31 20:38:21	php-fpm	343	/rc.newwanip: rc.newwanip: on (IP address: 192.168.78.2) (interface: OPT1[opt1]) (real interface: pppoe0).
Mar 31 20:38:21	php-fpm	343	/rc.newwanip: rc.newwanip: Info: starting on pppoe0.
Mar 31 20:38:20	ppp	12008	[opt1] IFACE: Rename interface ng0 to pppoe0
Mar 31 20:38:20	ppp	12008	[opt1] IFACE: Up event
Mar 31 20:38:20	check_reload_status	381	rc.newwanip starting pppoe0
Mar 31 20:38:19	check_reload_status	381	Rewriting resolv.conf
Mar 31 20:38:19	ppp	12008	[opt1] 192.168.78.2 -> 192.168.77.1
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: LayerUp
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: state change Ack-Sent --> Opened
Mar 31 20:38:19	ppp	12008	[opt1] PRIDNS 172.17.0.100
Mar 31 20:38:19	ppp	12008	[opt1] IPADDR 192.168.78.2
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: rec'd Configure Ack #8 (Ack-Sent)
Mar 31 20:38:19	ppp	12008	[opt1] PRIDNS 172.17.0.100
Mar 31 20:38:19	ppp	12008	[opt1] IPADDR 192.168.78.2
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: SendConfigReq #8
Mar 31 20:38:19	ppp	12008	[opt1] PRIDNS 172.17.0.100
Mar 31 20:38:19	ppp	12008	[opt1] 192.168.78.2 is OK
Mar 31 20:38:19	ppp	12008	[opt1] IPADDR 192.168.78.2
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: rec'd Configure Nak #7 (Ack-Sent)
Mar 31 20:38:19	ppp	12008	[opt1] PRIDNS 0.0.0.0
Mar 31 20:38:19	ppp	12008	[opt1] IPADDR 0.0.0.0
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: SendConfigReq #7
Mar 31 20:38:19	ppp	12008	[opt1] SECDNS 0.0.0.0
Mar 31 20:38:19	ppp	12008	[opt1] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: rec'd Configure Reject #6 (Ack-Sent)
Mar 31 20:38:19	ppp	12008	[opt1_link0] rec'd unexpected protocol CCP, rejecting
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: state change Req-Sent --> Ack-Sent
Mar 31 20:38:19	ppp	12008	[opt1] IPADDR 192.168.77.1
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: SendConfigAck #1
Mar 31 20:38:19	ppp	12008	[opt1] 192.168.77.1 is OK
Mar 31 20:38:19	ppp	12008	[opt1] IPADDR 192.168.77.1
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: rec'd Configure Request #1 (Req-Sent)
Mar 31 20:38:19	ppp	12008	[opt1] SECDNS 0.0.0.0
Mar 31 20:38:19	ppp	12008	[opt1] PRIDNS 0.0.0.0
Mar 31 20:38:19	ppp	12008	[opt1] COMPPROTO VJCOMP, 16 comp. channels, no comp-cid
Mar 31 20:38:19	ppp	12008	[opt1] IPADDR 0.0.0.0
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: SendConfigReq #6
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: state change Starting --> Req-Sent
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: Up event
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: LayerStart
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: state change Initial --> Starting
Mar 31 20:38:19	ppp	12008	[opt1] IPCP: Open event
Mar 31 20:38:19	ppp	12008	[opt1] Bundle: Status update: up 1 link, total bandwidth 64000 bps
Mar 31 20:38:19	ppp	12008	[opt1_link0] Link: Join bundle "opt1"
Mar 31 20:38:19	ppp	12008	[opt1_link0] Link: Matched action 'bundle "opt1" ""'
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: authorization successful
Mar 31 20:38:19	ppp	12008	[opt1_link0] MESG: Welcome
Mar 31 20:38:19	ppp	12008	[opt1_link0] PAP: rec'd ACK #1 len: 12
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: LayerUp
Mar 31 20:38:19	ppp	12008	[opt1_link0] PAP: sending REQUEST #1 len: 14
Mar 31 20:38:19	ppp	12008	[opt1_link0] PAP: using authname "test"
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: auth: peer wants PAP, I want nothing
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: state change Ack-Sent --> Opened
Mar 31 20:38:19	ppp	12008	[opt1_link0] MAGICNUM 0xe92dbee8
Mar 31 20:38:19	ppp	12008	[opt1_link0] MRU 1492
Mar 31 20:38:19	ppp	12008	[opt1_link0] PROTOCOMP
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: rec'd Configure Ack #3 (Ack-Sent)
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: state change Req-Sent --> Ack-Sent
Mar 31 20:38:19	ppp	12008	[opt1_link0] AUTHPROTO PAP
Mar 31 20:38:19	ppp	12008	[opt1_link0] MAGICNUM 0xe7f15140
Mar 31 20:38:19	ppp	12008	[opt1_link0] MRU 1492
Mar 31 20:38:19	ppp	12008	[opt1_link0] PROTOCOMP
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: SendConfigAck #1
Mar 31 20:38:19	ppp	12008	[opt1_link0] AUTHPROTO PAP
Mar 31 20:38:19	ppp	12008	[opt1_link0] MAGICNUM 0xe7f15140
Mar 31 20:38:19	ppp	12008	[opt1_link0] MRU 1492
Mar 31 20:38:19	ppp	12008	[opt1_link0] PROTOCOMP
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: rec'd Configure Request #1 (Req-Sent)
Mar 31 20:38:19	ppp	12008	[opt1_link0] MAGICNUM 0xe92dbee8
Mar 31 20:38:19	ppp	12008	[opt1_link0] MRU 1492
Mar 31 20:38:19	ppp	12008	[opt1_link0] PROTOCOMP
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: SendConfigReq #3
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: state change Starting --> Req-Sent
Mar 31 20:38:19	ppp	12008	[opt1_link0] LCP: Up event
Mar 31 20:38:19	ppp	12008	[opt1_link0] Link: UP event
Mar 31 20:38:19	ppp	12008	[opt1_link0] PPPoE: connection successful
Mar 31 20:38:19	ppp	12008	PPPoE: rec'd ACNAME "pfSense.kiokoman.home"
Mar 31 20:38:17	ppp	12008	[opt1_link0] PPPoE: Connecting to ''
Mar 31 20:38:17	ppp	12008	[opt1_link0] Link: reconnection attempt 20
Mar 31 20:38:13	ppp	12008	[opt1_link0] Link: reconnection attempt 20 in 4 seconds
......
Mar 31 20:34:31	ppp	12008	[opt1] IPCP: LayerFinish
Mar 31 20:34:31	ppp	12008	[opt1] IPCP: Down event
Mar 31 20:34:31	ppp	12008	[opt1] IPCP: state change Stopping --> Closing
Mar 31 20:34:31	ppp	12008	[opt1] IPCP: Close event
Mar 31 20:34:31	ppp	12008	[opt1] Bundle: Status update: up 0 links, total bandwidth 9600 bps
Mar 31 20:34:31	ppp	12008	[opt1_link0] Link: Leave bundle "opt1"
Mar 31 20:34:31	ppp	12008	[opt1_link0] LCP: state change Opened --> Stopping
Mar 31 20:34:31	ppp	12008	[opt1_link0] LCP: rec'd Terminate Request #2 (Opened)
Mar 31 20:34:31	ppp	12008	[opt1] IFACE: Rename interface pppoe0 to pppoe0
Mar 31 20:34:31	ppp	12008	[opt1] IFACE: Down event
Mar 31 20:34:31	check_reload_status	381	Rewriting resolv.conf
Mar 31 20:34:30	ppp	12008	[opt1] IPCP: LayerDown
Mar 31 20:34:30	ppp	12008	[opt1] IPCP: SendTerminateAck #5
Mar 31 20:34:30	ppp	12008	[opt1] IPCP: state change Opened --> Stopping
Mar 31 20:34:30	ppp	12008	[opt1] IPCP: rec'd Terminate Request #3 (Opened)
Mar 31 20:25:07	kernel		pppoe0: promiscuous mode enabled
Mar 31 20:25:07	SnortStartup	5730	Snort START for pppoe(pppoe0)...
Mar 31 20:25:07	php-fpm	95633	/rc.start_packages: Restarting/Starting all packages.
Mar 31 20:25:06	check_reload_status	381	Starting packages
Mar 31 20:25:06	php-fpm	344	/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - 192.168.78.2 -> 192.168.78.2 - Restarting packages.
Mar 31 20:25:04	php-fpm	344	/rc.newwanip: Creating rrd update script
Mar 31 20:25:04	php-fpm	344	/rc.newwanip: Resyncing OpenVPN instances for interface OPT1.
Mar 31 20:25:00	php-fpm	344	/rc.newwanip: rc.newwanip: on (IP address: 192.168.78.2) (interface: OPT1[opt1]) (real interface: pppoe0).
Mar 31 20:25:00	php-fpm	344	/rc.newwanip: rc.newwanip: Info: starting on pppoe0.
Mar 31 20:24:59	ppp	12008	[opt1] IFACE: Rename interface ng0 to pppoe0
Mar 31 20:24:59	ppp	12008	[opt1] IFACE: Up event
Mar 31 20:24:59	check_reload_status	381	rc.newwanip starting pppoe0
Mar 31 20:24:58	check_reload_status	381	Rewriting resolv.conf
Mar 31 20:24:58	ppp	12008	[opt1] 192.168.78.2 -> 192.168.77.1

snort start correctly, does not matter if i stop the pppoe server or i stop the pppoe client
but as you can see " IFACE Rename interface ng0 to pppoe" is present
this is mpd5 doing
it's not snort at fault here, you should investigate why the pppoe interface disappear leadig snort to stop working, and i don't think it's the only problem you will get from it
the test was done under pfSense 2.5.0, maybe i will try tomorrow with a new vm with 2.4.5

fireodo

@kiokoman said in Snort not restart on interface:

the more i think about it the more i find it strange
it's not snort at fault here, you should investigate why the pppoe interface disappear leadig snort to stop working, and i don't think it's the only problem you will get from it

I'll keep an eye on it!

the test was done under pfSense 2.5.0, maybe i will try tomorrow with a new vm with 2.4.5

OK, lets see what happends

Thanks for your effort!
fireodo

bmeeks

@kiokoman, thank you for the testing. I too am intrigued by why the interface disappears. That will obviously confuse Snort when it can't find the interface it is configured to sniff.

Initially I thought Snort was just a victim here, but I decided to look into other possibilities. However, that search and my own testing led me to think Snort is not really at fault here.

kiokoman

ok tested it today with a vm 2.4.5, same results
no problem when i stop the pppoe server or the client, must be something on your environment,or some special event that i can't replicate. did you set any particular settings for your pppoe connection? system tunable ?

fireodo

@kiokoman said in Snort not restart on interface:

ok tested it today with a vm 2.4.5, same results
no problem when i stop the pppoe server or the client, must be something on your environment, did you set any particular settings for your pppoe connection? system tunable ?

Nothing special - only the credentials necessary for provider. I dont use a pppoe server only a client. When I restart here the "wan" (pppoe) there is no problem - the problem occurs when by any circumstances the modem loose sync/or power.

DSLAM(provider)------>(my)DSL-Modem ------> pfsense (pppoe client) ------> LAN

kiokoman

yes , the pppoe server is used by me on another pfsense to simulate a provider for another vm with pfsense 2.4.5
so if i stop the pppoe server is like as you turn off your dsl modem
wan ->pfsense (pppoe server) --> another pfsense (pppoe client) -> lan

fireodo

@kiokoman said in Snort not restart on interface:

yes , the pppoe server is used by me on another pfsense to simulate a provider for another vm with pfsense 2.4.5
so if i stop the pppoe server is like as you turn off your dsl modem
wan ->pfsense (pppoe server) --> another pfsense (pppoe client) -> lan

Ah - OK! Thanks for testing - now I have to dig further to find what in my case went wrong.

bmeeks

@fireodo said in Snort not restart on interface:

@kiokoman said in Snort not restart on interface:

yes , the pppoe server is used by me on another pfsense to simulate a provider for another vm with pfsense 2.4.5
so if i stop the pppoe server is like as you turn off your dsl modem
wan ->pfsense (pppoe server) --> another pfsense (pppoe client) -> lan

Ah - OK! Thanks for testing - now I have to dig further to find what in my case went wrong.

Don't forget to either delete or go back and update your bug report on the pfSense Redmine site so that issue can maybe be closed if not an actual system bug.

fireodo

@bmeeks said in Snort not restart on interface:

@fireodo said in Snort not restart on interface:

@kiokoman said in Snort not restart on interface:

yes , the pppoe server is used by me on another pfsense to simulate a provider for another vm with pfsense 2.4.5
so if i stop the pppoe server is like as you turn off your dsl modem
wan ->pfsense (pppoe server) --> another pfsense (pppoe client) -> lan

Ah - OK! Thanks for testing - now I have to dig further to find what in my case went wrong.

Don't forget to either delete or go back and update your bug report on the pfSense Redmine site so that issue can maybe be closed if not an actual system bug.

Thanks, I'll do so!

NollipfSense

This is a very interesting case study, and analysis...thank you all for sharing!