Snort does not run in WAN interface in pfSense 2.1



  • Hi:

    Installed Snort and activated to all the interfaces. Snort is enabled and running in both LAN and OPT1 interfaces, but not running in the WAN interface though enabled. Trying to run by clicking on the red x icon does not do anything.

    Checking the snort log outputs segmentation fault.

    clog /var/log/snort/snort_bge057263/alert 
    Segmentation fault (core dumped)
    


  • @zenny:

    Hi:

    Installed Snort and activated to all the interfaces. Snort is enabled and running in both LAN and OPT1 interfaces, but not running in the WAN interface though enabled. Trying to run by clicking on the red x icon does not do anything.

    Checking the snort log outputs segmentation fault.

    clog /var/log/snort/snort_bge057263/alert 
    Segmentation fault (core dumped)
    

    Whoa!  A seg-fault is pretty severe.  First up, are all the interfaces the same hardware NIC?  Just trying to eliminate any driver issues.  Snort puts the interfaces in promiscuous mode.

    Here are some suggested troubleshooting steps:

    1. Try Snort on the WAN with no rules enabled at all.  See if it starts up.

    2.  If that works, then add a very minimal set of rules like the Emerging Threats ET_CIARMY rules, and try to stop and restart again.

    Post back with the results.

    Bill



  • @bmeeks:

    Whoa!   A seg-fault is pretty severe.  First up, are all the interfaces the same hardware NIC?  Just trying to eliminate any driver issues. Snort puts the interfaces in promiscuous mode.

    The WAN has broadcom gigabit (bge) and the rest two are intel gigabit (em). Two intel interfaces works fine but with bge it refuses to start. Necessary tweaks are applied for relevant network cards in /boot/loader.conf.local.

    Here are some suggested troubleshooting steps:

    1. Try Snort on the WAN with no rules enabled at all.  See if it starts up.

    Yes it did!

    2.  If that works, then add a very minimal set of rules like the Emerging Threats ET_CIARMY rules, and try to stop and restart again.

    Post back with the results.

    Bill

    Yes, with just the et_ciarmy rules, it runs, but when I added more ET rules, it didn't. On the other hand, 'Snort Text rules' and 'Snort SO rules' are greyed preventing from selection, except 'Snort GPLv2 Community Rules (VRT certified)' which can be (un)checked.



  • @zenny:

    Yes, with just the et_ciarmy rules, it runs, but when I added more ET rules, it didn't. On the other hand, 'Snort Text rules' and 'Snort SO rules' are greyed preventing from selection, except 'Snort GPLv2 Community Rules (VRT certified)' which can be (un)checked.

    Unless you have an Oinkcode and have entered it on the Global Settings tab, then the Snort VRT selections will be grayed-out.  You would also need to perform an update of the rules downloaded from the Updates tab once enabling the Snort VRT rules.

    Is this a 2.0.x or 2.1 platform?  If 2.0.x, you might consider trying the 2.1RC0 snapshot.  It will have newer drivers for the NICs.  I'm kind of thinking maybe the NIC driver is at fault here, but I'm not sure.  The fact Snort runs OK on the Intel NICs suggests the basic Snort installation is fine.  The exact same binary and libraries are used for each interface instance.  It just launches an additional instance of the executable binary for each interface.

    Are the Preprocessor settings configured the same on the interfaces?

    Bill



  • I have oinkcode inserted. Preprocessors configured on the same interface. This is on pfSense 2.1 RC0 snapshot.



  • @zenny:

    I have oinkcode inserted. Preprocessors configured on the same interface. This is on pfSense 2.1 RC0 snapshot.

    Couple more questions. I am trying to figure out why the Snort VRT things are grayed-out for you.  On the Categories tab, do you have "Use IPS Policy" checked?  That will automatically disable the checkboxes for the Snort Text and SO Rules when checked.  Also, I assume on the Global Settings tab that the Snort VRT rules are enabled (that box is checked)?  This is global, so if you are using Snort VRT rules on the other interfaces, then never mind.

    Have you tried configuring the exact same rules on the WAN as you have on a working interface just for testing purposes?  That would help eliminate some other possibilities.

    Bill



  • Thanks Bill for your explicit explanation.

    Yes, the 'Use IPS policy' checked which is causing the snort rules to grey out.

    I have the same configuration for DMZ (em) as WAN (bge), and it works flawlessly. Maybe it is a bge driver issue. In that case, the snort on bge interface works fine when I choose a single ET_CIARMY rule, but fails whenever other rules are chosen. Pretty weird, isn't it?!



  • @zenny:

    Thanks Bill for your explicit explanation.

    Yes, the 'Use IPS policy' checked which is causing the snort rules to grey out.

    I have the same configuration for DMZ (em) as WAN (bge), and it works flawlessly. Maybe it is a bge driver issue. In that case, the snort on bge interface works fine when I choose a single ET_CIARMY rule, but fails whenever other rules are chosen. Pretty weird, isn't it?!

    Yes, that is quite weird.  When you say a "single ET_CIARMY rule", do you literally mean a single rule out of that file, or all the rules that are default enabled in that particular Category file?  I assume it's the latter (all the default enabled rules in that single category).

    If you have not tried this already, delete the WAN interface entirely from the Snort Interfaces tab, and then add it back and configure rules on it again.  Just in case something got stored corrupted in the section of the config.xml file for the Snort WAN settings.

    I am running out of ideas at this point.

    Bill


  • Banned

    Stupid question but have you enabled multiple snort interface mappings??



  • @Supermule:

    Stupid question but have you enabled multiple snort interface mappings??

    pfSense GUI is so exhaustive (I am old-styled command line guy) that I could not figure out where to enable "multiple snort interface mappings"! Do you mean to check the checkbox against 'Enable' against each interfaces in snort gui? In that case, yes!



  • I thought that it has to do with the bge driver that Snort is not running on WAN as Bill suspected, and therefore, swapped the em driver running on LAN with the bge running on WAN. After the swapping, even with WAN with em (which is working in LAN) stopped to work and LAN with bge driver starts Snort (whereas WAN with bge was not working). Therefore, there is something that prevents WAN to run snort.

    The only thing that was displayed on console was:

    swap_pager: out of swap space
    swap_pager_getswapspace(16):failed

    And when I checked system.log

    It showed up:

    Jun 25 22:22:19 pfSense0 kernel: swap_pager: out of swap space
    Jun 25 22:22:19 pfSense0 kernel: swap_pager_getswapspace(16): failed
    ….
    Jun 25 22:31:10 pfSense0 snort[50004]: FATAL ERROR: fpcreate.c(1555) Failed to compile port group patterns.

    Any inputs appreciated!



  • @zenny:

    I thought that it has to do with the bge driver that Snort is not running on WAN as Bill suspected, and therefore, swapped the em driver running on LAN with the bge running on WAN. After the swapping, even with WAN with em (which is working in LAN) stopped to work and LAN with bge driver starts Snort (whereas WAN with bge was not working). Therefore, there is something that prevents WAN to run snort.

    The only thing that was displayed on console was:

    swap_pager: out of swap space
    swap_pager_getswapspace(16):failed

    And when I checked system.log

    It showed up:

    Jun 25 22:22:19 pfSense0 kernel: swap_pager: out of swap space
    Jun 25 22:22:19 pfSense0 kernel: swap_pager_getswapspace(16): failed
    ….
    Jun 25 22:31:10 pfSense0 snort[50004]: FATAL ERROR: fpcreate.c(1555) Failed to compile port group patterns.

    Any inputs appreciated!

    zenny:

    Something is corrupt in the snortglobals part of config.xml for the WAN interface.  If you feel comfortable with editing an XML file, here's what I would suggest.

    1.  Backup the existing config.xml (the configuration) by using the tool under Diagnostics…Backup/Restore.  Save the file off somewhere not on the firewall.  If you don't mind, I would like a copy of that file to see what might be wrong with it.  I will PM you my e-mail address.  Looking into the file will hopefully help me find the problem and see if any additional data scrubbing or validation might be needed in the code to prevent it from happening to others.

    2.  First try deleting the Snort WAN interface.  Go to Snort Interfaces, check the checkbox beside the WAN interface, and click the (x) icon to remove it.

    3.  Add the interface back and start over with the WAN configuration in Snort.

    If you have already tried steps #2 and #3 above and it still is broken, then proceed to #4 below.

    4.  Edit the config.xml file on the firewall by going to Diagnostics…Edit File and navigating to /conf/config.xml.

    5.  Scroll down towards the bottom of the file until you find the section <snortglobal>.

    6.  Within that section, there will be multiple <rule>sub-sections.  Each will be delimited by <rule>at the start and</rule> at the end.  Find the sub-section for the WAN interface.  There is an <interface>element tag in the file with the interface name specified.

    7.  Delete the entire <rule>to</rule> section for the WAN.  Be careful that you choose the correct start and end tags and do not go past them.  It's easiest to stay scrolled all the way to the left-hand margin and just use SHIFT+PAGE DOWN to select entire lines.  Delete the highlighted section.

    8.  Save the file, then go to the Snort Interfaces tab and add the WAN back.

    Note that unless the config.xml file is severely messed up, simply deleting the interface from the Snort GUI (steps #2 and #3 above) should clean it up without needing to resort to the manual edit.  It you mess it up, just restore the saved configuration.

    Bill</interface></rule></snortglobal>



  • @bmeeks:

    @zenny:

    I thought that it has to do with the bge driver that Snort is not running on WAN as Bill suspected, and therefore, swapped the em driver running on LAN with the bge running on WAN. After the swapping, even with WAN with em (which is working in LAN) stopped to work and LAN with bge driver starts Snort (whereas WAN with bge was not working). Therefore, there is something that prevents WAN to run snort.

    The only thing that was displayed on console was:

    swap_pager: out of swap space
    swap_pager_getswapspace(16):failed

    And when I checked system.log

    It showed up:

    Jun 25 22:22:19 pfSense0 kernel: swap_pager: out of swap space
    Jun 25 22:22:19 pfSense0 kernel: swap_pager_getswapspace(16): failed
    ….
    Jun 25 22:31:10 pfSense0 snort[50004]: FATAL ERROR: fpcreate.c(1555) Failed to compile port group patterns.

    Any inputs appreciated!

    zenny:

    Something is corrupt in the snortglobals part of config.xml for the WAN interface.  If you feel comfortable with editing an XML file, here's what I would suggest.

    1.  Backup the existing config.xml (the configuration) by using the tool under Diagnostics…Backup/Restore.  Save the file off somewhere not on the firewall.  If you don't mind, I would like a copy of that file to see what might be wrong with it.  I will PM you my e-mail address.  Looking into the file will hopefully help me find the problem and see if any additional data scrubbing or validation might be needed in the code to prevent it from happening to others.

    2.  First try deleting the Snort WAN interface.  Go to Snort Interfaces, check the checkbox beside the WAN interface, and click the (x) icon to remove it.

    3.  Add the interface back and start over with the WAN configuration in Snort.

    If you have already tried steps #2 and #3 above and it still is broken, then proceed to #4 below.

    4.  Edit the config.xml file on the firewall by going to Diagnostics…Edit File and navigating to /conf/config.xml.

    5.  Scroll down towards the bottom of the file until you find the section <snortglobal>.

    6.  Within that section, there will be multiple <rule>sub-sections.  Each will be delimited by <rule>at the start and</rule> at the end.  Find the sub-section for the WAN interface.  There is an <interface>element tag in the file with the interface name specified.

    7.  Delete the entire <rule>to</rule> section for the WAN.  Be careful that you choose the correct start and end tags and do not go past them.  It's easiest to stay scrolled all the way to the left-hand margin and just use SHIFT+PAGE DOWN to select entire lines.  Delete the highlighted section.

    8.  Save the file, then go to the Snort Interfaces tab and add the WAN back.

    Note that unless the config.xml file is severely messed up, simply deleting the interface from the Snort GUI (steps #2 and #3 above) should clean it up without needing to resort to the manual edit.  It you mess it up, just restore the saved configuration.

    Bill</interface></rule></snortglobal>

    Tried with step 4-8, didn't work!

    Then tried step 2-3, yet didn't work!

    Sending the config.xml to your email as per your PM!

    Thanks again!



  • @zenny:

    Tried with step 4-8, didn't work!

    Then tried step 2-3, yet didn't work!

    Sending the config.xml to your email as per your PM!

    Thanks again!

    Received your file and will take a look.  This one has me puzzled for sure!  Cleaning out the file should have fixed it.

    Bill



  • @bmeeks:

    @zenny:

    Tried with step 4-8, didn't work!

    Then tried step 2-3, yet didn't work!

    Sending the config.xml to your email as per your PM!

    Thanks again!

    Received your file and will take a look.  This one has me puzzled for sure!  Cleaning out the file should have fixed it.

    Bill

    Nope even cleaning out the file didn't help! Anyway thanks! It could be something messy with either 2.1, don't know.

    UPDATE: The culprit seems to be the 'IPS Policy' option which should not be. After I disabled IPS Policy and manually selected the ET and Snort rules, snort on WAN worked fine.



  • @zenny:

    Nope even cleaning out the file didn't help! Anyway thanks! It could be something messy with either 2.1, don't know.

    UPDATE: The culprit seems to be the 'IPS Policy' option which should not be. After I disabled IPS Policy and manually selected the ET and Snort rules, snort on WAN worked fine.

    Thanks for the clue.  Which policy were you selecting (Connectivity, Balanced or Security), and how much physical RAM is in your firewall?  Snort can chew through RAM, especially with lots of enabled rules.  I believe Supermule had an issue with running out of swap space with a large rule set on a firewall with either 1 GB or 2 GB of RAM in a virtual machine on VMware.  He bumped up the RAM and the problem fixed itself.  When the OS starts swapping to disk and even runs out of virtual RAM there, things go weird pretty quickly after that.

    Bill


  • Banned

    I had to upgrade the VM's to 4GB for it not to swap and run out of memory. It can easily be an issue there.



  • @bmeeks:

    @zenny:

    Nope even cleaning out the file didn't help! Anyway thanks! It could be something messy with either 2.1, don't know.

    UPDATE: The culprit seems to be the 'IPS Policy' option which should not be. After I disabled IPS Policy and manually selected the ET and Snort rules, snort on WAN worked fine.

    Thanks for the clue.  Which policy were you selecting (Connectivity, Balanced or Security), and how much physical RAM is in your firewall?  Snort can chew through RAM, especially with lots of enabled rules.  I believe Supermule had an issue with running out of swap space with a large rule set on a firewall with either 1 GB or 2 GB of RAM in a virtual machine on VMware.  He bumped up the RAM and the problem fixed itself.  When the OS starts swapping to disk and even runs out of virtual RAM there, things go weird pretty quickly after that.

    Bill

    Whatever policy I slecect among Connectivity, Balanced or Security, it does not work. The machine has 1.5GB of physical RAM. However, I cannot confirm whether it is a RAM issue or pfSense issue.


  • Banned

    2GB was not enough for me. Try disabling a lot of rules and use the policy again.



  • @Supermule:

    2GB was not enough for me. Try disabling a lot of rules and use the policy again.

    Thanks for the tip. There is no way to upgrade the RAM at the moment because the machine (Compaq Deskpro SFF dc7100sff with dual core P4 machine with 3.2Ghz processors) uses old PC3200 SDRAM which is pretty expensive compared to new DDR3 RAM. Worth upgrading the machine instead. ;-)

    Let me know if anyone knows of cheap vendors who has at least 4GB of PC3200 SDRAM so that I can upgrade. ;-)

    Actually I need 8 of these: http://www.newegg.com/Product/Product.aspx?Item=N82E16820141317 but it is out of stock. :-(


  • Banned



  • @Supermule:

    http://www.amazon.com/s/ref=nb_sb_noss_1/190-7687956-8103958?url=search-alias%3Daps&field-keywords=pc3200&sprefix=pc3200%2Caps&rh=i%3Aaps%2Ck%3Apc3200

    Thanks supermule for the link. But HP uses DDR Synch Dram PC3200 UNBUFFERED memory like Kingston's KTH-D530/1G. The motherboard seems very choosy about memories as I read at http://h30499.www3.hp.com/t5/Business-PCs-Compaq-Elite-Pro/DC7100-SDRAM-upgrade-2-x-1GB-appears-as-2-x-512MB/td-p/1152268, fyi.

    I have two machines HP Compaq Deskpro dc7600 sff (http://h10010.www1.hp.com/wwpc/ca/en/sm/WF06b/12132708-12132884-12132884-12132884-12221730-12221860-77102439.html?dnr=1) besides this one. I guess the same KTH-D530/1G applies to this one, too.

    Sorry, it sounds a hardware thread, but is the foundation to pfSense working ;-)



  • @zenny:

    Sorry, it sounds a hardware thread, but is the foundation to pfSense working ;-)

    Hi Zenny:

    I replied to your e-mail as well with essentiall the same info as this post.  1.5 GB of RAM is just not enough for Snort with a lot of rules on multiple interfaces.  I also noticed in the config.xml you sent me that a number of packages such as AV, SquidGuard, pfBlocker and others were also installed along with Snort.  If all those packages fire up, a 1.5 GB of RAM box is going to be pretty stressed.  As you are getting "out of swap space" errors, that leaves no doubt that the box is running out of physical RAM and even exhausting the virtual RAM in swap.

    I would recommend at least a 4GB RAM box.  Newegg in the USA did have a nice little 1U ASUS barebones server chassis for $279 US.  You would have to provide a CPU and RAM, so that would up the total cost.  There are also some Intel Atom-based servers made by Supermicro at Newegg.  Those start at $379 if I remember correctly.

    Bill



  • There must be something wrong with my system then  ;), because I have sensors on WAN, LAN and WLAN. LAN and WLAN are set on IPS-balanced and WAN has some ET rulesets. I have 2 GB Ram and system is using almost 50% of the installed memory. But I don't have other demanding packages installed.



  • @gogol:

    There must be something wrong with my system then  ;), because I have sensors on WAN, LAN and WLAN. LAN and WLAN are set on IPS-balanced and WAN has some ET rulesets. I have 2 GB Ram and system is using almost 50% of the installed memory. But I don't have other demanding packages installed.

    Zenny had a large number of other packages installed such as pfBlocker, Unbound, Squidguard, Sarg, spamd, HAVP, Squid, Varnish3, tinc, and a few others.  That much stuff along with Snort and a lot of enabled rules won't mix well with 1.5 GB of RAM. I  heard back that he is upgrading the firewall to 4 GB of RAM.  Hopefully that will do the trick.

    Bill



  • FYI

    The following being enabled kept the WAN interface from turning on.

    It will show enabled, but it will have that red X next to it and it refused to start.

    After troubleshooting, I narrowed it down to this very specific rule (which you need to add to your exception list)

    2011695 ET WEB_CLIENT Possible Microsoft Internet Explorer Dynamic Object Tag/URLMON Sniffing Cross Domain Information Disclosure Attempt Disclosure Attempt

    If I disable that, then the WAN interface is able to show the green play button (running) without issue.


Log in to reply