Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PfSense syslog and ELSA

    Scheduled Pinned Locked Moved General pfSense Questions
    45 Posts 11 Posters 15.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • BBcan177B
      BBcan177 Moderator
      last edited by

      Has anyone had any success getting pfSense Syslogs into ELSA

      https://code.google.com/p/enterprise-log-search-and-archive/

      I found this article below but I cant get the script to create the file.

      http://www.securitygrit.com/2013/03/pfsense-into-elsa.html

      "Experience is something you don't get until just after you need it."

      Website: http://pfBlockerNG.com
      Twitter: @BBcan177  #pfBlockerNG
      Reddit: https://www.reddit.com/r/pfBlockerNG/new/

      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        If you're on 2.1, add this patch:
        http://files.pfsense.org/jimp/patches/pf-log-oneline-option.diff

        And then check the box on the system log settings to force the firewall logs to one line.

        If you're on 2.0.x, use this patch instead:
        http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.0.x.diff

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • BBcan177B
          BBcan177 Moderator
          last edited by

          Thank You,

          Are there instruction on installing this patch?

          "Experience is something you don't get until just after you need it."

          Website: http://pfBlockerNG.com
          Twitter: @BBcan177  #pfBlockerNG
          Reddit: https://www.reddit.com/r/pfBlockerNG/new/

          1 Reply Last reply Reply Quote 0
          • jimpJ
            jimp Rebel Alliance Developer Netgate
            last edited by

            Use the System Patches package.

            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

            Need help fast? Netgate Global Support!

            Do not Chat/PM for help!

            1 Reply Last reply Reply Quote 0
            • BBcan177B
              BBcan177 Moderator
              last edited by

              Thanks Jimp,

              "Experience is something you don't get until just after you need it."

              Website: http://pfBlockerNG.com
              Twitter: @BBcan177  #pfBlockerNG
              Reddit: https://www.reddit.com/r/pfBlockerNG/new/

              1 Reply Last reply Reply Quote 0
              • BBcan177B
                BBcan177 Moderator
                last edited by

                Hi Jimp,

                I have updated pfSense with the patch and I am forwarding the logs to an ELSA server (Also tested with Kiwi Syslog) and I am noticing the following:

                Here is the output from pfSense for a "Block" in the Firewall log :

                01-05-2014 20:08:26 Local0.Info xxx.xxx.xxx.xxx Jan  5 20:08:27 pf: 00:00:00.570566 rule 5/0(match): block in on rl0: (tos 0x0, ttl 114, id 37724, offset 0, flags [none], proto TCP (6), length 48)  185.25.184.184.28796 > xxx.xxx.xxx.xxx.5900: Flags, cksum 0x8635 (correct), seq 979907292, win 65535, options [mss 1460,nop,nop,sackOK], length 0

                Is there any way to format this to show the the name of the Rule ie "pfBlocker ET" instead of rule 5/0(match).

                Elsa treats this as a single entry and can't pivot on any of the individual contents, ie Src/Dst IP address, or port. I am also investigating if the ELSA parser can decipher this better.

                Thanks.

                "Experience is something you don't get until just after you need it."

                Website: http://pfBlockerNG.com
                Twitter: @BBcan177  #pfBlockerNG
                Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                1 Reply Last reply Reply Quote 0
                • jimpJ
                  jimp Rebel Alliance Developer Netgate
                  last edited by

                  There isn't a way to get the rule description there.

                  Parsing might be a bit tricky but ELSA will need to know how to parse the pf or pfSense logs.

                  Not directly relevant, but in 2.2 we're completely reworking the raw filter logs to be in a nice, easily to parse CSV-like format.

                  Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                  Need help fast? Netgate Global Support!

                  Do not Chat/PM for help!

                  1 Reply Last reply Reply Quote 0
                  • BBcan177B
                    BBcan177 Moderator
                    last edited by

                    Thanks Jimp,

                    I will try to see if ELSA can parse the file better.

                    Looking forward to 2.2.

                    Thanks

                    "Experience is something you don't get until just after you need it."

                    Website: http://pfBlockerNG.com
                    Twitter: @BBcan177  #pfBlockerNG
                    Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                    1 Reply Last reply Reply Quote 0
                    • BBcan177B
                      BBcan177 Moderator
                      last edited by

                      When I modify a Firewall:Rule and hit "apply", I notice that all of the previous log entries in the pfSense Firewall log displays incorrect "Rule Names"?

                      All new entries in the Log will show the correct Rule name, but the older logs remain indicating an incorrect rule name?

                      Has anyone seen this issue before? I have tested this on two different boxes with the same issue.

                      I am using pfBlocker with aliases on pfSense v2.1.

                      I am also using a patch (http://files.pfsense.org/jimp/patches/pf-log-oneline-option.diff) to allow one line entries for a remote syslog.

                      Thanks

                      "Experience is something you don't get until just after you need it."

                      Website: http://pfBlockerNG.com
                      Twitter: @BBcan177  #pfBlockerNG
                      Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                      1 Reply Last reply Reply Quote 0
                      • jimpJ
                        jimp Rebel Alliance Developer Netgate
                        last edited by

                        @BBcan17:

                        When I modify a Firewall:Rule and hit "apply", I notice that all of the previous log entries in the pfSense Firewall log displays incorrect "Rule Names"?

                        All new entries in the Log will show the correct Rule name, but the older logs remain indicating an incorrect rule name?

                        That is normal. When the rules reload, the rule numbers can change and the descriptions no longer match up. We're working on a way around that for 2.2 as well.

                        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                        Need help fast? Netgate Global Support!

                        Do not Chat/PM for help!

                        1 Reply Last reply Reply Quote 0
                        • BBcan177B
                          BBcan177 Moderator
                          last edited by

                          Thanks Jim,

                          I look forward to the next version.

                          "Experience is something you don't get until just after you need it."

                          Website: http://pfBlockerNG.com
                          Twitter: @BBcan177  #pfBlockerNG
                          Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                          1 Reply Last reply Reply Quote 0
                          • F
                            fsom
                            last edited by

                            Hello,

                            I'm using pcSense 2.1 with Elsa and your patches and it's working fine for TCP and UDP packages. However, ICMP doesn't get parsed. I was trying to locate the problem the last days but without luck. If pfSense is sending an ICMP log to Elsa, I receive this in my log:

                            00:00:02.504973 rule 317/0(match): block in on bge1: (tos 0x0, ttl 64, id 315, offset 0, flags [none], proto ICMP (1), length 84) 172.27.0.36 > 172.27.1.3: ICMP echo request, id 44331, seq 0, length 64
                            host=172.27.1.3 program=pf class=NONE

                            I was looking at the parsing rules and run some tests by hand, it always get parsed. I then send the same message via netcat:

                            echo '<14>Jan 22 10:57:48 pf: 00:00:02.504973 rule 317/0(match): block in on bge1: (tos 0x0, ttl 64, id 315, offset 0, flags [none], proto ICMP (1), length 84) 172.27.0.36 > 172.27.1.3: ICMP echo request, id 44331, seq 0, length 64' | nc -v -u -w 0 172.27.50.77 514

                            and suddenly it gets parsed:

                            00:00:02.504973 rule 317/0(match): block in on bge1: (tos 0x0, ttl 64, id 315, offset 0, flags [none], proto ICMP (1), length 84) 172.27.0.36 > 172.27.1.3: ICMP echo request, id 44331, seq 0, length 64
                            host=172.27.0.162 program=pf class=FIREWALL_ACCESS_DENY proto=ICMP srcip=172.27.0.36 srcport=0 dstip=172.27.1.3 dstport=0 o_int=bge1 i_int= access_group=

                            Does anyone else has this problem?
                            Thanks in advance,
                            fsom

                            1 Reply Last reply Reply Quote 0
                            • BBcan177B
                              BBcan177 Moderator
                              last edited by

                              What are you using as the search Query in ELSA? Did you look at the Google ELSA group  https://groups.google.com/forum/#!forum/enterprise-log-search-and-archive

                              Try the following example Querys -

                              class=FIREWALL_CONNECTION_END groupby:host
                              class=FIREWALL_ACCESS_DENY groupby:host
                              class=FIREWALL_ACCESS_DENY icmp or tcp or udp dstip="x.x.x.x" groupby:srcip

                              Are you using a parser similar to this?

                              http://www.securitygrit.com/2013/03/pfsense-into-elsa.html

                              I have added the ruleset to the end of patterndb.xml between <patterndb></patterndb>and ran "service syslog-ng restart"

                              The syslogs are going into the existing database ELSA class/fields "FIREWALL_ACCESS_LOG" and "FIREWALL_CONNECTION_END".

                              Make sure you Select "Grid Display" to see it parsed properly.

                              "Experience is something you don't get until just after you need it."

                              Website: http://pfBlockerNG.com
                              Twitter: @BBcan177  #pfBlockerNG
                              Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                              1 Reply Last reply Reply Quote 0
                              • F
                                fsom
                                last edited by

                                Hello,
                                Thank you for your reply. ICMP packages coming from pfSense are going into the CLASS=NONE group, TCP/UDP into the FIREWALL_*. I tried your example queries but the only ICMP package I found was the one I spoofed via netcat. All the other ICMP packages are in the CLASS=none group. The parser I'm using is from http://www.securitygrit.com/2013/03/pfsense-into-elsa.html, before I added it, everything from pfSense was going into CLASS=NONE, after I applied and restarted syslogd they got assigned to the FIREWALL_ACCESS_DENY or FIREWALL_CONNECTION_END class. The thing I don't understand is why ICMP packages aren't going into these classes when coming from pfSense.

                                The parsing itself is working:
                                [root@elsa ~]# /usr/local/syslog-ng/bin/pdbtool match -p /usr/local/elsa/node/conf/patterndb.xml -P pf -M "00:03:54.505077 rule 50/0(match): block in on bge0: (tos 0x0, ttl 107, id 25728, offset 0, flags [none], proto ICMP (1), length 28) 172.16.250.251 > xxx.yyy.21.150: ICMP echo request, id 768, seq 51812, length 8"
                                MESSAGE=00:03:54.505077 rule 50/0(match): block in on bge0: (tos 0x0, ttl 107, id 25728, offset 0, flags [none], proto ICMP (1), length 28) 172.16.250.251 > 194.180.21.150: ICMP echo request, id 768, seq 51812, length 8
                                PROGRAM=pf
                                .classifier.class=2
                                .classifier.rule_id=2
                                s0=bge0
                                i0=ICMP
                                i1=172.16.250.251
                                i3=xxx.yyy.21.150

                                Does your ELSA handle ICMP like UDP/TCP ?

                                Thanks,
                                fsom

                                1 Reply Last reply Reply Quote 0
                                • C
                                  Cino
                                  last edited by

                                  I guess I've been sleeping, been waiting a long time for FW logs to be parse on a single line. Thanks Jim for the patch!! I'll have to give this a try or just wait for 2.2  ;)

                                  1 Reply Last reply Reply Quote 0
                                  • BBcan177B
                                    BBcan177 Moderator
                                    last edited by

                                    @fsom:

                                    ICMP packages coming from pfSense are going into the CLASS=NONE group

                                    I tested an icmp block rule and can confirm that ELSA is logging this as "class=none" instead of "class=FIREWALL_ACCESS_DENY"

                                    You should post a request to the Google ELSA group.

                                    "Experience is something you don't get until just after you need it."

                                    Website: http://pfBlockerNG.com
                                    Twitter: @BBcan177  #pfBlockerNG
                                    Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                    1 Reply Last reply Reply Quote 0
                                    • BBcan177B
                                      BBcan177 Moderator
                                      last edited by

                                      Hi Jimp,

                                      I have followed the following document -

                                      https://doc.pfsense.org/index.php/Why_can%27t_I_query_SNMP,_use_syslog,_NTP,_or_other_services_initiated_by_the_firewall_itself_over_IPsec_VPN%3F

                                      And it was working for about a month or so, but recently any Ubuntu machine that is in a Site-Site Ipsec tunnel can not get any connectivity when this Manual route is implemented. All other traffic from Windows based machines seems to be unaffected by this, Including the Syslogs from pfSense being directed to a remote syslog server.

                                      I have tried this on two different tunnels with different Ubuntu machines.

                                      When I delete the manual route and restart apinger, it doesn't clear the issue. A full reboot of the pfSense box seems to fix it.

                                      Do you have any suggestions?

                                      Thanks for your help.

                                      "Experience is something you don't get until just after you need it."

                                      Website: http://pfBlockerNG.com
                                      Twitter: @BBcan177  #pfBlockerNG
                                      Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                      1 Reply Last reply Reply Quote 0
                                      • jimpJ
                                        jimp Rebel Alliance Developer Netgate
                                        last edited by

                                        Ditch the route, update to 2.1.1, then Status > System Logs, Settings tab, pick LAN for the source address. :-)

                                        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                        Need help fast? Netgate Global Support!

                                        Do not Chat/PM for help!

                                        1 Reply Last reply Reply Quote 0
                                        • BBcan177B
                                          BBcan177 Moderator
                                          last edited by

                                          Is 2.1.1 stable enough now? If there are no other choices, i guess I will test it out.

                                          This is the reason why they invented "Alcohol!!"

                                          "Experience is something you don't get until just after you need it."

                                          Website: http://pfBlockerNG.com
                                          Twitter: @BBcan177  #pfBlockerNG
                                          Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                          1 Reply Last reply Reply Quote 0
                                          • jimpJ
                                            jimp Rebel Alliance Developer Netgate
                                            last edited by

                                            It's still got a couple issues yet but it may be good enough for most uses.

                                            Otherwise track down the commit(s) for the syslog source address selection and apply them manually.

                                            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                            Need help fast? Netgate Global Support!

                                            Do not Chat/PM for help!

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.