Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    netflow and graylog

    Scheduled Pinned Locked Moved General pfSense Questions
    19 Posts 2 Posters 2.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mcury @mcury
      last edited by

      This widget will show you the last 2 hours summary (top talkers).
      Outbound flows only.

      c5123e08-dad2-4816-956f-49016b88dc86-image.png

      nf_bytes:>1048576 AND _exists_:hostname_src AND NOT _exists_:hostname_dst AND NOT (nf_dst:172.16.200.* OR nf_src:172.16.200.*)
      

      The filter above will make sure that only hosts that are resolved will be showed.
      I'm not allowing hosts from my MGMT vlan to get their data in the widget (172.16.200.*).

      Widget in details:
      74da6a1d-f0d6-4320-8b44-223f11c4f822-image.png

      dead on arrival, nowhere to be found.

      M 1 Reply Last reply Reply Quote 0
      • M
        mcury @mcury
        last edited by mcury

        Now, for intervlan traffic I have:

        This widget will show you the last 5 days summary (top talkers).

        93ba3eeb-dd15-453a-bfce-9f4344ca7d75-image.png

        As you can see below, this filter is only getting connections from my LAN (192.168.255.2* to my WIFI network 192.168.10* and vice versa. And it will only report resolved hosts:

        ((nf_src:192.168.10.* AND nf_dst:192.168.255.2*) OR (nf_src:192.168.255.2* AND nf_dst:192.168.10.*)) AND (_exists_:hostname_src AND _exists_:hostname_dst)
        

        9fb14a35-e5e5-48ca-ae7d-bd579e874e34-image.png

        Edit: Note here that I'm using size_bytes2 !!

        dead on arrival, nowhere to be found.

        M 1 Reply Last reply Reply Quote 0
        • M
          mcury @mcury
          last edited by

          Now, for intervlan traffic I also have:

          This widget will show you the last 2 hours summary (top talkers).

          dcc5c48a-6dd6-4f48-8eca-0688ed92d39f-image.png

          ((nf_src:192.168.255.2* AND nf_dst:192.168.10.*) OR (nf_src:192.168.10.* AND nf_dst:192.168.255.2*)) AND (_exists_:hostname_src AND _exists_:hostname_dst)
          

          b54c3ed2-4d0c-4647-9846-ccbf67cfb10a-image.png

          Edit: Note here that I'm using size_bytes2 !!

          dead on arrival, nowhere to be found.

          M 1 Reply Last reply Reply Quote 0
          • M
            mcury @mcury
            last edited by mcury

            Make sure Pipeline is the latest thing in your message processors:

            6817890d-e2f3-40af-99fd-086adac3bc29-image.png

            Edit: For netflow, this is what I have, everything else is using syslog only.

            Edit2: Make sure netflow version is set 9 in Softflowd, and flow tracking is Full.

            dead on arrival, nowhere to be found.

            1 Reply Last reply Reply Quote 0
            • M
              michmoor LAYER 8 Rebel Alliance
              last edited by michmoor

              edit1: Having an even more fundamental problem. Netflow collection is no longer working for me in graylog. Receiving no messages.
              All other syslog messages are working........hmmmm
              Is softflowd working for you after the upgrade to 23.05.1

              edit2: A good restart of the input on graylog got things going. Pipelines getting messages

              On a basic level i think i got something wrong.
              I got the lookup tables configured - using DNS

              The pipelines arent seeing any messages but i know the stream is working.

              e4316e00-2f75-41b6-98e1-c527fdbbf0c2-image.png

              Firewall: NetGate,Palo Alto-VM,Juniper SRX
              Routing: Juniper, Arista, Cisco
              Switching: Juniper, Arista, Cisco
              Wireless: Unifi, Aruba IAP
              JNCIP,CCNP Enterprise

              M 1 Reply Last reply Reply Quote 0
              • M
                michmoor LAYER 8 Rebel Alliance @michmoor
                last edited by

                @mcury hope your back from your beer run !
                I got my lookup table working but i dont have the fields you do in your examples. e.g. Hostname

                939fbc2d-09a1-4e89-90ce-000ae854e88c-image.png

                Firewall: NetGate,Palo Alto-VM,Juniper SRX
                Routing: Juniper, Arista, Cisco
                Switching: Juniper, Arista, Cisco
                Wireless: Unifi, Aruba IAP
                JNCIP,CCNP Enterprise

                M 1 Reply Last reply Reply Quote 0
                • M
                  mcury @michmoor
                  last edited by

                  @michmoor hmm, I forgot to mention the extractors I had to create for netflow lookup to work.
                  Follows below:

                  183b7953-c31c-4777-b5a1-4f06a99fe4b9-image.png

                  {
                    "extractors": [
                      {
                        "title": "hostname_src",
                        "extractor_type": "lookup_table",
                        "converters": [],
                        "order": 0,
                        "cursor_strategy": "copy",
                        "source_field": "nf_src_address",
                        "target_field": "hostname_src",
                        "extractor_config": {
                          "lookup_table_name": "hostname"
                        },
                        "condition_type": "regex",
                        "condition_value": "192.168."
                      },
                      {
                        "title": "hostname_dst",
                        "extractor_type": "lookup_table",
                        "converters": [],
                        "order": 0,
                        "cursor_strategy": "copy",
                        "source_field": "nf_dst_address",
                        "target_field": "hostname_dst",
                        "extractor_config": {
                          "lookup_table_name": "hostname"
                        },
                        "condition_type": "regex",
                        "condition_value": "192.168."
                      }
                    ],
                    "version": "5.1.2"
                  }
                  

                  dead on arrival, nowhere to be found.

                  M 1 Reply Last reply Reply Quote 0
                  • M
                    michmoor LAYER 8 Rebel Alliance @mcury
                    last edited by michmoor

                    @mcury

                    Re: netflow and graylog

                    Last few days have been a crash course in GROK and creating my pipeline rules. I thought of this thread and im ready to return to it. Now that i understand GrayLog pipelines way more than i did back last year i can safely say this is pretty easy to get going.
                    I just dont know how to enrich data using dns for IP lookups but thats ok

                    Thanks @mcury

                    edit: The graylog v5.2 has a rule creator which is wayyyyy easier than writing the code which in turn makes managing those pipeline rules even easier. Highly recommended to upgrade.

                    Firewall: NetGate,Palo Alto-VM,Juniper SRX
                    Routing: Juniper, Arista, Cisco
                    Switching: Juniper, Arista, Cisco
                    Wireless: Unifi, Aruba IAP
                    JNCIP,CCNP Enterprise

                    M 1 Reply Last reply Reply Quote 1
                    • M
                      michmoor LAYER 8 Rebel Alliance @michmoor
                      last edited by

                      Few things ive been able to do with GROK parsing is not to clean up my unbound log files and create fields that are important to me and good for tracking.

                      34aa20ba-7c37-4812-ad77-1a0306cb7107-image.png

                      Firewall: NetGate,Palo Alto-VM,Juniper SRX
                      Routing: Juniper, Arista, Cisco
                      Switching: Juniper, Arista, Cisco
                      Wireless: Unifi, Aruba IAP
                      JNCIP,CCNP Enterprise

                      M 1 Reply Last reply Reply Quote 1
                      • M
                        mcury @michmoor
                        last edited by mcury

                        @michmoor said in netflow and graylog:

                        Few things ive been able to do with GROK parsing is not to clean up my unbound log files and create fields that are important to me and good for tracking.

                        I'm running Graylog 5.2 now, had to build gcc 11.1.0+ from source, it took a few hours in my raspberry pi 4 but it is working :)

                        @michmoor said in netflow and graylog:

                        I just dont know how to enrich data using dns for IP lookups but thats ok

                        I'm using PTR for that purpose, if there is something I can help, just let me know.

                        @michmoor said in netflow and graylog:

                        Few things ive been able to do with GROK parsing is not to clean up my unbound log files and create fields that are important to me and good for tracking.

                        Ow, that is really nice :) If it is possible, can you share how you are getting those statistics from Unbound ?

                        dead on arrival, nowhere to be found.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.