netflow and graylog

michmoor

@mcury Hey !
Following up on the other thread, can you help out and send over the netflow dashboard config for Graylog!

mcury

@michmoor Hey, let me try to put it all here in a way that isn't too long and easy to understand.
Assuming that you have Mongod, Java and Graylog installed, and working.

But first, I need to give credits to Lawrence System, who first showed me this tool.
Great channel in Youtube: [https://www.youtube.com/@lawrencesystems
Visit his forum: https://forums.lawrencesystems.com/

I'll just focus on the Graylog configuration here, assuming that you are already exporting everything to it.

In the step below, you can configure how long you want to keep the logs, be careful here because depending on how much logs you will be sending to Graylog, more RAM you will need.

Some useful links I used to start my journey:

Graylog: https://docs.graylog.org/docs/ubuntu
Graylog SSL/HTTPS https://docs.graylog.org/docs/https
Extractor URL: https://github.com/loganmarchione/Graylog_Extractors_pfSense
Lawrence System: https://www.youtube.com/watch?v=rtfj6W5X0YA
Note that for the Extractor, there are other options out there that you might prefer.
Lawrence's System Extractor: https://github.com/lawrencesystems

Now, I'll proceed with the configuration in the next posts,

Edit: Just give me a few minutes because I'm out of beers, I'll have to go out to buy more

michmoor

@mcury said in netflow and graylog:

ust focus on the Graylog configuration here, assuming that

Oh yeah im fully utilizing graylog for all my logs. Even got syslog alerting to my email.
Just the netflow piece is where im stuck on

mcury

@michmoor said in netflow and graylog:

Just the netflow piece is where im stuck on

Oh, great.

Let me focus on the netflow part them:

I'm using pipelines since Graylog use bytes and I want to see data in megabytes.

Pipeline setup:

Note that I'm using two pipelines, this is necessary due to inter-vlan connections.

Bytes_to_megabytes transforms bytes in megabytes.
Bytes_to_megabytes_2 transforms bytes in megabytes /2.

When connection comes from one VLAN to the other, pfSense sends netflow data from both VLANs, and because of that, Graylog will report it doubled.
To circumvent that, I use the pipeline Bytes_to_megabytes_2 to get the data in MB divided by two.

In pipelines, Manage rules, Bytes_to_megabytes :

rule "arithmetic"
when
has_field("nf_bytes")
then
let size_kb = to_long($message.nf_bytes);
set_field("size_bytes", size_kb / 1048576);
end

In pipelines, Manage rules, Bytes_to_megabytes_2 :

rule "arithmetic2"
when
has_field("nf_bytes")
then
let size_kb = to_long($message.nf_bytes);
set_field("size_bytes2", (size_kb / 1048576) / 2);
end

I'll get more into Pipeline later, when I'll speak about creating the widgets.

Please let me know if you want help to configure lookup tables in Graylog

Edit: I'm exporting netflow from interfaces LAN and WIFI.
So , connections from LAN to WAN, uses bytes_to_megabytes
Connections from WIFI to WAN, uses bytes_to_megabytes.
Connections from WIFI to LAN and vice versa (intervlan), uses bytes_to_megabytes_2

michmoor

@mcury Ok this is what i got so far

mcury

To configure lookup tables, I used this guide: https://go2docs.graylog.org/5-1/making_sense_of_your_log_data/lookup_tables.html

Graylog is getting DNS from my AD (samba-ad), so, just internal hosts are being resolved, you can filter what networks will be resolved.

It is pretty straight forward, just follow the guide and it will work.

Now, next post, widgets:

mcury

This widget will show you the last 5 days summary (top talkers).

nf_bytes:>1048576 AND _exists_:hostname_dst AND NOT _exists_:hostname_src AND NOT (nf_dst:172.16.200.* OR nf_src:172.16.200.*)

The filter above will make sure that only hosts that are resolved will be showed.
I'm not allowing hosts from my MGMT vlan to get their data in the widget (172.16.200.*).

Widget in details:

mcury

This widget will show you the last 2 hours summary (top talkers).
Inbound flows only.

nf_bytes:>1048576 AND _exists_:hostname_dst AND NOT _exists_:hostname_src AND NOT (nf_dst:172.16.200.* OR nf_src:172.16.200.*)

The filter above will make sure that only hosts that are resolved will be showed.
I'm not allowing hosts from my MGMT vlan to get their data in the widget (172.16.200.*).

mcury

This widget will show you the last 5 days summary (top talkers).
Outbound flows only.

nf_bytes:>1048576 AND _exists_:hostname_src AND NOT _exists_:hostname_dst AND NOT (nf_dst:172.16.200.* OR nf_src:172.16.200.*)

The filter above will make sure that only hosts that are resolved will be showed.
I'm not allowing hosts from my MGMT vlan to get their data in the widget (172.16.200.*).

Widget in details:

mcury

This widget will show you the last 2 hours summary (top talkers).
Outbound flows only.

nf_bytes:>1048576 AND _exists_:hostname_src AND NOT _exists_:hostname_dst AND NOT (nf_dst:172.16.200.* OR nf_src:172.16.200.*)

The filter above will make sure that only hosts that are resolved will be showed.
I'm not allowing hosts from my MGMT vlan to get their data in the widget (172.16.200.*).

Widget in details:

mcury

Now, for intervlan traffic I have:

This widget will show you the last 5 days summary (top talkers).

As you can see below, this filter is only getting connections from my LAN (192.168.255.2* to my WIFI network 192.168.10* and vice versa. And it will only report resolved hosts:

((nf_src:192.168.10.* AND nf_dst:192.168.255.2*) OR (nf_src:192.168.255.2* AND nf_dst:192.168.10.*)) AND (_exists_:hostname_src AND _exists_:hostname_dst)

Edit: Note here that I'm using size_bytes2 !!

mcury

Now, for intervlan traffic I also have:

This widget will show you the last 2 hours summary (top talkers).

((nf_src:192.168.255.2* AND nf_dst:192.168.10.*) OR (nf_src:192.168.10.* AND nf_dst:192.168.255.2*)) AND (_exists_:hostname_src AND _exists_:hostname_dst)

Edit: Note here that I'm using size_bytes2 !!

mcury

Make sure Pipeline is the latest thing in your message processors:

Edit: For netflow, this is what I have, everything else is using syslog only.

Edit2: Make sure netflow version is set 9 in Softflowd, and flow tracking is Full.

michmoor

edit1: Having an even more fundamental problem. Netflow collection is no longer working for me in graylog. Receiving no messages.
All other syslog messages are working........hmmmm
Is softflowd working for you after the upgrade to 23.05.1

edit2: A good restart of the input on graylog got things going. Pipelines getting messages

On a basic level i think i got something wrong.
I got the lookup tables configured - using DNS

The pipelines arent seeing any messages but i know the stream is working.

michmoor

@mcury hope your back from your beer run !
I got my lookup table working but i dont have the fields you do in your examples. e.g. Hostname

mcury

@michmoor hmm, I forgot to mention the extractors I had to create for netflow lookup to work.
Follows below:

{
  "extractors": [
    {
      "title": "hostname_src",
      "extractor_type": "lookup_table",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "nf_src_address",
      "target_field": "hostname_src",
      "extractor_config": {
        "lookup_table_name": "hostname"
      },
      "condition_type": "regex",
      "condition_value": "192.168."
    },
    {
      "title": "hostname_dst",
      "extractor_type": "lookup_table",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "nf_dst_address",
      "target_field": "hostname_dst",
      "extractor_config": {
        "lookup_table_name": "hostname"
      },
      "condition_type": "regex",
      "condition_value": "192.168."
    }
  ],
  "version": "5.1.2"
}

michmoor

@mcury

Re: netflow and graylog

Last few days have been a crash course in GROK and creating my pipeline rules. I thought of this thread and im ready to return to it. Now that i understand GrayLog pipelines way more than i did back last year i can safely say this is pretty easy to get going.
I just dont know how to enrich data using dns for IP lookups but thats ok

Thanks @mcury

edit: The graylog v5.2 has a rule creator which is wayyyyy easier than writing the code which in turn makes managing those pipeline rules even easier. Highly recommended to upgrade.

michmoor

Few things ive been able to do with GROK parsing is not to clean up my unbound log files and create fields that are important to me and good for tracking.

mcury

@michmoor said in netflow and graylog:

Few things ive been able to do with GROK parsing is not to clean up my unbound log files and create fields that are important to me and good for tracking.

I'm running Graylog 5.2 now, had to build gcc 11.1.0+ from source, it took a few hours in my raspberry pi 4 but it is working :)

@michmoor said in netflow and graylog:

I just dont know how to enrich data using dns for IP lookups but thats ok

I'm using PTR for that purpose, if there is something I can help, just let me know.

@michmoor said in netflow and graylog:

Few things ive been able to do with GROK parsing is not to clean up my unbound log files and create fields that are important to me and good for tracking.

Ow, that is really nice :) If it is possible, can you share how you are getting those statistics from Unbound ?