NAT Logs

[michmoor]

@mcury Ok yeah thats the way to go BUT....i have to use NetFlow as netflow already has bytes parsed in the output while IPFix does not.

Still figuring out how to normalize the nf_bytes column to megabytes

[mcury]

@michmoor said in NAT Logs:

Still figuring out how to normalize the nf_bytes column to megabytes

Try this pipeline rule:

rule "arithmetic"
when
has_field("octetDeltaCount")
then
let size_kb = to_double($message.octetDeltaCount);
set_field("size_bytes", size_kb / to_double(value:"1048576"));
end

This will give you something like this:

[michmoor]

@mcury Not sure if its working to be honest.
There are no errors showing up in the pipeline so messages are being processed.

Message:

[mcury]

@michmoor said in NAT Logs:

@mcury Not sure if its working to be honest.
There are no errors showing up in the pipeline so messages are being processed.

Did you edit the pipeline connections and assigned to the ipfix stream ?

[michmoor]

@mcury
Im on Graylog 6.0.5

Yes my pipeline is connected to my stream. Im using Netflow data just for testing.

[mcury]

@michmoor I'm on Graylog 6.1.5, but it should work on 6.0.5, no problems with that.

Is Configurations tab, Manage processors, is pipeline active ?

[michmoor]

@mcury
Yep its active.
I have other pipelines in effect so this is all working. The new pipeline for Netflow is the latest addition. I guess I'm trying to figure out how to prove its working
With my other pipelines, I'm extracting fields into Grok Patterns so i can tell immediately if its working.

Considering i see no errors in the pipeline i have to imagine its working.

[mcury]

@michmoor said in NAT Logs:

Considering i see no errors in the pipeline i have to imagine its working.

But do you see messages being processed in the Throughput ?

[michmoor]

@mcury

[mcury]

@michmoor said in NAT Logs:

@mcury

Now, check if the field exists:

[michmoor]

@mcury

It does. I feel like we are getting close to a confirmation. LOL

Is my pipeline rule correct?

[mcury]

@michmoor said in NAT Logs:

It does. I feel like we are getting close to a confirmation. LOL

So, the rule is processing the data, and the field exists.
It must be showing up in the logs..

@michmoor said in NAT Logs:

Is my pipeline rule correct?

That is an old rule, I'm using now octetDeltaCount, replace the rule with the one I posted above.

After that, go to rule simulator, JSON, paste the example below, then click in run rule simulator, and check if the field size_bytes will appear

{
  "destinationTransportPort": 3246,
  "size_bytes": 0,
  "gl2_remote_ip": "192.168.255.249",
  "gl2_remote_port": 20815,
  "sourceIPv4Address": "192.168.255.249",
  "source": "192.168.255.249",
  "ipClassOfService": 0,
  "gl2_source_input": "6627f43d99aaec416300cd0e",
  "hostname_src": "pfsense.home.arpa.",
  "egressInterface": 1,
  "octetDeltaCount": 168,
  "gl2_source_node": "0f929def-42b9-4ea6-8980-799a62f7bdb3",
  "sourceTransportPort": 3246,
  "flowEndMilliseconds": "2024-04-23T20:21:02.098Z",
  "timestamp": "2024-04-23T20:21:24.000Z",
  "destinationIPv4Address": "192.168.255.253",
  "gl2_accounted_message_size": 575,
  "streams": [
    "63fe3ab48b6393126ef3c2f7"
  ],
  "gl2_message_id": "01HW69XJD000004NFV0XGMQ27B",
  "message": "Ipfix [192.168.255.249]:3246 <> [192.168.255.253]:3246 proto:1 pkts:2 bytes:168",
  "ingressInterface": 1,
  "packetDeltaCount": 2,
  "protocolIdentifier": 1,
  "hostname_dst": "rpi4.home.arpa.",
  "_id": "0ea147d7-01af-11ef-8e3e-dca632a54719",
  "flowStartMilliseconds": "2024-04-23T20:21:01.096Z"
}

[michmoor]

@mcury Rule simulation returns without a problem

So I'm a bit confused as to how you figured out there was a field called "size_bytes" as that does not come up for me when i review the available fields. In fact, unlike NetFlow which has a field called "nf_bytes" there is nothing similar in IPFix within Graylog.

[mcury]

@michmoor said in NAT Logs:

@mcury Rule simulation returns without a problem

So I'm a bit confused as to how you figured out there was a field called "size_bytes" as that does not come up for me when i review the available fields. In fact, unlike NetFlow which has a field called "nf_bytes" there is nothing similar in IPFix within Graylog.

I'm using IPFIX UDP, the rule "arithmetic" above reflects that.

The size_bytes field is the one we are creating with the pipeline.
I'm transforming the field octetDeltaCount, which is in bytes, to size_bytes , which is in megabytes.

[michmoor]

@mcury You're the man.
I corrected some things looking at your example and i got it working.

One last question, how did you get DNS lookups as part of your set up?

edit:

I have updated some Dashboards to reflect the new data points IPFix provides.
@stephenw10 pflow definitely helped out in this but....would be nice to just have straight-up logging within pfsense just saying

[mcury]

@michmoor said in NAT Logs:

One last question, how did you get DNS lookups as part of your set up?

There are a few steps, hope I remember all of them in this post (22 o´clock here already).
1- First, you need a reverse zone in your DNS.

• This reverse zone can be dynamically updated or not, you choose. I opted by having static IP leases and creating A and PTR records for each host individually, but you don't need to do it.
  2- Import these extractors to any input you want to use them:

{
  "extractors": [
    {
      "title": "hostname_src",
      "extractor_type": "lookup_table",
      "converters": [],
      "order": 12,
      "cursor_strategy": "copy",
      "source_field": "sourceIPv4Address",
      "target_field": "hostname_src",
      "extractor_config": {
        "lookup_table_name": "hostname"
      },
      "condition_type": "regex",
      "condition_value": "192.168.255.25|192.168.10."
    },
    {
      "title": "hostname_dst",
      "extractor_type": "lookup_table",
      "converters": [],
      "order": 13,
      "cursor_strategy": "copy",
      "source_field": "destinationIPv4Address",
      "target_field": "hostname_dst",
      "extractor_config": {
        "lookup_table_name": "hostname"
      },
      "condition_type": "regex",
      "condition_value": "192.168.255.25|192.168.10."
    }
  ],
  "version": "6.1.5"
}

Or, through the GUI:

The condition value was added to allow only local addresses to be resolved, but if you want the entire world to be resolved, just remove that.
3- Go to lookup tables in Graylog, create an adapter, a cache and a lookup table, point it your DNS server.

That alone should do it, if you have any doubts just ask, but I will only answer tomorrow, 22:30 here already.

[michmoor]

@mcury Got it working.

Ever thought about creating a blog and posting this? At the very least this is extremely useful and should be preserved

[mcury]

@michmoor said in NAT Logs:

@mcury Got it working.

Ever thought about creating a blog and posting this? At the very least this is extremely useful and should be preserved

I liked the idea Perhaps in a near future, have to organize a few things here and there before

[michmoor]

@mcury Had to update the JVM Heap size. Looks like IPFix widget i made was killing my VM.

[mcury]

@michmoor said in NAT Logs:

Had to update the JVM Heap size. Looks like IPFix widget i made was killing my VM.

DNS data can be overwhelming.
I had to increase the JVM heap size and configure zswap with zmalloc and zstd to avoid crashes.
Then, checked every rule that was logging, or tracking, and optimized that also.
Created a bunch of no log rules, changed to 5 days of live data only, older indices are closed but I can open them anytime I need.