Grafana Dashboard using Telegraf with additional plugins

erbalo

@ab5g said in Grafana Dashboard using Telegraf with additional plugins:

@erbalo

Getting Mbps on the graph is easy.

Select the WAN Panel > Edit > Panel Settings > Axis / Left Y - Change this to bits/sec
Also change the query to add *8

So your query will look like this

SELECT derivative(mean("bytes_recv"), 1s) *8 FROM "net" WHERE ("host" =~ /^$Host$/ AND "interface" =~ /^$WAN$/) AND $timeFilter GROUP BY time($__interval) fill(null)

Do the same for the LAN Panel (the first one). Then click 'LAN Interfaces' (the master container for all LAN panels - see pic) . Click Settings > Repeat for Datasouce and this Mbps setting applies to all LAN panels.

Thank you, for th WAN it is now correct just i am seeing there Mb/s - Megabit per second.
But for the LAN i need to see MB/s = Megabyte per second. I mean i need to see the exact value such as when you copy a file from your NAS, you see there the rate MB/s.

masterhead

This post is deleted!

bigjohns97

I noticed recently that the systems row is no longer tracking the time range that is selected up at the top right.

Is this something that should be defined in the query or did something break with an upgrade somewhere along the way?

Can someone else confirm if they are able to see changes in the systems row statistics whenever the change the time frame in the upper right hand corner of the dashboard?

I did go ahead and pull down the latest json from github just to be sure that wasn't the problem.

AB5G

@bigjohns97 Seems to be working for me.

bigjohns97

@ab5g I did recently upgrade pretty much everything (pfsense 12, pfblockerng 3.007 grafana. Etc)

Just a heads up. I haven't had a chance to dig into the data to see if the dates aren't being kept or what really has changed.

AB5G

@bigjohns97 I am on 11.3 with pfBlockerng at 3.0.0_7. If you recently updated to 12, make sure that the Telegraf additional configs are in place and your scripts are still in usr/local/bin with the correct permissions.

bigjohns97

@ab5g yep that is all confirmed working and I was sure it was working after the upgrade to 12.x and 3.x so I am leaning towards new version of grafana or possibly a combination of the two.

bigjohns97

Well I kinda found out what happened, I believe something went haywire during one of the many upgrades and ended up putting some log entries without time or dates into my dnsbl.log which in turn was put into my influxdb.

I ended up having to delete the dnsbl.log and let pfblockerng automatically create it and then I deleted the database from within influx and everything was working again.

There was nothing wrong with grafana.

pfsencue

I'm seeing multiple widgets for each of my WAN items:

Screen Shot 2021-01-11 at 11.36.27 AM.png

I also had to manually add the additional code via the command line to the telelgraf.conf file manually as when I add it via the browser it adds the ^M characters. I'm on a Mac so that shouldn't happen, anyone else see this?

VictorRobellini

@ab5g This is a great idea! I'm adding it to my next commit.

Thanks

AB5G

@victorrobellini

I also have telegram integration with Grafana setup as per here - https://gist.github.com/ilap/cb6d512694c3e4f2427f85e4caec8ad7

I use this to receive notifications if my WAN and Wireguard tunnels go down. Screenshot 2021-01-12 at 10.34.43 AM.png

Screenshot 2021-01-12 at 10.34.04 AM.png

Screenshot 2021-01-12 at 10.33.45 AM.png

Screenshot 2021-01-12 at 10.33.28 AM.png

pfsencue

@pfsencue ok i just grabbed the one from the grafana website you created @VictorRobellini and it works great from there... looking forward to that update above!

bigjohns97

I was playing with the telegraph_unbound script and noticed it wasn't working and ended up replacing the command in the script with the following

unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total.num

This provides a cache hit stats output and was wondering if anyone had any skills with grafana to provide a nice panel addition to this already great dashboard. Was looking for something like this.

alt text

I found this thread from reddit where they were discussing how to optimize cache hits and couldn't figure out how to get the following panel to show.

If anyone has the knowledge on how to create this panel I would greatly appreciate it!

(I tried to post the original thread from reddit but it kept flagging this post as spam)

bigjohns97

@bigjohns97 said in Grafana Dashboard using Telegraf with additional plugins:

I was playing with the telegraph_unbound script and noticed it wasn't working and ended up replacing the command in the script with the following
unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total.num
This provides a cache hit stats output and was wondering if anyone had any skills with grafana to provide a nice panel addition to this already great dashboard. Was looking for something like this.

I found this thread from reddit where they were discussing how to optimize cache hits and couldn't figure out how to get the following panel to show.

If anyone has the knowledge on how to create this panel I would greatly appreciate it!

(I tried to post the original thread from reddit but it kept flagging this post as spam)

Was able to reach out to the originator of this post from reddit and get a working version of this

SELECT "total_num_cachehits" FROM "unbound" WHERE ("host" = 'pfSense.localdomain') AND $timeFilter

SELECT "total_num_cachemiss" FROM "unbound" WHERE ("host" = 'pfSense.localdomain') AND $timeFilter

{
  "aliasColors": {
    "Hits": "#629e51",
    "Misses": "#bf1b00"
  },
  "breakPoint": "50%",
  "cacheTimeout": null,
  "combine": {
    "label": "Others",
    "threshold": 0
  },
  "datasource": "$dataSource",
  "decimals": null,
  "fieldConfig": {
    "defaults": {
      "custom": {}
    },
    "overrides": []
  },
  "fontSize": "100%",
  "format": "short",
  "gridPos": {
    "h": 6,
    "w": 3,
    "x": 13,
    "y": 7
  },
  "hideTimeOverride": false,
  "id": 23763571993,
  "interval": null,
  "legend": {
    "header": "",
    "percentage": true,
    "percentageDecimals": 0,
    "show": true,
    "sortDesc": true,
    "values": false
  },
  "legendType": "On graph",
  "links": [],
  "maxDataPoints": 3,
  "nullPointMode": "connected",
  "pieType": "donut",
  "pluginVersion": "6.3.3",
  "strokeWidth": "2",
  "targets": [
    {
      "alias": "Hits",
      "groupBy": [],
      "measurement": "unbound",
      "orderByTime": "ASC",
      "policy": "default",
      "refId": "A",
      "resultFormat": "time_series",
      "select": [
        [
          {
            "params": [
              "total_num_cachehits"
            ],
            "type": "field"
          }
        ]
      ],
      "tags": [
        {
          "key": "host",
          "operator": "=~",
          "value": "/^$Host$/"
        }
      ]
    },
    {
      "alias": "Misses",
      "groupBy": [],
      "measurement": "unbound",
      "orderByTime": "ASC",
      "policy": "default",
      "refId": "B",
      "resultFormat": "time_series",
      "select": [
        [
          {
            "params": [
              "total_num_cachemiss"
            ],
            "type": "field"
          }
        ]
      ],
      "tags": [
        {
          "key": "host",
          "operator": "=~",
          "value": "/^$Host$/"
        }
      ]
    }
  ],
  "thresholds": [],
  "timeFrom": null,
  "timeShift": null,
  "title": "DNS Cache Hit/Miss Ratio",
  "type": "grafana-piechart-panel",
  "valueName": "current"
}

Here is my current stats (I set min ttl to 3600 in unbound)

EDIT : I forgot to mention you have to use this command in the telegraf_unbound.sh and make sure you uncomment it from the telegraf config in the install instructions

unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total.num

von Papst

I'm not sure where do I install these plugins and telegraf config. On my pfsense or on my Linux box where I have telegraf and influxdb installed?

bigjohns97

@von-papst said in Grafana Dashboard using Telegraf with additional plugins:

I'm not sure where do I install these plugins and telegraf config. On my pfsense or on my Linux box where I have telegraf and influxdb installed?

Telegraf should be on your pfsense box and it should send to influxdb on your linux box, you shouldn't need Telegraf on your linux box for anything.

erbalo

@bigjohns97 said in Grafana Dashboard using Telegraf with additional plugins:

@bigjohns97 said in Grafana Dashboard using Telegraf with additional plugins:
I was playing with the telegraph_unbound script and noticed it wasn't working and ended up replacing the command in the script with the following
unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total.num
This provides a cache hit stats output and was wondering if anyone had any skills with grafana to provide a nice panel addition to this already great dashboard. Was looking for something like this.

I found this thread from reddit where they were discussing how to optimize cache hits and couldn't figure out how to get the following panel to show.

If anyone has the knowledge on how to create this panel I would greatly appreciate it!

(I tried to post the original thread from reddit but it kept flagging this post as spam)
Was able to reach out to the originator of this post from reddit and get a working version of this
SELECT "total_num_cachehits" FROM "unbound" WHERE ("host" = 'pfSense.localdomain') AND $timeFilter
SELECT "total_num_cachemiss" FROM "unbound" WHERE ("host" = 'pfSense.localdomain') AND $timeFilter
{
  "aliasColors": {
    "Hits": "#629e51",
    "Misses": "#bf1b00"
  },
  "breakPoint": "50%",
  "cacheTimeout": null,
  "combine": {
    "label": "Others",
    "threshold": 0
  },
  "datasource": "PfSense",
  "decimals": null,
  "fieldConfig": {
    "defaults": {
      "custom": {}
    },
    "overrides": []
  },
  "fontSize": "100%",
  "format": "short",
  "gridPos": {
    "h": 6,
    "w": 3,
    "x": 13,
    "y": 7
  },
  "hideTimeOverride": false,
  "id": 23763571993,
  "interval": null,
  "legend": {
    "header": "",
    "percentage": true,
    "percentageDecimals": 0,
    "show": true,
    "sortDesc": true,
    "values": false
  },
  "legendType": "On graph",
  "links": [],
  "maxDataPoints": 3,
  "nullPointMode": "connected",
  "pieType": "donut",
  "pluginVersion": "6.3.3",
  "strokeWidth": "2",
  "targets": [
    {
      "alias": "Hits",
      "groupBy": [],
      "measurement": "unbound",
      "orderByTime": "ASC",
      "policy": "default",
      "refId": "A",
      "resultFormat": "time_series",
      "select": [
        [
          {
            "params": [
              "total_num_cachehits"
            ],
            "type": "field"
          }
        ]
      ],
      "tags": [
        {
          "key": "host",
          "operator": "=",
          "value": "pfSense.localdomain"
        }
      ]
    },
    {
      "alias": "Misses",
      "groupBy": [],
      "measurement": "unbound",
      "orderByTime": "ASC",
      "policy": "default",
      "refId": "B",
      "resultFormat": "time_series",
      "select": [
        [
          {
            "params": [
              "total_num_cachemiss"
            ],
            "type": "field"
          }
        ]
      ],
      "tags": [
        {
          "key": "host",
          "operator": "=",
          "value": "pfSense.localdomain"
        }
      ]
    }
  ],
  "thresholds": [],
  "timeFrom": null,
  "timeShift": null,
  "title": "DNS Cache Hit/Miss Ratio",
  "type": "grafana-piechart-panel",
  "valueName": "current"
}
Here is my current stats (I set min ttl to 3600 in unbound)

EDIT : I forgot to mention you have to use this command in the telegraf_unbound.sh and make sure you uncomment it from the telegraf config in the install instructions
unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total.num

i am interesting on this graph. What can you do/analyze really with this graph DNS cache hit?

Do i only uncomend the unbound part on telegraf on pfsense and put the unbound-control.. to the telegraf.sh ?

#[[inputs.unbound]]

server = "127.0.0.1:953"

binary = "/usr/local/bin/telegraf_unbound.sh"

What should be the Server IP adres? is that my pfsense box?

bigjohns97

@erbalo said in Grafana Dashboard using Telegraf with additional plugins:

@bigjohns97 said in Grafana Dashboard using Telegraf with additional plugins:
@bigjohns97 said in Grafana Dashboard using Telegraf with additional plugins:
I was playing with the telegraph_unbound script and noticed it wasn't working and ended up replacing the command in the script with the following
unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total.num
This provides a cache hit stats output and was wondering if anyone had any skills with grafana to provide a nice panel addition to this already great dashboard. Was looking for something like this.

I found this thread from reddit where they were discussing how to optimize cache hits and couldn't figure out how to get the following panel to show.

If anyone has the knowledge on how to create this panel I would greatly appreciate it!

(I tried to post the original thread from reddit but it kept flagging this post as spam)
Was able to reach out to the originator of this post from reddit and get a working version of this
SELECT "total_num_cachehits" FROM "unbound" WHERE ("host" = 'pfSense.localdomain') AND $timeFilter
SELECT "total_num_cachemiss" FROM "unbound" WHERE ("host" = 'pfSense.localdomain') AND $timeFilter
{
  "aliasColors": {
    "Hits": "#629e51",
    "Misses": "#bf1b00"
  },
  "breakPoint": "50%",
  "cacheTimeout": null,
  "combine": {
    "label": "Others",
    "threshold": 0
  },
  "datasource": "PfSense",
  "decimals": null,
  "fieldConfig": {
    "defaults": {
      "custom": {}
    },
    "overrides": []
  },
  "fontSize": "100%",
  "format": "short",
  "gridPos": {
    "h": 6,
    "w": 3,
    "x": 13,
    "y": 7
  },
  "hideTimeOverride": false,
  "id": 23763571993,
  "interval": null,
  "legend": {
    "header": "",
    "percentage": true,
    "percentageDecimals": 0,
    "show": true,
    "sortDesc": true,
    "values": false
  },
  "legendType": "On graph",
  "links": [],
  "maxDataPoints": 3,
  "nullPointMode": "connected",
  "pieType": "donut",
  "pluginVersion": "6.3.3",
  "strokeWidth": "2",
  "targets": [
    {
      "alias": "Hits",
      "groupBy": [],
      "measurement": "unbound",
      "orderByTime": "ASC",
      "policy": "default",
      "refId": "A",
      "resultFormat": "time_series",
      "select": [
        [
          {
            "params": [
              "total_num_cachehits"
            ],
            "type": "field"
          }
        ]
      ],
      "tags": [
        {
          "key": "host",
          "operator": "=",
          "value": "pfSense.localdomain"
        }
      ]
    },
    {
      "alias": "Misses",
      "groupBy": [],
      "measurement": "unbound",
      "orderByTime": "ASC",
      "policy": "default",
      "refId": "B",
      "resultFormat": "time_series",
      "select": [
        [
          {
            "params": [
              "total_num_cachemiss"
            ],
            "type": "field"
          }
        ]
      ],
      "tags": [
        {
          "key": "host",
          "operator": "=",
          "value": "pfSense.localdomain"
        }
      ]
    }
  ],
  "thresholds": [],
  "timeFrom": null,
  "timeShift": null,
  "title": "DNS Cache Hit/Miss Ratio",
  "type": "grafana-piechart-panel",
  "valueName": "current"
}
Here is my current stats (I set min ttl to 3600 in unbound)

EDIT : I forgot to mention you have to use this command in the telegraf_unbound.sh and make sure you uncomment it from the telegraf config in the install instructions
unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total.num
i am interesting on this graph. What can you do/analyze really with this graph DNS cache hit?

Do i only uncomend the unbound part on telegraf on pfsense and put the unbound-control.. to the telegraf.sh ?

#[[inputs.unbound]]

server = "127.0.0.1:953"

binary = "/usr/local/bin/telegraf_unbound.sh"

What should be the Server IP adres? is that my pfsense box?

Correct, uncomment out the part you posted and then go and edit the sh script with the "unbound-control" command I posted above.

As far as what purpose this serves, for me this helped me get to a point where the majority of my dns requests were cached and being responded to locally by pfsense, this greatly speeds up internet experience.

A lot of IOT devices today use very small TTLs which means pfsense is constantly sending out DNS traffic to DNS servers on the internet for domain queries. With a min TTL of 3600 I am telling unbound to always use at least 3600 as a minimum ttl. This sets a low watermark for all domains but allows domains above 3600 to keep their designed TTL as well.

As you can see it gives me a very high cache rate, and I have done this since way back when I was running pihole and have never seen any issues with it but you will hear from quite a few who believe this is a bad practice because it goes against what the domain owner intended. I have always felt that domain owners have to account for the lowest common denominator however so for me a 3600 min TTL is a nice middle ground between performance and proper name resolution.

erbalo

@bigjohns97

Thank you, which IP should i put here and late the port number 953 ?

server = "127.0.0.1:953"

erbalo

Should that be ok so?

#!/bin/sh
/usr/local/sbin/unbound-control -c /var/unbound/unbound.conf $* | grep -vE 'thread[0-9]+'
unbound-control -c /var/unbound/unbound.conf stats_noreset | grep total.num