Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Grafana Dashboard using Telegraf with additional plugins

    pfSense Packages
    28
    173
    59.7k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • V
      VictorRobellini @SeaMonkey
      last edited by

      @seamonkey said in Grafana Dashboard using Telegraf with additional plugins:

      The solution was to add an override

      What version of Grafana are you using? Check out the Github page to see compatible versions

      S 1 Reply Last reply Reply Quote 0
      • V
        VictorRobellini @wrightsonm
        last edited by VictorRobellini

        @wrightsonm

        This is as granular as you can really get with the data collected from pfblocker. You could go deeper, but I don't think it would provide any real value. The blank borders around the DNSBL stuff really bug me!

        cef9c2f2-0885-4efc-80ab-35b1bff0a1ab-image.png

        W 1 Reply Last reply Reply Quote 0
        • W
          wrightsonm @VictorRobellini
          last edited by

          @victorrobellini

          Nice work there. I think you have a typo on one of the charts on the left hand side

          IP-Top 10 Blocked - OUT (By Host/Protocol) should read IN.

          You could probably layout the objects in a different way to end up with less black space in the bottom left /right. Hard to visualise off the top of my head. You might find the layout might work better in columns. i.e. IP - Top 10 Block - * is 3 items in a row. perhaps pivoting it all to be vertical may work better?

          To use up the free space, one thing that could be useful is a variation on the blocked packet stats graph. You could create stacked bar charts grouped by ip. This would give a visual indication of the proportion of traffic that is coming from one of your internal devices that is being blocked for outbound, and for inbound would visually highlight when a particular ip is attacking you.

          What do you think to my suggestion above about hyperlinking the public ip's to greynoise.io? Or another alternative? Maybe could put the url fragment in a variable so people can pick their favourite ip analysis website?

          1 Reply Last reply Reply Quote 1
          • S
            SeaMonkey @VictorRobellini
            last edited by

            @victorrobellini 7.1.1, but I wouldn't think that would make a difference regarding my particular issue. Your design applies the value mapping function to all fields within the panel, but instead it should only be applied to the Status column exclusively under the Override settings. I've fixed this on my own dashboard and I started to attempt a pull request to fix it, but didn't have the patience to manually adjust only the relevant lines in your JSON file and reimport for testing.

            S 1 Reply Last reply Reply Quote 0
            • V
              VictorRobellini
              last edited by

              I made a few updates to the plugins, telegraf config, and the dashboard.

              More gateway details
              More pfBlocker details (in a separate panel so you can hide it if you don't care)

              Screenshot to be updated

              W 1 Reply Last reply Reply Quote 3
              • W
                wrightsonm @VictorRobellini
                last edited by

                @victorrobellini exciting. I look forward to the screenshot / git commit. 😀

                V 1 Reply Last reply Reply Quote 0
                • V
                  VictorRobellini @wrightsonm
                  last edited by VictorRobellini

                  @wrightsonm

                  Screenshot updated. The only thing I need to figure out is what to put in the pfSense Details blank space. Adding graphs for the sake of filling space means unnecessary influx queries and CPU usage on the host. I'm running my influx on a J5005 with a bunch of other services. I'll deal with the blank space for now.

                  I added the extra pfBlocker details to its own panel so it can be collapsed if it's not used.

                  1 Reply Last reply Reply Quote 0
                  • W
                    wrightsonm
                    last edited by

                    @victorrobellini i'll pull down the latest update this eve and take a look. (thanks in advance) I have a few more cpu cycles available in my setup:

                    NUC10i7FNK -> proxmox -> debian vm -> docker -> influxdb

                    Latest update on the ntopng experiment. It is up and running. Had to customise the startup script such that I could create custom applications in the GUI. Bit of a pain that a key feature isn't supported OOTB.

                    I've noticed that Network Discovery gets itself confused and assigns device names incorrectly based on mdns. Haven't found a way to turn this off yet. For example, sometimes ntopng thinks my firewall network interface is called "google home mini", which is clearly incorrect. It might be related to running avahi / pimd to get chromecast running across vlans. More investigations to follow.

                    Haven't tried exporting ntopng to influxdb yet.
                    I'm in 2 minds to skip over ntopng and investigate elastiflow instead.

                    W 1 Reply Last reply Reply Quote 0
                    • W
                      wrightsonm @wrightsonm
                      last edited by

                      @VictorRobellini I've pulled down the latest version of the conf, plugins and dashboard and I have noticed 1 thing that has stopped working.

                      This is the old gateway RTT dashboard. Also note the Gateway list:
                      702c58c8-4a08-45f1-8c6c-ca31775a4a06-image.png

                      This is the new dashboard. The gateway names have changed and RTT chart is no longer working. Does it work for you?
                      5f91b997-a192-4fd8-a5f3-a4bf9f35df9f-image.png

                      When running telegraf --test I think this is the relevant snippet after changing the dashboard and plugins to the new version:

                      > gateways,host=fw,interface=igb0 defaultgw=1,delay=2.117,gwdescr="Interface WAN_DHCP Gateway",loss=0,monitor="192.168.0.1",source="192.168.0.30",status="online",stddev=3.885,substatus="none" 1619643426000000000
> gateways,host=fw,interface=igb0 defaultgw=0,delay=0,gwdescr="Interface WAN_DHCP6 Gateway",loss=0,monitor="",source="",status="",stddev=0,substatus="" 1619643426000000000
> gateways,host=fw,interface=igb0 defaultgw=1,delay=2.117,gwdescr="Interface WAN_DHCP Gateway",loss=0,monitor="192.168.0.1",source="192.168.0.30",status="online",stddev=3.885,substatus="none" 1619643426000000000
> gateways,host=fw,interface=igb0 defaultgw=0,delay=0,gwdescr="Interface WAN_DHCP6 Gateway",loss=0,monitor="",source="",status="",stddev=0,substatus="N/A" 1619643426000000000
                      V 1 Reply Last reply Reply Quote 0
                      • V
                        VictorRobellini @wrightsonm
                        last edited by

                        @wrightsonm
                        Did you drop your gateways measurement? I eliminated unused tags in the influx data. I added a blurb in the Readme about things not rendering properly.

                        1 Reply Last reply Reply Quote 0
                        • J
                          jpetovello
                          last edited by

                          Great looking dashboard!

                          Wondering if there are step by step instructions on how to go about installing and configuring, as I'm finding the github instructions rather lacking.

                          V 1 Reply Last reply Reply Quote 0
                          • W
                            wrightsonm
                            last edited by

                            Ok so steps i've performed so far. I think there are bugs in the latest git repository.

                            docker exec -it influxdb /bin/sh

influx delete --bucket pfsense --start '1970-01-01T00:00:00Z' --stop $(date +"%Y-%m-%dT%H:%M:%SZ")  --predicate '_measurement="tail_dnsbl_log"'

influx delete --bucket pfsense --start '1970-01-01T00:00:00Z' --stop $(date +"%Y-%m-%dT%H:%M:%SZ")  --predicate '_measurement="tail_ip_block_log"'

influx delete --bucket pfsense --start '1970-01-01T00:00:00Z' --stop $(date +"%Y-%m-%dT%H:%M:%SZ")  --predicate '_measurement="gateways"' --org-id [id]

influx delete --bucket pfsense --start '1970-01-01T00:00:00Z' --stop $(date +"%Y-%m-%dT%H:%M:%SZ")  --predicate '_measurement="interface"' --org-id [id]

                            If your WAN config looks like mine with ipv4 and ipv6 enabled, then you end up with a problem with the gateways telegraf lines as igb0 exists twice. Gateways used to use gateway_name probably for this reason. It means that the tag grouping won't work if you have ivp4 and ipv6 enabled.

                            So i've had to change telegraph_gateways.php to include gateway_name and then adjust the grafana dashboard to use gateway_name instead of interface. (basically reverting it back to how it worked on the previous revision.

                            Next issue is telegraf is nolonger exporting the interface measurement.

                            Looks like this changeset replaced "interface" with "gateways". Looks like a copy paste issue to me.
                            github diff

                            As a result of this problem, none of the interfaces show up correctly on the dashboards.
                            The interface summaries work with the old version of telegraf_pfinterface.php

                            Final note, is this dashboard now requires a 1440p monitor to view everything without horizontal scrolling. Would be nice if it would display on 1080p monitors

                            W V 2 Replies Last reply Reply Quote 0
                            • W
                              wrightsonm @wrightsonm
                              last edited by wrightsonm

                              @wrightsonm
                              This is the dashboard working once again for me:

                              30209c7e-4ed6-405a-8d7f-f3db254d2e0c-image.png

                              telegraf --test --config /usr/local/etc/telegraf.conf

                              Grafana Gateway Variable changed back to:

                              SHOW TAG VALUES FROM "gateways" WITH KEY = "gateway_name" WHERE "host" =~ /^$Host$/

                              Modified plugins:
                              telegraf_gateways.php.txt

                              Reverted plugin:
                              telegraf_pfinterface.php.txt

                              Modified dashboard:
                              grafana.json.txt

                              1 Reply Last reply Reply Quote 0
                              • V
                                VictorRobellini
                                last edited by

                                @wrightsonm said in Grafana Dashboard using Telegraf with additional plugins:

                                with ipv4 and ipv6 enabled, then you end up with a problem with the gateways telegraf lines as igb0 exists twice

                                Does this happen for the interface table? If you are willing to troubleshoot this with me, please open a github issue so we don't flood this thread with troubleshooting.

                                W 1 Reply Last reply Reply Quote 0
                                • V
                                  VictorRobellini @jpetovello
                                  last edited by

                                  @jpetovello This is a pet project I built for use with my homelab and documented to help others. The main prereqs are that you have influx and grafana already set up. I didn't document that since everyone is going to have their systems set up differently. Luckily, there's no shortage of tutorials available online. My recommendation is to get your influxdb and grafana setup, read the github readme, read through this thread, and search the closed issues in the github project.

                                  1 Reply Last reply Reply Quote 0
                                  • W
                                    wrightsonm @VictorRobellini
                                    last edited by

                                    @victorrobellini yeah sure. I've not got any time until Tuesday now. The solution is in the attachments and pictures above. I can raise a Pull Request with the fixes on Tuesday if you like?

                                    V 1 Reply Last reply Reply Quote 0
                                    • V
                                      VictorRobellini @wrightsonm
                                      last edited by VictorRobellini

                                      @wrightsonm I just reread what you wrote. I have no idea how that happened. Thanks for the heads up. I may just merge the 2 scripts since they are doing very similar stuff. I'll take a look this weekend.

                                      Update: I merged the scripts and updated the repo. Everything should be working now.

                                      1 Reply Last reply Reply Quote 0
                                      • V
                                        VictorRobellini @wrightsonm
                                        last edited by

                                        @wrightsonm said in Grafana Dashboard using Telegraf with additional plugins:

                                        Final note, is this dashboard now requires a 1440p monitor to view everything without horizontal scrolling.

                                        Sorry, I run this on either a 4k or an ultrawide 1080p display. If you have it reformatted for 1080p just add the json with a new name and submit a PR.

                                        1 Reply Last reply Reply Quote 0
                                        • von PapstV
                                          von Papst
                                          last edited by

                                          I copied the new scripts for plugin and configured permissions on them (755), imported new dashboard but now I'm not getting any temperatures reading, IP or DNSBL lists, no network interface summary, no Gateway RTT or loss. Any hint what I missed in the upgrade process?

                                          von PapstV 1 Reply Last reply Reply Quote 0
                                          • von PapstV
                                            von Papst @von Papst
                                            last edited by

                                            @von-papst Got temperatures working (CRLF from M$ messed up script). Still missing lists from pfblocker-ng and interface summary.

                                            W 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post