Telegraf Module Source Code?

xanaro

Is there a git repository somewhere for the Telegraf module source code?

I have a Grafana dashboard based on this: https://github.com/VictorRobellini/pfSense-Dashboard

The problem is that if you pick a large time frame, it crawls to its knees, absolutely horrible performance.

I would like to see how difficult it would be to add PostgreSQL as a Telegraf output.

I was reading over this article and it seems for larger datasets PostgreSQL blows influxdb out of the water in all areas except storage space:

https://portavita.github.io/2018-07-31-blog_influxdb_vs_postgresql

EDIT: found it I think.

this:
https://github.com/pfsense/FreeBSD-ports/tree/devel/net-mgmt/telegraf

or this:
https://github.com/pfsense/FreeBSD-ports/tree/devel/net-mgmt/pfSense-pkg-Telegraf

xanaro

@xanaro Seems I do not need to reinvent the wheel.... somebody may have already done exactly what im looking to do: https://blog.timescale.com/blog/introducing-the-postgresql-timescaledb-output-plugin-for-telegraf/

So maybe it is just a matter of getting telegraf updated to a later version or adding the additional telegraf plugin to pfsense?

eeeek from reading this timescale blog it seems they submitted a pull request testing and all but it never got merged.... however the timescale people compiled the telegraf plugin including support for postgresql and timescaledb, so I am wondering if I can swap out the telegraf binary used in pfsense for the telegraf binary built with postgresql and timescaledb support.

bmeeks

Packages used on pfSense generally consist of two distinct parts.

There is usually a binary component (https://github.com/pfsense/FreeBSD-ports/tree/devel/net-mgmt/telegraf in this case) that does the bulk of the actual work. Binary components come from the FreeBSD Ports tree, but they must be compiled for the same underlying FreeBSD version as pfSense in order to work properly. Right now that version is FreeBSD 12.2/STABLE.

The second piece of most pfSense packages is a GUI component written in PHP that the user sees and interacts with in order to configure and start/stop the underlying binary piece. For telegraf, the GUI component is https://github.com/pfsense/FreeBSD-ports/tree/devel/net-mgmt/pfSense-pkg-Telegraf. Anytime you see "pfSense-pkg" as the beginning of the name, that's your clue you are looking at the GUI component.

Packages, when installed, usually bring in their own set of dependencies in the form of additional shared libraries. Swapping databases is sure to bring in a lot of new shared libraries, and that has the potential to break the firewall. Or at least require some careful testing as the new binary and its new dependencies are compiled and then installed. You would want to first do this on a test box in a lab environment.

If you want to proceed down this road, I recommend you configure a FreeBSD 12.2/STABLE virtual machine and compile the new telegraf package on it. Then copy the package over to a test pfSense machine and attempt to install/update the package from the command line using the pkg utility. Be prepared to crash and burn the test machine, thus do this within a lab setup of some type. Because you are modifying the underlying database, odds are you will need to also change some of the PHP GUI code to account for the new database. Possible changes would be connection strings, table names, etc.

xanaro

@bmeeks I was under the impression that telegraf simple write to the remote database you configure.... at least that is what I have it doing now.

currently I have an influxdb install on a different machine, in an LXC container sitting on a proxmox host. Telegraf in pfsense is configured to write to that remote host, all I am talking about doing is replacing the Telegraf included with pfsense with a telegraf that is also able to write to postgresql in addition to the ones you see that it can currently write to (InfluxDB, ElasticSearch, Graphite)

but maybe there is something about this that I am not understanding, thanks for the reply.

bmeeks

In order to actually write to database engines, a local database connectivity client is required. Think ODBC driver from Windows in the past. Each database (MySQL, PostgreSQL, etc.) has its own client piece of software (again, think ODBC driver from Windows). So in order for telegraf to talk to PostgreSQL, it must be compiled with the option to load and call the shared library for a PostgreSQL database client. Or again with my Windows scenario, telegraf must be compiled with the instructions necessary to load and call routines within the ODBC driver for PostgreSQL. Right now, on pfSense, that driver does not exist for telegraf.

So you first need to enable the PostgreSQL option within the binary portion of telegraf (assuming it has a configure script option knob for that). Then you will also need to likely make changes in the PHP GUI package code to properly format the connection string and any other configuration settings needed for telegraf to speak "PostgreSQL".

Whoever wrote the current telegraf GUI package on pfSense may have left it "open enough" so that you can type any kind of connection info in those text boxes. If so, that is great. But you still must recompile and replace the underlying binary piece as you surmise. But that has to be done with care.

xanaro

@bmeeks awesome! thanks for the info! gives me a jump start when i tinker on this during the week, will definitely lab test it on some spare hardware.