Grafana and InfluxDB#

InfluxDB is a database that is optimized for time-series data and analytics.

Grafana is a powerful and flexible dashboard and visualization tool from Grafana Labs (https://grafana.com/). It is available as Open Source software, but Grafana Labs does offer commercial support as well.

InfluxDB#

While InfluxDB is technically a separate tool, the Telegraf data collection tool is created by the same company and provides optimized plugins that push a variety of metrics into InfluxDB. Both tools are Open Source, but commercial support is available through InfluxData Inc. (https://www.influxdata.com/).

The InfluxDB database is queried through the Flux or InfluxQL language. Flux is a powerful data scripting and processing language and InfluxQL is SQL-like. Documentation is available from https://docs.influxdata.com/flux/v0/ and https://docs.influxdata.com/influxdb/v2/reference/syntax/influxql/.

For example, the following Flux query finds the CPU usage data for all nodes in the cluster:

from(bucket: "telegraf")
    |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
    |> filter(fn: (r) => r["_measurement"] == "cpu")
    |> filter(fn: (r) => r["_field"] == "usage_system")

This Flux example illustrates a typical approach to querying and filtering data in InfluxDB/Flux. First, a stream of records is pulled from a "bucket" (similar to a table in relational databases). That stream may include multiple metrics stored in multiple measurements and fields. A given Telegraf data collection cycle, for example, may include CPU and memory measurements, and each of those may have multiple fields of information: the CPU measurement may have per-CPU-core data, the memory measurement may have total, used, and free memory values. Each of those measurement-field combinations is a separate record in the stream.

It is often useful to reduce the time-range early in the query, otherwise all records from all time are processed by the stages of the query. The "earlier" the stream's data is reduced, the greater the speed-up is seen in the query process. In this example, the range is reduced to whatever has been selected by the GUI, with the "v." notation being used to reference one of the dashboard-shared "variables". Since a measurement can include multiple fields, it is useful to filter on measurement next since it will often provide another large reduction in the amount of stream data. Additional filtering on specific field information can be used to further refine the data.

Note

In August 2023, InfluxData announced plans to de-prioritize their future investments in Flux and to put the project into "maintenance mode". They explicitly state that Flux is not going End-Of-Life, and that they do anticipate supporting customers for some time to come. InfluxData also offers an SQL-like language, InfluxQL, and has plans for full SQL in version 3 of the InfluxDB database.

Starting with the ClusterWareAI ™ 13.0.0 release, connectors for both Flux and InfluxQL are deployed for Grafana, so you can use either language in your queries. Given the status of Flux, Penguin Solutions recommends using InfluxQL for easier porting to SQL in the future.

Data Retention#

By default, the ClusterWareAI platform retains 7 days of InfluxDB data. You can adjust the retention time and frequency of downsampling. A primary concern in adjusting these intervals is available disk space.

To update the InfluxDB data retention:

Obtain the bucket-id by running:
```
influx bucket list
```
Update the InfluxDB bucket retention by running:
```
influx bucket update -i <bucket-id> -r <new period>
```
Where <new-period> is an integer concatenated with a one-letter abbreviation of a time period. For example, "7d" or "1w" for one week, "14w" for 14 weeks, "12h" for 12 hours, "1y" for one year.

The longer the retention period means the greater the size of retained data. See https://docs.influxdata.com/influxdb/v2.6/reference/internals/data-retention/ for details.

To adjust downsampling, you need both a source and destination bucket. Then you create a task that runs a script to downsample data after a certain period of time. See the following InfluxData documentation for details:

Creating a new bucket: https://docs.influxdata.com/influxdb/v2/admin/buckets/create-bucket/
Creating a task: https://docs.influxdata.com/influxdb/v2/process-data/manage-tasks/create-task/
Downsampling data requirements and example: https://docs.influxdata.com/influxdb/v2/process-data/common-tasks/downsample-data/

If you want to visualize your downsampled data with Grafana, reference the appropriate bucket in your queries.

Update Node Name in InfluxDB#

If you rename a compute node in the ClusterWareAI platform, the Telegraf metrics for the old hostname remain in InfluxDB and must be manually removed before the new hostname is used.

Restart Telegraf on the renamed node:

cw-nodectl -i <node> exec systemctl restart telegraf

On a head node, delete the old hostname from InfluxDB.

List the host tags in InfluxDB:

/opt/scyld/clusterware/bin/influx-delete-host --show-influx-hosts

Preview the delete command:

/opt/scyld/clusterware/bin/influx-delete-host --dry-run HOSTNAME

Delete the old hostname.

On a single head node:

/opt/scyld/clusterware/bin/influx-delete-host HOSTNAME

On all head nodes:

/opt/scyld/clusterware/bin/influx-delete-host --all-heads HOSTNAME

Learn More About InfluxDB#

InfluxDB has a vibrant community of users and plugin developers. The official community forum at https://community.influxdata.com/ often has answers directly from the InfluxData team.

Refer to the official documentation from InfluxData for additional examples and videos:

https://docs.influxdata.com/flux/v0/get-started/
InfluxDB University: https://www.influxdata.com/university/ and https://university.influxdata.com/

Grafana#

Besides drawing charts and graphs, Grafana can provide alerting capabilities and, through the Loki plugin, may be used for log aggregation and analysis.

At a high level, a Grafana dashboard is a set of panels where each panel is a query against a data source along with a visualization for the resulting data. When the ClusterWareAI platform is installed, the InfluxDB data-source is configured, which connects to the InfluxDB/Telegraf data; it also comes with a cluster-wide and single-node dashboards. Either of those can be copied and then customized to meet any local needs.

Panels added to a dashboard show up as a blank panel. The corners of the panel can easily be clicked and dragged to be any size. Changing the query or visualization type requires editing the panel. The query is empty by default, so there may be no data displayed in the panel at first. Consider starting with the table visualization while working on the query since this gives direct feedback even if the query itself is in error or if it's producing different kind of data than expected. Once the query is refined, select a proper visualization type for that data.

Since the ClusterWareAI installation configures the InfluxDB data-source to use the Flux or InfluxQL language, all queries in Grafana must also use one of these languages.

Extensive GrafanaLabs documentation is available from their website: https://grafana.com/docs/grafana/latest/