Creating New Plugins#

The ClusterWareAI ™ platform provides a flexible plugin system for collecting a variety of node information – status, hardware, and Telegraf monitoring. The following plugins are available:

  • Status Plugins: These are called every status cycle, usually every 10 seconds. Status plugins can return any datatype. For example, free RAM is given as a float, distro is a string, and so on. The script can return multiple sensor readings.

  • Hardware Plugins: These are called less often than regular status plugins, usually every 300 seconds. Hardware plugins can return any datatype. One script can return multiple sensor readings.

  • Telegraf Plugins: These are more granular Telegraf configuration files that the ClusterWareAI platform can individually enable and disable. Each script is a self-contained piece of Telegraf configuration that contains settings for a given Telegraf input module.

Note

While plugins do run as root on the compute nodes, they can still be restricted by SELinux.

Creating Status Plugins#

Broadly speaking, a status plugin is a shell-script that returns one or more JSON-like blobs of data for a named sensor reading:

“<sensor_name>”:  <JSON_data>

For a single plugin that emits multiple sensor readings, each reading should be separated by a newline:

“<sensor_name1>”:  <JSON_data>
“<sensor_name2>”:  <JSON_data>

Important

Remember that this is not true JSON format because there is no comma separating the two sensors.

The script runs as root and can use any tool, or sequence of tools, on the system to collect and format the information.

While the script could reach across the network to run commands on other systems, this would potentially take significantly longer to process and should be avoided in status plugins.

The first line of the script should be a #! line giving the shell or interpreter to use. Every plugin is sent the path to the clusterware-node package as the first argument since there is a useful library of routines in the functions.sh script. While less useful for the usually-dynamic status information, every script is sent a cache directory as the second argument, usually /opt/scyld/clusterware-node/etc/status.d. This directory can be used to store information for the next cycle rather than recomputing some lengthy calculation.

The sensor name can be any quoted string, though it is recommended to not have spaces or odd characters to avoid later problems when trying to use the data. Instead, use alphanumeric characters and underscores (A-Za-z0-9_). The JSON-data portion can be any JSON data: a string, number, boolean, list, or dictionary.

To debug, execute the plugin scripts manually and verify that the output is correct. The plugin system does very little error-checking on the scripts. Since it is executing so often, it also does not do a formal check of the JSON. If bad data is returned, failures occur in the status-update cycle. On the compute node, the failure is likely silent (unless _status_debug=1) so it is best to check the head node logs to get more information on the error.

Note

Changes to the _status_plugins attribute are processed on the next status-update cycle, usually every 10 seconds.

Creating Hardware Plugins#

Broadly speaking, a hardware plugin is also just a shell-script that returns one or more JSON-like blobs of data for a named sensor reading:

“<sensor_name>”:  <JSON_data>

For a single plugin that emits multiple sensor readings, each reading should be separated by a newline. As with status plugins, the script can run any tool, or sequence of tools, on the system.

The first line of the script should be a #! line giving the shell or interpreter to use.

Since hardware information is usually more static, it may make sense for hardware plugins to calculate information once and then cache it to a file on disk to reduce future computational effort needed. Every script is sent the cache directory as the second argument, usually /opt/scyld/clusterware-node/etc/hardware.d. Every hardware plugin is also sent the path to the clusterware-node package as the first argument since there is a useful library of routines in the functions.sh script.

The sensor name can be any quoted string, though it is recommended to not have spaces or odd characters to avoid later problems when trying to use the data. Use alphanumeric characters and underscores (A-Za-z0-9_). The JSON-data portion can be any JSON data: a string, number, boolean, list, or dictionary.

To debug, execute the plugin scripts manually and verify that the output is correct. As with the status plugins, there is very little error-checking done on the scripts.

Note

Changes to _hardware_plugins is typically detected on the next status cycle (every 10 seconds), but is not processed until the next hardware-update cycle, usually every 300 seconds.

Creating Telegraf Plugins#

Where the status plugins are small scripts that are run during the periodic status-update cycle, Telegraf plugins are small configuration files that can be enabled or disabled.

Developing Telegraf plugins for the ClusterWareAI platform really means the creation of configuration files, often by splitting up one of the larger Telegraf sample configuration files into smaller, self-contained pieces. Developing brand-new Telegraf plugins is outside the scope of this document.

A list of Telegraf plugins can be found at: https://docs.influxdata.com/telegraf/v1/plugins/

Note

Head nodes can only use the telegraf-enabled directory as there is no attribute for Telegraf plugins. Run the /opt/scyld/clusterware-telegraf/bin/reconfig-telegraf.sh script manually to push any changes into production.

Telegraf Plugin Examples#

As an example, if a cluster is running the Ceph file system, you could use the Telegraf “inputs.ceph” module: influxdata/telegraf

  1. Create a new plugin file /opt/scyld/clusterware-telegraf/telegraf-available/ceph.conf:

    [[inputs.ceph]]
interval = '1m'
ceph_binary = "/usr/bin/ceph"
socket_dir = "/var/run/ceph"
mon_prefix = "ceph-mon"
osd_prefix = "ceph-osd"
mds_prefix = "ceph-mds"
rgw_prefix = "ceph-client"
socket_suffix = "asok"
ceph_user = "client.admin"
ceph_config = "/etc/ceph/ceph.conf"
gather_admin_socket_stats = true
gather_cluster_stats = false

  2. Enable the plugin on compute nodes with the _telegraf_plugins attribute:

    cw-nodectl --all set _telegraf_plugins=ceph

    Alternatively, sym-link the plugin into telegraf-enabled.

Once the plugin is enabled through either method, the Telegraf daemon on the compute nodes begins sending Ceph data to the head nodes.

Another example is using the Ping plugin (influxdata/telegraf) to ping several remote servers instead of just the parent head node. This could be useful to detect issues with a compute node's connection to the primary storage server and a network gateway.

  1. Edit the ping.conf file in the telegraf-available directory and include the storage server name in the list of URLs to ping:

    [[inputs.ping]]
## List of urls to ping
urls = ["parent-head-node", “storage-server”, “gateway-server”]
## number of pings to send per collection (ping -c <COUNT>)
count = 1

  2. Restart Telegraf to have the system re-read the modified configuration file:

    systemctl restart telegraf

Telegraf's exec and execd modules allow arbitrary scripts or executables to run, with Telegraf ingesting the output. This can be a very powerful tool for writing custom scripts specific to your cluster. See the InfluxData documentation for more information:

  • The exec module launches a new shell every update cycle when it runs the script influxdata/telegraf

  • The execd module creates one shell during Telegraf start-up. That shell is assumed to launch a daemon process that repeatedly sends data influxdata/telegraf