Grafana Login#

The ICE ClusterWare ™ Monitoring graphical interface employs the Open Source Grafana, InfluxDB, and Telegraf to collect data from compute nodes and head nodes and present the data visually to authorized users. InfluxDB retains data for one week by default.

Access the Monitoring GUI through the Health + Monitoring > Telemetry Dashboard link in the ClusterWare GUI left navigation panel, directly using http://<HEADNODE_IP>/grafana, or for a specific node from the node details within the ClusterWare GUI.

Note

The URL http://<HEADNODE_IP>/grafana may differ if the cluster administrator has switched to HTTPS or otherwise modified the Apache configuration.

Grafana User Accounts#

By default the Grafana Setup Script creates the following Grafana user accounts:

  • The admin user account is a "global admin" or "server admin" Grafana user account.

  • The gr_root user account is similar to the admin account and can modify the Grafana instance.

  • The gr_admin user account does not have the same level of permissions as admin or gr_root, but can add or delete users, data, and dashboards. This account should only be shared with trusted users.

  • The gr_editor user account can add or delete dashboards, but not data or users.

  • The gr_viewer user account is the least permissive and is typically sufficient for most users.

The admin user password is equal to the database.admin_pass. The usernames and passwords for the other Grafana user accounts are written to the /opt/scyld/clusterware-grafana/lib/grafana-users file.

In a multi-tenant cluster, the tenancy creation process also creates the same Grafana user accounts. The file with usernames and passwords is only accessible by the superadministrator. The gr_viewer account associated with the tenancy should be sufficient for most tenancy administrators. Sharing the admin, gr_root, or gr_admin user accounts with a tenancy administrator is not recommended.

For more information about Grafana user roles and associated permissions, see the Grafana documentation: https://grafana.com/docs/grafana/latest/administration/roles-and-permissions/

Log In as Grafana Global Admin#

When the home page is loaded for the first time, log in with username "admin" and the database.admin_pass from the base.ini (sudo grep pass /opt/scyld/clusterware/conf/base.ini).

Typically after the initial "admin" database.admin_pass login you should edit the Preferences to change the Name, Email address, and Username to use for subsequent logins. Then, click "Change Password" and change the password for those subsequent logins.

Additional Configuration#

A basic Grafana Monitoring capability is installed preconfigured in the ClusterWare software. You can further modify this configuration to suit your local cluster needs when logged in as the Grafana global admin (gr_root by default). You can create new dashboards, or add new display panels to the existing ones to show more customized information. Grafana includes a suite of visualization tools like scatter, line, bar, and pie charts, as well as tables, gauges, and histograms. Since the underlying ClusterWare monitoring database is InfluxDB, any valid Flux-language query can be used to filter or process the data. For more information on InfluxDB and Grafana customization, including links to tutorials, see InfluxDB and Grafana.

To facilitate monitoring of compute node GPU activity:

  1. Install the NVidia System Management Interface utility (nvidia-smi) into the GPU compute node image(s). The NVidia System Management Interface utility ships with NVidia GPU drivers.

  2. In the compute node image(s), copy /etc/telegraf/telegraf.d/nvidia-smi.conf.example (distributed in the clusterware-node RPM) to /etc/telegraf/telegraf.d/nvidia-smi.conf.

See https://developer.nvidia.com/nvidia-system-management-interface for details about the NVidia System Management Interface utility and https://www.cyberciti.biz/faq/how-to-install-nvidia-driver-on-centos-7-linux/ for a description of how to install NVidia drivers.