Auditing#

The ICE ClusterWare ™ auditing system records significant events within the application using audit calls across the software.

Each audit call results in:

  • A JSON entry written to a local file.

  • The same JSON data published to a Mosquitto (MQTT) broker.

Audit Areas#

Auditing covers the following areas of the ClusterWare software:

  • api: Records API calls for all primitives (nodes, attribute groups, admins, and so on), mainly API calls of a write operation. Default logging level is INFO.

  • exc: Records exceptions raised internally. Default logging level is INFO.

  • http: Records all HTTP transactions as result of API calls. Default logging level is INFO.

  • log: Records all logging calls in audit.log. Default logging level is WARNING to reduce the number of records.

  • system: Records events occurred in various ClusterWare services. Default logging level is INFO.

Control Auditing Verbosity Level#

The ClusterWare auditing verbosity levels follow the standard syslog convention: DEBUG, INFO, WARNING, ERROR. Audit logging levels differ between subjects by default. You can set a global verbosity level by adding the following to the base.ini file on each head node:

audit.default_level = <level>

Where <level> is DEBUG, INFO, WARNING, ERROR, or DISABLED

You can override the global verbosity level by setting the level for each area. Add the following to the base.ini file on each head node:

audit.<area> = <level>

Where

  • <area> is api, exc, http, log, or system

  • <level> is DEBUG, INFO, WARNING, ERROR, or DISABLED

Auditing is enabled by default. To disable auditing for all areas, add the following to the base.ini file on each head node:

audit.default_level = DISABLED

You can disable auditing for a specific area by adding the following to the base.ini file on each head node:

audit.<area> = DISABLED

Where <area> is api, exc, http, log, or system.

Local Audit Log#

The default JSON audit log location is /var/log/clusterware/clusterware.audit. You can change the default location by adding an audit.path variable in the base.ini file for each head node. The audit log file contains all events in a flat JSON format.

All events written to the audit log file have common fields. For example:

{
   "when": <timestamp>,
   "level": <logging level>,
   "caller": <internal function that records this event>,
   "thread": <internal thread that records this event>,
   "event": <event name or a shorthand to describe this event>
}

Individual audit sections may have additional fields.

MQTT Messaging#

The ClusterWare software uses the Mosquitto (MQTT) protocol with a publish/subscribe architecture to generate structured messaging. Each auditing section is tagged with a topic. You can use a MQTT client to subscribe to auditing topics that are applicable to you. Auditing data is visualized in the ClusterWare - Audit Log Viewer dashboard in Grafana. See Grafana Cluster Monitoring for details.

Note

The list below is meant to be representative of the types of topics available, but is not a complete list. Additional topics for each subject may be available.

API Endpoint Topics

/clusterware/v1/api/attributegroups
/clusterware/v1/api/dynamicgroups
/clusterware/v1/api/admins
/clusterware/v1/api/admins/byname/<name>
/clusterware/v1/api/admins/byuid/<uid>
/clusterware/v1/api/images

Exception Topics

/clusterware/v1/exc
/clusterware/v1/exc/system
/clusterware/v1/exc/validation
/clusterware/v1/exc/http
/clusterware/v1/exc/api

HTTP Topics

/clusterware/v1/http/request
/clusterware/v1/http/response

Logs Topics

/clusterware/v1/log
/clusterware/v1/log/debug
/clusterware/v1/log/info
/clusterware/v1/log/warn
/clusterware/v1/log/error

System Topics

/clusterware/v1/system/daemons/leases
/clusterware/v1/system/daemons/mounter
/clusterware/v1/system/daemons/heartbeat
/clusterware/v1/system/daemons/node-states
/clusterware/v1/system/services/telegraf-relay
/clusterware/v1/system/services/influxdb