Auditing#

The ClusterWareAI ™ auditing system records significant events within the application using audit calls across the software.

Each audit call results in:

A JSON entry written to a local file.
The same JSON data published to a Mosquitto (MQTT) broker.

Audit Areas#

Auditing covers the following areas of the ClusterWareAI software:

api: Records API calls for all primitives (nodes, attribute groups, admins, and so on), mainly API calls of a write operation. Default logging level is INFO.
exc: Records exceptions raised internally. Default logging level is INFO.
http: Records all HTTP transactions as result of API calls. Default logging level is INFO.
log: Records all logging calls in audit.log. Default logging level is WARNING to reduce the number of records.
system: Records events occurred in various ClusterWareAI services. Default logging level is INFO.

Control Auditing Verbosity Level#

The ClusterWareAI auditing verbosity levels follow the standard syslog convention: DEBUG, INFO, WARNING, ERROR. Audit logging levels differ between subjects by default. You can set a global verbosity level by adding the following to the base.ini file on each head node:

audit.default_level = <level>

Where <level> is DEBUG, INFO, WARNING, ERROR, or DISABLED

You can override the global verbosity level by setting the level for each area. Add the following to the base.ini file on each head node:

audit.<area> = <level>

Where

<area> is api, exc, http, log, or system
<level> is DEBUG, INFO, WARNING, ERROR, or DISABLED

Auditing is enabled by default. To disable auditing for all areas, add the following to the base.ini file on each head node:

audit.default_level = DISABLED

You can disable auditing for a specific area by adding the following to the base.ini file on each head node:

audit.<area> = DISABLED

Where <area> is api, exc, http, log, or system.

Local Audit Log#

The default JSON audit log location is /var/log/clusterware/clusterware.audit. You can change the default location by adding an audit.path variable in the base.ini file for each head node. The audit log file contains all events in a flat JSON format.

All events written to the audit log file have common fields. For example:

{
   "when": <timestamp>,
   "level": <logging level>,
   "caller": <internal function that records this event>,
   "thread": <internal thread that records this event>,
   "event": <event name or a shorthand to describe this event>
}

Individual audit sections may have additional fields.

MQTT Messaging#

The ClusterWareAI software uses the Mosquitto (MQTT) protocol with a publish/subscribe architecture to generate structured messaging. Each auditing section is tagged with a topic. You can use an MQTT client to subscribe to auditing topics that are applicable to you. Auditing data is visualized in the ClusterWareAI - Audit Log Viewer dashboard in Grafana. See Grafana Cluster Monitoring for details.

Note

The list below is meant to be representative of the types of topics available, but is not a complete list. Additional topics for each subject may be available.

AI Factory Operations Agent Topics

ai-agent/v1/audit/chat
ai-agent/v1/audit/tool
ai-agent/v1/audit/session
ai-agent/v1/audit/authz

API Endpoint Topics

/clusterware/v1/api/attributegroups
/clusterware/v1/api/dynamicgroups
/clusterware/v1/api/admins
/clusterware/v1/api/admins/byname/<name>
/clusterware/v1/api/admins/byuid/<uid>
/clusterware/v1/api/images

ARS and CHMS Topics

ars/v1/cluster/events/error_detected
ars/v1/logs/skyhook
ars/v1/logs/state_machine

Exception Topics

/clusterware/v1/exc
/clusterware/v1/exc/system
/clusterware/v1/exc/validation
/clusterware/v1/exc/http
/clusterware/v1/exc/api

HTTP Topics

/clusterware/v1/http/request
/clusterware/v1/http/response

Logs Topics

/clusterware/v1/log
/clusterware/v1/log/debug
/clusterware/v1/log/info
/clusterware/v1/log/warn
/clusterware/v1/log/error

System Topics

/clusterware/v1/system/daemons/leases
/clusterware/v1/system/daemons/mounter
/clusterware/v1/system/daemons/heartbeat
/clusterware/v1/system/daemons/node-states
/clusterware/v1/system/services/telegraf-relay
/clusterware/v1/system/services/influxdb

Adjust MQTT Message Storage#

By default, MQTT messages for calls to the Multiplexer (Mux) Endpoint are not stored in InfluxDB to reduce disk space requirements. You can add those messages back to the database or adjust other messages that are less important to you to save additional space.

Copy the base mqtt_consumer.conf file into telegraf-enabled:
```
cp /opt/scyld/clusterware-telegraf/telegraf-available/mqtt_consumer.conf \
   /opt/scyld/clusterware-telegraf/telegraf-enabled.conf
```
Note

If the file already has a symlink, delete the symlink first and then copy the file.
Edit the /opt/scyld/clusterware-telegraf/telegraf-enabled/mqtt_consumer.conf file to add and remove MQTT topics from InfluxDB storage. A best practice is to listen to a larger list of topics and narrow down the list using inputs.mqtt_consumer.tagdrop.
1. To add to the list of topics to listen to, modify the list under topics. Use the number sign (#) to indicate a wildcard. For example:
```
topics = [
  "#"
]
```
  Wildcards can also be used inside topic paths. For example, /api/v1/nodes/# listens for topics with the specified prefix.
2. To remove topics you are not interested in, add to the list under inputs.mqtt_consumer.tagdrop. Use an asterisk (*) to indicate a wildcard. Matched topics are not stored in InfluxDB. For example, the following does not store messages related to the mux or attribute group endpoints:
```
[inputs.mqtt_consumer.tagdrop]
  topic = [
    "*/v1/api/mux",
    "*/api/v1/attrib"
  ]
```
Run the /opt/scyld/clusterware-telegraf/bin/reconfig_telegraf.sh script. The script copies the updated file to the configuration directory and restarts Telegraf.