ICE ClusterWare Log Files

The /var/log/clusterware/ folder contains several log files that may help diagnose problems. Additionally, the ICE ClusterWare™ database service may have useful information in its logs.

For etcd, see /var/log/clusterware/etcd.log.

On a typical head node the /var/log/clusterware/ folder contains api_access_log and api_error_log files. These are the Apache logs for the service providing the REST API. The log level available in this file is controlled by the Pyramid logging configuration in the /opt/scyld/clusterware/conf/pyramid.ini file. The Pyramid project documentation contains details of the pertinent variables https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html

A selection of log statements from the api_error_log are also logged to the ClusterWare database and then copied to the logging folder on each head node. A separate log file is created for each head node and is named based on the head node UID, i.e. head_293aafd3f635448e9aaa76fc998ebc0c.log. This should allow a cluster administrator to diagnose many problems without needing to contact every head node individually. The log level for this file is controlled by the logging.level variable in each head node's /opt/scyld/clusterware/conf/base.ini file. The default log level of WARNING should be useful but not overly verbose. The options from most terse to most verbose are AUDIT, ERROR, WARNING, INFO, DEBUG.

The various /var/log/clusterware/* logfiles are periodically rotated, as directed by the /etc/logrotate.d/clusterware, /etc/logrotate.d/clusterware-dnsmasq, and /etc/logrotate.d/clusterware-iscdhcp configuration files that distributed distributed in the clusterware, clusterware-dnsmasq, and clusterware-iscdhcp RPMs, respectively.

Note

If the local cluster administrator modifies the /etc/logrotate.d/clusterware file, then a subsequent update of clusterware RPM will install a new version as /etc/logrotate.d/clusterware.rpmnew. The cluster administrator should merge this clusterware.rpmnew into the local customized /etc/logrotate.d/clusterware. Similar treatment of clusterware-dnsmasq and clusterware-iscdhcp is advised.