Integrating Keycloak with ICE ClusterWare#

Keycloak is a powerful, flexible authentication and authorization system with the ability to directly store user credentials and to link to other authentication providers. The ICE ClusterWare ™ platform’s use of Keycloak does not necessarily use all of its features, and the full configuration process for all of Keycloak is beyond the scope of this document. For more in-depth information, admins can refer to the Keycloak website and, particularly, its documentation link:

https://www.keycloak.org/documentation

Configure Keycloak and ClusterWare#

Assuming Keycloak is installed and operational, the process to integrate it with the ClusterWare platform involves selecting or creating a realm, creating a new client within that realm, adding users to the realm, assigning or creating roles that map to ClusterWare roles, and finally, configuring and restarting the ClusterWare software.

Note

Additional configuration is required to use Keycloak with SELinux. Contact Penguin Computing to learn more.

Install Keycloak#

Installation instructions can be found in Keycloak’s documentation. This includes how-to guides for several different types of installations: bare metal, Docker, Podman, Kubernetes, and OpenShift.

Select a Keycloak Realm#

In Keycloak, a "realm" is an administrative domain that includes users, groups, roles, and client application information. A fresh installation of Keycloak only has one realm, called "master", and it is highly recommended that admins create at least one other realm to use in production. The "master" realm should not be used for daily operations. In an enterprise organization, there may already be a suitable realm available. Since client-specific information can be compartmentalized even within a realm, there shouldn’t be any concern about the ClusterWare platform’s use of that realm interfering with other operations.

Note

When creating a new realm, the name that is given is used as the unique identifier for the realm and shows up in the URL.

More information can be found in Keycloak’s Server Administration documentation:

https://www.keycloak.org/docs/latest/server_admin/index.html#_configuring-realms

Create a Keycloak Client#

A client is an administrative container for information related to a specific application or use case within this realm.

Create a new client for the ClusterWare platform. Giving the ClusterWare platform its own client-container means it does not interfere with any other applications that may be using the same Keycloak instance.

When creating the client, note that the "client ID" acts as the unique identifier and as a "short name" for the client. It is displayed on many of the Keycloak webpages and is part of the URL.

Specific settings for the ClusterWare client within Keycloak:

  • The "Client authentication" and "Authorization" checkboxes must be enabled.

  • In the "Authentication flow" section, the "Standard flow" and "Direct access grants" checkboxes must be enabled.

  • The "Use refresh tokens" checkbox should be enabled (in the web-UI, this is under the "Advanced" tab).

When creating the client, take note of the following information as it is needed later during the ClusterWare configuration process:

  • The server connection information, including the server address or name and any port information.

  • The realm name and client ID.

  • The "client secret" – in the web UI. This can be found in the Credentials tab. The "Client Authenticator" should be "Client Id and Secret", and under it should be "Client Secret".

More information can be found in Keycloak’s Server Administration documentation:

Add Keycloak Users#

After setting up these basic containers for interacting with the ClusterWare platform, the next step is to create any users that need ClusterWare admin roles. For enterprise organizations, these accounts may already be created and can be re-used for ClusterWare roles.

The user creation process is straight-forward – each user needs a unique username within the realm, and can optionally be given first/last names and an email address. You can also configure users to be required to change their password at first login, required to verify their email address, and so on. If Keycloak was configured for other organizational uses, it may require additional user information, such as a physical address or an employee ID number.

Additionally, you may consider Keycloak’s "Groups" feature to categorize and organize users to simplify user-to-role mapping. For example, you can define a group for HPC-Admins and that group can then be granted a role of hpc.fulladmin. Any user added to that group will become a Full Admin. This can greatly simplify the process of adding roles to large numbers of users.

More details can be found in Keycloak’s Server Administration documentation:

https://www.keycloak.org/docs/latest/server_admin/index.html#assembly-managing-users_server_administration_guide

Select or Create Keycloak Roles#

For a given user, the ClusterWare platform looks at the Keycloak-provided list of roles to determine what that user is or is not allowed to do. The configuration process needs to map each of the ClusterWare roles to a Keycloak-provided role. For enterprise organizations, there may be existing roles that can be re-used for ClusterWare roles. If not, create any necessary roles.

A full list of the default ClusterWare roles is available in Role-Based Access Control System.

Once the roles are defined, users can then be assigned to those roles either directly or through a group.

Note

Admins may choose to not provide a mapping for a given ClusterWare role. In that case, then that role is simply ignored. For example, if there is no mapping for the ClusterWare Manager role, then no one is ever able to be assigned that role.

More information can be found in Keycloak’s Server Administration documentation:

https://www.keycloak.org/docs/latest/server_admin/index.html#assigning-permissions-using-roles-and-groups

Configure Keycloak Roles in ClusterWare#

After completing the Keycloak configuration, configure the ClusterWare software to match. In particular, the ClusterWare software needs to know the basic Keycloak configuration and the role-mapping details.

The Keycloak configuration information is stored in the ClusterWare database with the configctl tool, found at /opt/scyld/clusterware/bin/configctl.

  1. Use configctl to set the values with your Keycloak instance data:

    configctl set keycloak base_url <keycloak server address>
configctl set keycloak client <client ID>
configctl set keycloak realm <realm name>
configctl set keycloak secret <client secret>

    For example:

    configctl set keycloak base_url https://192.168.122.33:8080
configctl set keycloak client clusterware
configctl set keycloak realm penguin
configctl set keycloak secret abc123def456ghi789

  2. Configure role mapping using a comma-separated list of keycloak_role=clusterware_role settings:

    configctl set keycloak role_mapping "<keycloak role>=<clusterware role>"

    For example:

    configctl set keycloak role_mapping "kc_fulladmin=role.fulladmin,kc_manager=role.manager"

  3. Use the configctl tool to double-check the data in the ClusterWare database:

    configctl show

  4. Modify the main ClusterWare configuration to swap out the standard ClusterWare authentication system (the appauth module) with the Keycloak module. Edit the /opt/scyld/clusterware/conf/base.ini file and replace plugins.auth = appauth with:

    plugins.auth = keycloakauth

  5. [Optional] It may be helpful to add at least one user to the auth.tmpadmins list in the base.ini file as well. If anything goes wrong with the Keycloak integration process, most users will not be able to authenticate. However, the tmpadmins are allowed into the system and have full admin privileges.

  6. Save the file and restart the ClusterWare service:

    systemctl restart clusterware

Important

For multi-head clusters, the base.ini file should be modified on all heads, then the service on each head node should be restarted.

Use Keycloak for Authentication#

After configuring Keycloak to work with ClusterWare and restarting the ClusterWare service, the cluster uses Keycloak for authentication credentials and roles. For production operations, there are several important changes to how ClusterWare admins interact with the system.

User Management#

Adding new users and modifying roles must now be done in both the Keycloak and ClusterWare systems.

  1. Add the user to Keycloak and assign appropriate roles. See Add Keycloak Users and Select or Create Keycloak Roles.

  2. Add the user to the ClusterWare platform using the cw-adminctl command:

    cw-adminctl create name=username

    Note

    When using Keycloak, no roles should be assigned in the ClusterWare platform – all roles must be assigned through Keycloak.

  3. To allow admins to create new tokens using cw-adminctl token, enable the token-exchange and admin-fine-grained-authz features in Keycloak. The features can be set inside the config file or on the command line.

    Note

    The token-exchange and admin-fine-grained-authz features are marked as "Preview" features by the Keycloak team and may not be suitable for production environments.

Impersonate Keycloak User#

Keycloak and the ClusterWare software support user impersonation via the Keycloak token-exchange feature. By default, the Keycloak user impersonation token has a lifespan of 300 seconds.

  1. Enable the token-exchange, admin-fine-grained-authz, and impersonation features on the Keycloak server.

    For Keycloak version 26.3.3:

    kc.sh start-dev --features=token-exchange

    For Keycloak version 26.0.4:

    bin/kc.sh start-dev --features=token-exchange,admin-fine-grained-authz,impersonation

  2. In Keycloak, assign the "impersonate" permission to all users or groups who are allowed to do user impersonation. If the permission is not available in your client, enable fine-grained permissions for the client in Keycloak and it should appear.

Logging and Auditing#

Keycloak includes configurable logging and auditing settings that admins can use to directly track user behavior on the cluster, or alternatively, to feed into another system for intrusion detection or log analysis.

By default, the log file is keycloak.log and is found in the root directory where Keycloak was installed, typically the <kc-install-root>/data/log directory. The log file should include all log messages except DEBUG level messages (INFO, WARNING, ERROR, and so on). The file follows standard Unix/Linux syslog formatting and should be easily interpreted by other tools.

There are several sections of the Keycloak documentation that may be of interest and include examples of forwarding the data to Graylog and ELK log aggregation systems:

Access Token Lifespan#

When an admin logs in to a ClusterWare cluster, the ClusterWare server passes the information to Keycloak, which verifies the credentials and returns a set of tokens (JWT, Java Web Tokens). The tokens are cached in each admin’s .scyldcw directory, in the auth_token.hdr file. That file includes the current "access token" that can be used to perform actions in the ClusterWare system, as well as a "refresh token" that can be used to refresh the access token if or when it expires. If the refresh token expires, the user is prompted for their password, which grants them a new set of tokens. The ClusterWare tools automatically use the tokens in a way that should reduce the number of times that the admin has to enter their password.

Current best practice is to limit the access token to a lifespan of approximately 5-10 minutes while allowing the refresh token to have a longer duration, such as 30-60 minutes. If a credential is stolen (or the file copied), then the Keycloak admins can invalidate the refresh token and limit exposure to the compromised tokens. Token expiration settings for both access and refresh tokens is managed in Keycloak, so admins may re-configure those settings to reduce the frequency that passwords need to be re-entered.