Integrating Keycloak with ICE ClusterWare for RBAC#
Installation#
Keycloak is a powerful, flexible authentication and authorization system with the ability to directly store user credentials and to link to other authentication providers. The ICE ClusterWare™ platform’s use of Keycloak does not necessarily use all of its features, and the full configuration process for all of Keycloak is beyond the scope of this document. For more in-depth information, admins can refer to the Keycloak website and, particularly, its documentation link:
https://www.keycloak.org/documentation
Assuming Keycloak is installed and operational, the process to integrate it with the ClusterWare platform involves selecting or creating a realm, creating a new client within that realm, adding users to the realm, assigning or creating roles that will map to ClusterWare roles, and finally, configuring and restarting the ClusterWare software.
Installation instructions can be found in Keycloak’s documentation. This includes how-to guides for several different types of installations: bare metal, Docker, Podman, Kubernetes, and OpenShift.
https://www.keycloak.org/guides#getting-started https://www.keycloak.org/operator/basic-deployment
Select a Realm#
In Keycloak, a "realm" is an administrative domain that includes users, groups, roles, and client application information. A fresh installation of Keycloak will only have one realm, called "master", and it is highly recommended that admins create at least one other realm to use in production. The "master" realm should not be used for daily operations. In an enterprise organization, there may already be a suitable realm available. Since client-specific information can be compartmentalized even within a realm, there shouldn’t be any concern about the ClusterWare platform’s use of that realm interfering with other operations.
Note
When creating a new realm, the name that is given will be used as the unique identifier for the realm, and will show up in the URL.
More information can be found in Keycloak’s Server Administration documentation:
https://www.keycloak.org/docs/latest/server_admin/index.html
Configuring realms:
https://www.keycloak.org/docs/latest/server_admin/index.html#configuring-realms
Create a New Client#
Inside the selected realm, admins should create a new client – a client is an administrative container for information related to a specific application or use-case within this realm. In this case, the ClusterWare platform will be given its own client-container so that it does not interfere with any other applications that may be using the same Keycloak instance.
When creating the client, note that the "client ID" acts as the unique identifier and as a "short name" for the client - it will be displayed on many of the Keycloak webpages and it will show up in the URL.
Specific settings to be verified:
The "Client authentication" and "Authorization" checkboxes must be enabled.
In the "Authentication flow" section, the "Standard flow" and "Direct access grants" checkboxes must be enabled.
The "Use refresh tokens" checkbox should be enabled (in the web-UI, this is under the "Advanced" tab).
Take note of the following information as it will be needed later during the ClusterWare configuration process:
The server connection information, including the server address or name and any port information
The realm name and client-ID
The "client secret" – in the web-UI, this can be found in the Credentials tab; the "Client Authenticator" should be "Client Id and Secret", and under it should be "Client Secret"
More information can be found in Keycloak’s Server Administration documentation:
https://www.keycloak.org/docs/latest/server_admin/index.html https://www.keycloak.org/docs/latest/server_admin/index.html#_oidc_clients
Add Users#
After setting up these basic "containers" for interacting with the ClusterWare platform, the next step is to create any users that will need ClusterWare admin roles. For enterprise organizations, these accounts may already be created and can be re-used for ClusterWare roles.
The user creation process is straightforward – each user needs a unique username (unique within the realm), and can optionally be given first/last names and an email address. If desired, users can be configured to be required to change their password at first login, or required to verify their email address, etc. If Keycloak was configured for other organizational uses, it may require additional or different user information, e.g. a physical address or an employee ID number.
It may be worth considering Keycloak’s "Groups" feature to categorize and
organize users to simplify user-to-role mapping.
E.g. a group could be defined for HPC-Admins and that group can then be
granted a role of hpc.fulladmin; any user added
to that group will thus become a Full Admin. This can greatly simplify the
process of adding roles to large numbers of users.
More details can be found in Keycloak’s Server Administration documentation:
https://www.keycloak.org/docs/latest/server_admin/index.html
User management:
Select or Create Roles#
For a given user, the ClusterWare platform will look at the Keycloak-provided list of roles to determine what that user is or is not allowed to do. The configuration process (see below) will need to map each of the ClusterWare roles to a Keycloak-provided role. For enterprise organizations, there may be existing roles that can be re-used for ClusterWare roles, but otherwise, admins will have to create any needed roles.
The ClusterWare roles are:
On-site Engineer
Imaging Engineer
Production Engineer
Manager
Full Admin
Authenticated User (this role is also granted to anyone in any of the other roles)
Once the roles are defined, users can then be assigned to those roles either directly or through a group.
Note
Admins may choose to not provide a mapping for a given ClusterWare role. In that case, then that role will simply be ignored. For example, if there is no mapping for the Manager role, then no one will ever be able to be assigned that role.
More information can be found in Keycloak’s Server Administration documentation:
https://www.keycloak.org/docs/latest/server_admin/index.html
Roles and Groups:
Configuring ClusterWare Software#
With the Keycloak configuration from above, admins can now configure the ClusterWare software to match. In particular, the ClusterWare software needs to know the basic configuration and the role-mapping details.
The Keycloak configuration info is stored in the ClusterWare database with the
authctl tool, found at /opt/scyld/clusterware/bin/authctl. When using
authctl, replace the values with info from the actual Keycloak instance as
noted above:
authctl set keycloak base_url https://192.168.122.33:8080
authctl set keycloak client clusterware
authctl set keycloak realm penguin
authctl set keycloak secret abc123def456ghi789
Next, the role-mapping needs to be configured. The mapping is given as a
comma-separated list of keycloak_role=clusterware_role
settings. For example:
authctl set keycloak role_mapping "kc_fulladmin=role.fulladmin,kc_manager=role.manager"
Note that the authctl tool can also be used to double-check the data in the
ClusterWare database:
authctl show
Finally, the main ClusterWare configuration needs to be modified to swap out the
standard authentication system (the appauth module)
with the Keycloak module. Edit /opt/scyld/clusterware/conf/base.ini and look
for the plugins.auth section:
plugins.auth = appauth
Change this to:
plugins.auth = keycloakauth
Note that it may be helpful to add a user or two to the auth.tmpadmins list
in base.ini as well. If anything goes wrong with the
Keycloak integration process, most users will not be able to authenticate – only
the tmpadmins will be allowed into the system,
and they will have full admin privileges.
When those file modifications are complete, save the file and restart the ClusterWare service:
systemctl restart clusterware
Important
For multi-head clusters, the base.ini file should be modified on all heads, then the service on each head node should be restarted.
Production Operations#
Once the steps in the Installation process are completed and the ClusterWare service is restarted, the cluster should now be looking to Keycloak for authentication credentials and roles. For production operations, there are several important changes to how admins interact with the system.
User Management#
Adding new users and modifying roles must now be done in both the Keycloak and
ClusterWare systems. The instructions for adding a new user to
Keycloak are shown in the "Add Users" and "Select or Create Roles" sections above.
Separately, the user must be added to the ClusterWare platform
through the scyld-adminctl command:
scyld-adminctl create name=username
Note
When using Keycloak, no roles should be assigned in the ClusterWare platform – all roles must be assigned through Keycloak.
To allow admins to create new tokens using scyld-adminctl token, several features must be enabled when starting Keycloak: token-exchange and admin-fine-grained-authz. Note that these features are marked by the Keycloak team as "Preview" features and may not be suitable for production environments. The features may be set inside the config file or on the command line.
Logging and Auditing#
Keycloak includes configurable logging and auditing settings that admins can use to directly track user behavior on the cluster, or alternatively, to feed into another system for intrusion detection or log analysis.
By default, the log file will be keycloak.log and will be found in the root
directory where keycloak was installed,
i.e. the <kc-install-root>/data/log directory. It should include all log
messages except DEBUG level messages (so INFO, WARNING, ERROR,
etc. should all be logged). The file follows standard Unix/Linux syslog
formatting and should be easily interpretable by other tools.
The documentation links below include examples of forwarding the data to Graylog
and ELK log aggregation systems.
There are several sections of the Keycloak documentation that may be of interest:
Access Token Lifespan#
When an admin logs in to a ClusterWare cluster, the ClusterWare server will pass
the information to Keycloak, which will verify the credentials
and return a set of tokens (JWT, Java Web Tokens). Those tokens will be cached
in each admin’s .scyldcw directory, in the auth_token.hdr file.
That file will include the current "access token" that can be used to perform
actions in the ClusterWare system, as well as a "refresh token" that can
be used to refresh the access token if or when it expires. If the refresh token
does expire, the user will be prompted for their password,
which will grant them a new set of tokens. The ClusterWare tools will
automatically use the tokens in a way that should reduce the number
of times the admin has to enter their password.
Current best practice is to limit the access token to a lifetime of approximately 5-10 minutes while allowing the refresh token to have a longer duration, such as 30-60 minutes. If a credential is stolen (or the file copied), then the Keycloak admins can potentially invalidate the refresh token and limit exposure to the compromised tokens. Token expiration settings for both access and refresh tokens will now be managed in Keycloak, so admins may re-configure those settings to reduce the frequency that passwords need to be re-entered.