Keycloak identity and access management

Template version:v24-12-11

Helm charts used:codecentric/keycloakx v2.6.0 app v26.0bitnami/postgresql v16.2.3 app v16

This template contains the configuration files needed to run Keycloak in a Kubernetes cluster.

Note that this chart is the Quarkus based logical successor of the Wildfly based codecentric/keycloak chart.

Keycloak is a high performance Java-based identity and access management solution. It lets developers add an authentication layer to their applications with minimum effort.

Template override parameters

File _values-tpl.yaml contains template configuration parameters and their default values:

#
#   _values-tpl.yaml
#
#   cskygen template default values file
#
_tplname: k8s-keycloakx
_tpldescription: Kubernetes Keycloak namespace
_tplversion: 24-12-11
#
# Values to override
#
## k8s cluster credentials kubeconfig file
kubeconfig: config-k8s-mod
namespace:
  ## k8s namespace name
  name: keycloakx
  ## Service domain name
  domain: cskylab.net
publishing:
  ## External url
  url: keycloakx.mod.cskylab.net
  ## Password for administrative user
  password: 'NoFear21'
certificate:
  ## Cert-manager clusterissuer
  clusterissuer: ca-test-internal
registry:
  ## Proxy Repository for Docker
  proxy: harbor.cskylab.net/dockerhub
  ## Private Repository for private images uploading
  private: harbor.cskylab.net/cskylab
  username: admin
  password: 'NoFear21'
## Local storage PV's node affinity (Configured in pv*.yaml)
localpvnodes:    # (k8s node names)
  all_pv: k8s-mod-n1
  # k8s nodes domain name
  domain: cskylab.net
  # k8s nodes local administrator
  localadminusername: kos
localrsyncnodes: # (k8s node names)
  all_pv: k8s-mod-n2
  # k8s nodes domain name
  domain: cskylab.net
  # k8s nodes local administrator
  localadminusername: kos

TL;DR

Prepare LVM Data services for PV's:

• LVM Data Services

Install namespace and charts:

# Pull charts to './charts/' directory
./csdeploy.sh -m pull-charts
# Install  
./csdeploy.sh -m install
# Check status
./csdeploy.sh -l

Run:

• Published at: {{ .publishing.url }}/auth
• Username: keycloak
• Password: {{ .publishing.password }}

Prerequisites

• Administrative access to Kubernetes cluster.
• Helm v3.

keycloak-theme-provider image

Keycloak-theme-provider image must exist on your private registry {{ .registry.private }}.

To build and push the image, docker must be installed on your local machine and your private registry URL must be published with a trusted certificate. Install certificate and restart docker before pushing the image to the registry.

• Open console in keycloak-theme-provider directory and run:

# cSkyLab docker image build script
./csbuild.sh -b  # To build the image
./csbuild.sh -p  # To build and push the image

LVM Data Services

Data services are supported by the following nodes:

PV node is the node that supports the data service in normal operation.

RSync node is the node that receives data service copies synchronized by cron-jobs for HA.

To create the corresponding LVM data services, execute from your mcc management machine the following commands:

#
# Create LVM data services in PV node
#
echo \
&& echo "******** START of snippet execution ********" \
&& echo \
&& ssh {{ .localpvnodes.localadminusername }}@{{ .localpvnodes.all_pv }}.{{ .localpvnodes.domain }} \
  'sudo cs-lvmserv.sh -m create -qd "/srv/{{ .namespace.name }}" \
&& mkdir "/srv/{{ .namespace.name }}/data/postgresql"' \
&& echo \
&& echo "******** END of snippet execution ********" \
&& echo

#
# Create LVM data services in RSync node
#
echo \
&& echo "******** START of snippet execution ********" \
&& echo \
&& ssh {{ .localrsyncnodes.localadminusername }}@{{ .localrsyncnodes.all_pv }}.{{ .localrsyncnodes.domain }} \
  'sudo cs-lvmserv.sh -m create -qd "/srv/{{ .namespace.name }}" \
&& mkdir "/srv/{{ .namespace.name }}/data/postgresql"' \
&& echo \
&& echo "******** END of snippet execution ********" \
&& echo

To delete the corresponding LVM data services, execute from your mcc management machine the following commands:

#
# Delete LVM data services in PV node
#
echo \
&& echo "******** START of snippet execution ********" \
&& echo \
&& ssh {{ .localpvnodes.localadminusername }}@{{ .localpvnodes.all_pv }}.{{ .localpvnodes.domain }} \
  'sudo cs-lvmserv.sh -m delete -qd "/srv/{{ .namespace.name }}"' \
&& echo \
&& echo "******** END of snippet execution ********" \
&& echo

#
# Delete LVM data services in RSync node
#
echo \
&& echo "******** START of snippet execution ********" \
&& echo \
&& ssh {{ .localrsyncnodes.localadminusername }}@{{ .localrsyncnodes.all_pv }}.{{ .localrsyncnodes.domain }} \
  'sudo cs-lvmserv.sh -m delete -qd "/srv/{{ .namespace.name }}"' \
&& echo \
&& echo "******** END of snippet execution ********" \
&& echo

Persistent Volumes

Review values in all Persistent volume manifests with the name format ./pv-*.yaml.

The following PersistentVolume & StorageClass manifests are applied:

# PV manifests
pv-postgresql.yaml

The node assigned in nodeAffinity section of the PV manifest, will be used when scheduling the pod that holds the service.

How-to guides

Pull Charts

To pull charts, change the repositories and charts needed in variable source_charts inside the script csdeploy.sh and run:

# Pull charts to './charts/' directory
  ./csdeploy.sh -m pull-charts

When pulling new charts, all the content of ./charts directory will be removed, and replaced by the new pulled charts.

After pulling new charts redeploy the new versions with: ./csdeploy -m update.

Install

To create namespace, persistent volumes and install charts:

  # Create namespace, PV's and install charts
    ./csdeploy.sh -m install

Notice that PV's are not namespaced. They are deployed at cluster scope.

Update

To update charts settings, change values in files values-chart.yaml.

Redeploy or upgrade charts by running:

  # Redeploy or upgrade charts
    ./csdeploy.sh -m update

Uninstall

To uninstall charts, remove namespace and PV's run:

  # Uninstall charts, remove PV's and namespace
    ./csdeploy.sh -m uninstall

Remove

This option is intended to be used only to remove the namespace when chart deployment is failed. Otherwise, you must run ./csdeploy.sh -m uninstall.

To remove PV's, namespace and all its contents run:

  # Remove PV's namespace and all its contents
    ./csdeploy.sh -m remove

Display status

To display namespace, persistence and chart status run:

  # Display namespace, persistence and charts status:
    ./csdeploy.sh -l

Backup & data protection

Backup & data protection must be configured on file cs-cron_scripts of the node that supports the data services.

RSync HA copies

Rsync cronjobs are used to achieve service HA for LVM data services that supports the persistent volumes. The script cs-rsync.sh perform the following actions:

• Take a snapshot of LVM data service in the node that supports the service (PV node)
• Copy and syncrhonize the data to the mirrored data service in the kubernetes node designed for HA (RSync node)
• Remove snapshot in LVM data service

To perform RSync manual copies on demand, execute from your mcc management machine the following commands:

Warning: You should not make two copies at the same time. You must check the scheduled jobs in cs-cron-scripts and disable them if necesary, in order to avoid conflicts.

#
# RSync data services
#
echo \
&& echo "******** START of snippet execution ********" \
&& echo \
&& ssh {{ .localpvnodes.localadminusername }}@{{ .localpvnodes.all_pv }}.{{ .localpvnodes.domain }} \
  'sudo cs-rsync.sh -q -m rsync-to -d /srv/{{ .namespace.name }} \
  -t {{ .localrsyncnodes.all_pv }}.{{ .namespace.domain }}' \
&& echo \
&& echo "******** END of snippet execution ********" \
&& echo

RSync cronjobs:

The following cron jobs should be added to file cs-cron-scripts on the node that supports the service (PV node). Change time schedule as needed:

################################################################################
# /srv/{{ .namespace.name }} - RSync LVM data services
################################################################################
##
## RSync path:  /srv/{{ .namespace.name }}
## To Node:     {{ .localrsyncnodes.all_pv }}
## At minute 0 past every hour from 8 through 23.
# 0 8-23 * * *     root run-one cs-lvmserv.sh -q -m snap-remove -d /srv/{{ .namespace.name }} >> /var/log/cs-rsync.log 2>&1 ; run-one cs-rsync.sh -q -m rsync-to -d /srv/{{ .namespace.name }} -t {{ .localrsyncnodes.all_pv }}.{{ .namespace.domain }}  >> /var/log/cs-rsync.log 2>&1

Restic backup

Restic can be configured to perform data backups to local USB disks, remote disk via sftp or cloud S3 storage.

To perform on-demand restic backups execute from your mcc management machine the following commands:

Warning: You should not launch two backups at the same time. You must check the scheduled jobs in cs-cron-scripts and disable them if necesary, in order to avoid conflicts.

#
# Restic backup data services
#
echo \
&& echo "******** START of snippet execution ********" \
&& echo \
&& ssh {{ .localpvnodes.localadminusername }}@{{ .localpvnodes.all_pv }}.{{ .localpvnodes.domain }} \
  'sudo cs-restic.sh -q -m restic-bck -d  /srv/{{ .namespace.name }} -t {{ .namespace.name }}' \
&& echo \
&& echo "******** END of snippet execution ********" \
&& echo

To view available backups:

echo \
&& echo "******** START of snippet execution ********" \
&& echo \
&& ssh {{ .localpvnodes.localadminusername }}@{{ .localpvnodes.all_pv }}.{{ .localpvnodes.domain }} \
  'sudo cs-restic.sh -q -m restic-list  -t {{ .namespace.name }}' \
&& echo \
&& echo "******** END of snippet execution ********" \
&& echo

Restic cronjobs:

The following cron jobs should be added to file cs-cron-scripts on the node that supports the service (PV node). Change time schedule as needed:

################################################################################
# /srv/{{ .namespace.name }}- Restic backups
################################################################################
##
## Data service:  /srv/{{ .namespace.name }}
## At minute 30 past every hour from 8 through 23.
# 30 8-23 * * *   root run-one cs-lvmserv.sh -q -m snap-remove -d /srv/{{ .namespace.name }} >> /var/log/cs-restic.log 2>&1 ; run-one cs-restic.sh -q -m restic-bck -d  /srv/{{ .namespace.name }}  -t {{ .namespace.name }}  >> /var/log/cs-restic.log 2>&1 && run-one cs-restic.sh -q -m restic-forget   -t {{ .namespace.name }}  -f "--keep-hourly 6 --keep-daily 31 --keep-weekly 5 --keep-monthly 13 --keep-yearly 10" >> /var/log/cs-restic.log 2>&1

Upgrade PostgreSQL database version

1. Backup Running Container

The pg_dumpall utility is used for writing out (dumping) all of your PostgreSQL databases of a cluster. It accomplishes this by calling the pg_dump command for each database in a cluster, while also dumping global objects that are common to all databases, such as database roles and tablespaces.

The official PostgreSQL Docker image come bundled with all of the standard utilities, such as pg_dumpall, and it is what we will use in this tutorial to perform a complete backup of our database server.

If your Postgres server is running as a Kubernetes Pod, you will execute the following command:

kubectl -n {{ .namespace.name }} exec -i postgresql-0 -- /bin/bash -c "PGPASSWORD='{{ .publishing.password }}' pg_dumpall -U postgres" > postgresql.dump

2. Deploy New Postgres Image in a limited namespace

The second step is to deploy a new Postgress container using the updated image version. This container MUST NOT mount the same volume from the older Postgress container. It will need to mount a new volume for the database.

Note: If you mount to a previous volume used by the older Postgres server, the new Postgres server will fail. Postgres requires the data to be migrated before it can load it.

To deploy the new version on an empty volume:

• Uninstall the namespace containing the PostgreSQL service (Keycloakx)
• Delete the PostgreSQL data service
• Re-Create the PostgreSQL data service
• Change csdeploy.sh file commenting all helm pull deploying charts lines except helm pull bitnami/postgresql
• Remove all charts and pull only bitnami/postgresql chart by running csdeploy.sh - m pull-charts
• Deploy the namespace by running csdeploy.sh -m install

3. Import PostgreSQL Dump into New Pod With the new Postgres container running with a new volume mount for the data directory, you will use the psql command to import the database dump file. During the import process Postgres will migrate the databases to the latest system schema.

kubectl -n {{ .namespace.name }} exec -i postgresql-0 -- /bin/bash -c "PGPASSWORD='{{ .publishing.password }}' psql -U postgres" < postgresql.dump

4. Deploy the namespace with all charts

Once the PosgreSQL container is running with the new version and dumped data successfully restored, the namespace can be re-started with all its charts:

• Uninstall the namespace
• Change csdeploy.sh file un-commenting all helm pull deploying charts lines
• Re-Import all charts by running csdeploy.sh - m pull-charts
• Deploy the namespace by running csdeploy.sh -m install

Customize Keycloak Themes

You should personalize the provided customization themes with your own branding for each keycloak application client (keycloak, gitlab, nextcloud... etc).

The themes provided in the model environment are:

• /themes/bootstrap-mod-keycloak
• /themes/bootstrap-mod-gitlab
• /themes/bootstrap-mod-nextcloud

The customizable themes provided for your own publishing environment are:

• /themes/bootstrap-pub-keycloak
• /themes/bootstrap-pub-gitlab
• /themes/bootstrap-pub-nextcloud

Under the each theme login resource folder, the customized keycloak templates/functionalities are:

• login-page-expired.ftl
• login-reset-password.ftl
• login-update-profile.ftl
• login-verify-email.ftl
• login.ftl
• register.ftl

To customize a theme you should follow this procedure:

Customize login and account creation pages:

• Change the images files at /themes/bootstrap-pub-keycloak/login/resources/img/

  • brand.svg : Brand image
  • logo.png : Brand logo (96 x 96 pixels)
  • favicon.ico : Icon for browser tabs (48 x 48 pixels)
  • If required, ajust brand.svg image propeties by editing the #brand-logo class in the /themes/bootstrap-pub-keycloak/login/resources/css/login.css file.

• Change labels and description in resources messages files at /themes/bootstrap-pub-keycloak/login/messages.

  • usernameOrEmail label: How you named the username label depending if you enable "login with email" options at the Keycloak realm configuration.
  • applicationName: The name of the application you whant to customize the login page for.
  • applicationDescription: A short description or notice for your users.
  • loginIntroText: Subtitle or notice to your brand logo.
  • resetPasswordLabel: Label to reset password action.
  • emailVerifyTitle: Requiered "email verification" notice.
  • doCreateAccount: Create account action button label.
  • registerTitle: Registering window title.
  • registerAccountSectionTitle: Account fields group title
  • updateProfileTitle: Requiered "update profile action" notice.
  • isRequieredField: Requiered field error message.
  • minLengthField: Minimun length in characters a field should have to be valid.
  • maxLengthField: Maximun length in charactes a field must have to be valid.
  • minLengthFieldMessage: Error message to show when field value does not have the minimum field length defined in minLengthField variable. The asterisk symbol will be dinamically replaced with the minLengthField defined value.
  • maxLengthFieldMessage: Error message to show when field value has more characters thant the value defined at maxLengthField variable. The asterisk symbol will be dinamically replaced with the maxLengthField defined value.

Note: There are two resource language files predefined. If you add or delete language resource files, you need to modify accordingly the locales property on the /themes/bootstrap-pub-keycloak/login/theme.properties file. See more at https://www.keycloak.org/docs/latest/server_development/#_themes.

All validation values are for client validation only and should match the realm password policy defined in Keycloak.

• If needed by your branding requierements, you can change color schemes:
  • Choose a color scheme from https://bootstrapcolor.net/
  • Push Dowload CSS (bootstrap.css + boostrap.min.css)
  • Extract the zip file and look for file boostrap.min.css.
  • Replace the /themes/bootstrap-pub-keycloak/login/resources/css/bootstrap.min.css file with the downloaded version.

To learn more about keycloak themes customization:

• https://www.keycloak.org/docs/latest/server_development/#_themes
• https://github.com/keycloak/keycloak/tree/master/themes/src/main/resources/theme

Create the Theme Provider Image

• Edit the file values-keycloak.yaml to map themes folders to the Keycloak instance container under the extraInitContainers, extraVolumeMounts and extraVolumes values sections.

# file: values-keycloak.yaml 
# Additional init containers, e. g. for providing custom themes
extraInitContainers: |
  - name: keycloak-theme-provider
    image: {{ .registry.private }}/keycloak-theme-provider:stable
    imagePullPolicy: Always
    command:
      - sh
    args:
      - -c
      - |
        echo "Copying themes..."
        cp -R /themes/bootstrap-mod-keycloak theme1  
        cp -R /themes/bootstrap-mod-nextcloud theme2 
        cp -R /themes/bootstrap-mod-gitlab theme3
        cp -R /themes/bootstrap-pub-keycloak theme4  
        cp -R /themes/bootstrap-pub-nextcloud theme5 
        cp -R /themes/bootstrap-pub-gitlab theme6     
    volumeMounts:
      - name: theme1
        mountPath: /theme1/bootstrap-mod-keycloak
      - name: theme2
        mountPath: /theme2/bootstrap-mod-nextcloud
      - name: theme3
        mountPath: /theme3/bootstrap-mod-gitlab
      - name: theme4
        mountPath: /theme4/bootstrap-pub-keycloak
      - name: theme5
        mountPath: /theme5/bootstrap-pub-nextcloud
      - name: theme6
        mountPath: /theme6/bootstrap-pub-gitlab        
extraVolumeMounts: |
  - name: theme1
    mountPath: /opt/keycloak/themes/bootstrap-mod-keycloak
  - name: theme2
    mountPath: /opt/keycloak/themes/bootstrap-mod-nextcloud
  - name: theme3
    mountPath: /opt/keycloak/themes/bootstrap-mod-gitlab
  - name: theme4
    mountPath: /opt/keycloak/themes/bootstrap-pub-keycloak
  - name: theme5
    mountPath: /opt/keycloak/themes/bootstrap-pub-nextcloud
  - name: theme6
    mountPath: /opt/keycloak/themes/bootstrap-pub-gitlab    
extraVolumes: |
  - name: theme1
    emptyDir: {}
  - name: theme2
    emptyDir: {}
  - name: theme3
    emptyDir: {}
  - name: theme4
    emptyDir: {}
  - name: theme5
    emptyDir: {}
  - name: theme6
    emptyDir: {}    

Build and deploy Theme Provider Image

• To build the keycloak-theme-provider image, open a terminal in keycloak-theme-provider directory and run:

# Build and push to repository
./csbuild.sh -p

• After the image is pushed to your private registry, redeploy keycloak namespace by running:

  # Redeploy keycloak namespace
    ./csdeploy.sh -m update

Configure the new themes in Keycloak

Theme configuration can take place at Realm scope (all clients) or for each Client application.

• Login to keycloak

• For Realm scope configuration

  • Choose the realm
  • Go to Realm Settings
  • Go to Themes tab
  • Choose the appropiate theme in Login Theme box
  • Go to Login tab
  • Choose the appropiate options in Login screen customization section

• For Client scope configuration

  • Go to Clients and choose the client application
  • Choose the appropiate theme in Login Theme box
  • Save changes

Note: Some login actions like "user registration" or "reset password", are available accordingly with the realm configuration choosen at Login tab of the Realm Settings window.

Utilities

Passwords and secrets

Generate passwords and secrets with:

# Screen
echo $(head -c 512 /dev/urandom | LC_ALL=C tr -cd 'a-zA-Z0-9' | head -c 16)
# File (without newline)
printf $(head -c 512 /dev/urandom | LC_ALL=C tr -cd 'a-zA-Z0-9' | head -c 16) > RESTIC-PASS.txt

Change the parameter head -c 16 according with the desired length of the secret.

Reference

To learn more see:

• https://www.keycloak.org/
• https://github.com/codecentric/helm-charts/tree/master/charts/keycloak

Helm charts and values

Scripts

cs-deploy

Purpose:
  Kubernetes Keycloak identity and access management.
Usage:
  sudo csdeploy.sh [-l] [-m <execution_mode>] [-h] [-q]
Execution modes:
  -l  [list-status]     - List current status.
  -m  <execution_mode>  - Valid modes are:
      [pull-charts]     - Pull charts to './charts/' directory.
      [install]         - Create namespace, PV's and install charts.
      [update]          - Redeploy or upgrade charts.
      [uninstall]       - Uninstall charts, remove PV's and namespace.
      [remove]          - Remove PV's namespace and all its contents.
Options and arguments:  
  -h  Help
  -q  Quiet (Nonstop) execution.
Examples:
  # Pull charts to './charts/' directory
    ./csdeploy.sh -m pull-charts
  # Create namespace, PV's and install charts
    ./csdeploy.sh -m install
  # Redeploy or upgrade charts
    ./csdeploy.sh -m update
  # Uninstall charts, remove PV's and namespace
    ./csdeploy.sh -m uninstall
  # Remove PV's namespace and all its contents
    ./csdeploy.sh -m remove
  # Display namespace, persistence and charts status:
    ./csdeploy.sh -l

License

Copyright © 2021 cSkyLab.com ™

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.