Close
    logo                                         

    3 - Kubernetes

    Building k8s-mod & k8s-pro clusters

    This procedure explains how to build cSkyLab k8s clusters infrastructure

    cskylab-layers-k8s
    cskylab-layers-k8s

    Install & configure k8s nodes from mcc

    Inject ssh keys

    • From VS Code Remote connected to mcc, open terminal at root folder repository
    • Customize and execute the following command to inject ssh keys to k8s nodes:

    Note: Use default virtual machine creation password for local admin user (Default is NoFear21).

    # Set environment variables
    export REPO_DIR="$HOME/mpb-1100"
    # Inject ssh keys to k8s nodes
    echo && echo "******** SOE - START of execution ********" && echo \
    && cd ${REPO_DIR}/cs-mod/k8s-mod-master \
    && ./csinject.sh -qk \
    && cd ${REPO_DIR}/cs-mod/k8s-mod-n1 \
    && ./csinject.sh -qk \
    && cd ${REPO_DIR}/cs-mod/k8s-mod-n2 \
    && ./csinject.sh -qk \
    && cd ${REPO_DIR}/cs-mod/k8s-mod-n3 \
    && ./csinject.sh -qk \
    && cd ${REPO_DIR}/cs-mod/k8s-mod-n4 \
    && ./csinject.sh -qk \
    && cd ${REPO_DIR}/cs-pro/k8s-pro-master \
    && ./csinject.sh -qk \
    && cd ${REPO_DIR}/cs-pro/k8s-pro-n1 \
    && ./csinject.sh -qk \
    && cd ${REPO_DIR}/cs-pro/k8s-pro-n2 \
    && ./csinject.sh -qk \
    && cd ${REPO_DIR}/cs-pro/k8s-pro-n3 \
    && ./csinject.sh -qk \
    && cd ${REPO_DIR}/cs-pro/k8s-pro-n4 \
    && ./csinject.sh -qk \
    && echo && echo "******** EOE - END of execution ********" && echo

    Inject and execute net-config

    • From VS Code Remote connected to mcc, open terminal at root folder repository
    • Customize and execute the following command to inject configuration files and execute net-config in k8s nodes:
    # Set environment variables
    export REPO_DIR="$HOME/mpb-1100"
    # Inject and execute net-config in k8s nodes
    echo && echo "******** SOE - START of execution ********" && echo \
    ; cd ${REPO_DIR}/cs-mod/k8s-mod-master \
    ; ./csinject.sh -qdm net-config \
    ; cd ${REPO_DIR}/cs-mod/k8s-mod-n1 \
    ; ./csinject.sh -qdm net-config \
    ; cd ${REPO_DIR}/cs-mod/k8s-mod-n2 \
    ; ./csinject.sh -qdm net-config \
    ; cd ${REPO_DIR}/cs-mod/k8s-mod-n3 \
    ; ./csinject.sh -qdm net-config \
    ; cd ${REPO_DIR}/cs-mod/k8s-mod-n4 \
    ; ./csinject.sh -qdm net-config \
    ; cd ${REPO_DIR}/cs-pro/k8s-pro-master \
    ; ./csinject.sh -qdm net-config \
    ; cd ${REPO_DIR}/cs-pro/k8s-pro-n1 \
    ; ./csinject.sh -qdm net-config \
    ; cd ${REPO_DIR}/cs-pro/k8s-pro-n2 \
    ; ./csinject.sh -qdm net-config \
    ; cd ${REPO_DIR}/cs-pro/k8s-pro-n3 \
    ; ./csinject.sh -qdm net-config \
    ; cd ${REPO_DIR}/cs-pro/k8s-pro-n4 \
    ; ./csinject.sh -qdm net-config \
    ; echo && echo "******** EOE - END of execution ********" && echo

    Note: These commands MUST be executed with ";" concatenation because of machine reboot at the end of each command.

    Inject and execute install

    • From VS Code Remote connected to mcc, open terminals at root folder repository
    • Customize & execute in parallel the following commands to inject and execute install in k8s nodes:
    # Set environment variables
    export REPO_DIR="$HOME/mpb-1100"
    # Inject and execute install in k8s-mod nodes
    echo && echo "******** SOE - START of execution ********" && echo \
    ; cd ${REPO_DIR}/cs-mod/k8s-mod-master \
    ; ./csinject.sh -qdm install \
    ; cd ${REPO_DIR}/cs-mod/k8s-mod-n1 \
    ; ./csinject.sh -qdm install \
    ; cd ${REPO_DIR}/cs-mod/k8s-mod-n2 \
    ; ./csinject.sh -qdm install \
    ; cd ${REPO_DIR}/cs-mod/k8s-mod-n3 \
    ; ./csinject.sh -qdm install \
    ; cd ${REPO_DIR}/cs-mod/k8s-mod-n4 \
    ; ./csinject.sh -qdm install \
    ; echo && echo "******** EOE - END of execution ********" && echo
    # Set environment variables
    export REPO_DIR="$HOME/mpb-1100"
    # Inject and execute install in k8s-pro nodes
    echo && echo "******** SOE - START of execution ********" && echo \
    ; cd ${REPO_DIR}/cs-pro/k8s-pro-master \
    ; ./csinject.sh -qdm install \
    ; cd ${REPO_DIR}/cs-pro/k8s-pro-n1 \
    ; ./csinject.sh -qdm install \
    ; cd ${REPO_DIR}/cs-pro/k8s-pro-n2 \
    ; ./csinject.sh -qdm install \
    ; cd ${REPO_DIR}/cs-pro/k8s-pro-n3 \
    ; ./csinject.sh -qdm install \
    ; cd ${REPO_DIR}/cs-pro/k8s-pro-n4 \
    ; ./csinject.sh -qdm install \
    ; echo && echo "******** EOE - END of execution ********" && echo

    Note: These commands MUST be executed with ";" concatenation because of machine reboot at the end of each command.

    Inject ssh keys between k8s nodes

    From each node, inject ssh keys to the other node in k8s cluster

    Note: Use password for local admin user in your installation (See file secrets/admin-passwords).

    k8s-mod nodes:

    • From VS Code Remote connected to mcc, open terminals at each k8s node and connect inside the machine with ./csconnect.sh:
      • k8s-mod-n1
      • k8s-mod-n2
      • k8s-mod-n3
      • k8s-mod-n4
    # Inject ssh keys from k8s-mod-n1
    sudo ssh-copy-id kos@k8s-mod-n2.cskylab.net
    sudo ssh-copy-id kos@k8s-mod-n3.cskylab.net
    sudo ssh-copy-id kos@k8s-mod-n4.cskylab.net
    # Inject ssh keys from k8s-mod-n2
    sudo ssh-copy-id kos@k8s-mod-n1.cskylab.net
    sudo ssh-copy-id kos@k8s-mod-n3.cskylab.net
    sudo ssh-copy-id kos@k8s-mod-n4.cskylab.net
    # Inject ssh keys from k8s-mod-n3
    sudo ssh-copy-id kos@k8s-mod-n1.cskylab.net
    sudo ssh-copy-id kos@k8s-mod-n2.cskylab.net
    sudo ssh-copy-id kos@k8s-mod-n4.cskylab.net
    # Inject ssh keys from k8s-mod-n4
    sudo ssh-copy-id kos@k8s-mod-n1.cskylab.net
    sudo ssh-copy-id kos@k8s-mod-n2.cskylab.net
    sudo ssh-copy-id kos@k8s-mod-n3.cskylab.net

    k8s-pro nodes:

    • From VS Code Remote connected to mcc, open terminals at each k8s node and connect inside the machine with ./csconnect.sh:
      • k8s-pro-n1
      • k8s-pro-n2
      • k8s-pro-n3
      • k8s-pro-n4
    # Inject ssh keys from k8s-pro-n1
    sudo ssh-copy-id kos@k8s-pro-n2.cskylab.net
    sudo ssh-copy-id kos@k8s-pro-n3.cskylab.net
    sudo ssh-copy-id kos@k8s-pro-n4.cskylab.net
    # Inject ssh keys from k8s-pro-n2
    sudo ssh-copy-id kos@k8s-pro-n1.cskylab.net
    sudo ssh-copy-id kos@k8s-pro-n3.cskylab.net
    sudo ssh-copy-id kos@k8s-pro-n4.cskylab.net
    # Inject ssh keys from k8s-pro-n3
    sudo ssh-copy-id kos@k8s-pro-n1.cskylab.net
    sudo ssh-copy-id kos@k8s-pro-n2.cskylab.net
    sudo ssh-copy-id kos@k8s-pro-n4.cskylab.net
    # Inject ssh keys from k8s-pro-n4
    sudo ssh-copy-id kos@k8s-pro-n1.cskylab.net
    sudo ssh-copy-id kos@k8s-pro-n2.cskylab.net
    sudo ssh-copy-id kos@k8s-pro-n3.cskylab.net

    Initialize k8s clusters

    • From VS Code Remote connected to mcc, open terminals at each k8s master and connect to the machine with ./csconnect.sh:
      • k8s-mod-master
      • k8s-pro-master
    • Initialize k8s cluster executing the following command in each machine:
    # Initialize k8s cluster
    sudo cs-k8init.sh -m init-master
    • In each machine copy the displayed kubeadm join command with token to join cluster, and save it into the following files in your installation repository:
      • secrets/k8s-mod-join-command
      • secrets/k8s-pro-join-command
    • Check cluster initialization with kubectl get nodes
    • Exit from console connections

    Copy k8s clusters kubeconfig files

    • Open console into repository secrets folder in mcc machine
    • Customize and use the following SCP command to copy kubeconfig files:
    # Set environment variables
    export REPO_DIR="$HOME/mpb-1100"
    # Inject and execute install in k8s nodes
    echo && echo "******** SOE - START of execution ********" && echo \
    && cd ${REPO_DIR}/secrets \
    && scp kos@k8s-mod-master.cskylab.net:~/.kube/config ${REPO_DIR}/secrets/config-k8s-mod \
    && scp kos@k8s-pro-master.cskylab.net:~/.kube/config ${REPO_DIR}/secrets/config-k8s-pro \
    && echo && echo "******** EOE - END of execution ********" && echo

    Modify clustername in k8s kubeconfig files

    • Edit config-k8s-mod in your secrets repository folder
    • Change all the entries named kubernetes to k8s-mod
    • Edit config-k8s-pro in your secrets repository folder
    • Change all the entries named kubernetes to k8s-pro
    • Save the files and commit to git repository

    Configure credentials for k8s clusters administration

    • Open console into repository secrets folder in mcc machine

    • Customize and execute this command to create .kube configuration directory, copy and merge config files:

    # Set environment variables
    export REPO_DIR="$HOME/mpb-1100"
    # Configure credentials for k8s clusters
    echo && echo "******** SOE - START of execution ********" && echo \
    && cd ${REPO_DIR}/secrets \
    && ls ./config-k8s-mod \
    && ls ./config-k8s-pro \
    && mkdir ${HOME}/.kube \
    ; echo \
    && cp -av ${REPO_DIR}/secrets/config-k8s-mod  ${HOME}/.kube/ \
    && cp -av ${REPO_DIR}/secrets/config-k8s-pro  ${HOME}/.kube/ \
    && KUBECONFIG=/${HOME}/.kube/config-k8s-mod:/${HOME}/.kube/config-k8s-pro kubectl config view --merge --flatten > /${HOME}/.kube/config \
    && chmod 600 ${HOME}/.kube/* \
    && ls -lah ${HOME}/.kube/ \
    && echo && echo "******** EOE - END of execution ********" && echo

    Install kubernetes extension to VS Code

    To install kubernetes extension in remote VS Code:

    • Click on Extensions icon at the sidebar
    • Search for Kubernetes extension
    • Click Install in SSH to complete extension installation

    To check kubernetes extension:

    • Click on installed Kubernetes extension icon at the sidebar
    • Select remote kubeconfig by cliking on CLUSTERS ... -> Set kubeconfig -> /home/kos/.kube/config
    • Check status of both clusters k8s-mod and k8s-pro

    Join k8s-nodes to clusters

    • From VS Code Remote connected to mcc, open terminals at each k8s node and connect inside the machine with ./csconnect.sh:

      • k8s-mod-n1
      • k8s-mod-n2
      • k8s-mod-n3
      • k8s-mod-n4
      • k8s-pro-n1
      • k8s-pro-n2
      • k8s-pro-n3
      • k8s-pro-n4

    • Copy kubeadm command from secrets/k8s-mod-join-command and execute it with sudo privileges in:

      • k8s-mod-n1 console
      • k8s-mod-n2 console
      • k8s-mod-n3 console
      • k8s-mod-n4 console

    • Copy kubeadm command from secrets/k8s-pro-join-command and execute it with sudo privileges in:

      • k8s-pro-n1 console
      • k8s-pro-n2 console
      • k8s-pro-n3 console
      • k8s-pro-n4 console

    Check k8s clusters status

    • In your installation repository, go to root folder and check both cluster status by customizing and running the following command:
    # Set environment variables
    export REPO_DIR="$HOME/mpb-1100"
    # k8s clusters status
    echo && echo "******** SOE - START of execution ********" && echo \
    && cd ${REPO_DIR}/cs-mod/k8s-mod/ \
      && pwd \
      && echo \
      && direnv allow \
      && kubectl --kubeconfig=$HOME/.kube/config-k8s-mod get nodes \
      && echo \
      && kubectl --kubeconfig=$HOME/.kube/config-k8s-mod get pod --all-namespaces -o wide \
      && echo \
      && echo \
    && cd ${REPO_DIR}/cs-pro/k8s-pro/ \
      && pwd \
      && echo \
      && direnv allow \
      && kubectl --kubeconfig=$HOME/.kube/config-k8s-pro get nodes \
      && echo \
      && kubectl --kubeconfig=$HOME/.kube/config-k8s-pro get pod --all-namespaces -o wide \
    && echo && echo "******** EOE - END of execution ********" && echo

    Configure LVM data services at k8s nodes

    Create volgroup and Thin LVM in all nodes

    • From VS Code Remote connected to mcc, open terminals at each k8s-mod node and connect inside the machine with ./csconnect.sh:

      • k8s-mod-n1
      • k8s-mod-n2
      • k8s-pro-n1
      • k8s-pro-n2

    • Create volgroup and Thin LVM in each node:

    # Create volgroup and thin LVM
    sudo cs-volgroup.sh -qm create

    Create data services k8s-mod-n1 & k8s-mod-n2

    • From VS Code Remote connected to mcc, open terminals at each k8s-mod node and connect inside the machine with ./csconnect.sh:
      • k8s-mod-n1
      • k8s-mod-n2
    • Create LVM data services in each node by running the following command:
    # Create LVM data services for k8s-mod-n1/n2
    echo && echo "******** SOE - START of execution ********" && echo \
    && sudo cs-lvmserv.sh -m create -qd "/srv/gitlab" \
      && mkdir "/srv/gitlab/data/postgresql" \
      && mkdir "/srv/gitlab/data/minio" \
      && mkdir "/srv/gitlab/data/redis-master" \
      && mkdir "/srv/gitlab/data/gitaly" \
      && mkdir "/srv/gitlab/data/task-runner"\
    && sudo cs-lvmserv.sh -m create -qd "/srv/harbor" \
      && mkdir "/srv/harbor/data/chartmuseum" \
      && mkdir "/srv/harbor/data/jobservice" \
      && mkdir "/srv/harbor/data/postgresql" \
      && mkdir "/srv/harbor/data/redis" \
      && mkdir "/srv/harbor/data/registry" \
      && mkdir "/srv/harbor/data/trivy"\
    && sudo cs-lvmserv.sh -m create -qd "/srv/keycloak" \
      && mkdir "/srv/keycloak/data/postgresql" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/miniostalone" \
      && mkdir "/srv/miniostalone/data/miniostalone" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/nextcloud" \
      && mkdir "/srv/nextcloud/data/ncdata" \
      && mkdir "/srv/nextcloud/data/mariadb" \
      && mkdir "/srv/nextcloud/data/redis-master" \
    && echo && echo "******** EOE - END of execution ********" && echo

    Create minio-tenant data services

    • From VS Code Remote connected to mcc, open terminals at each k8s-mod node and connect inside the machine with ./csconnect.sh:
      • k8s-mod-n1
      • k8s-mod-n2
      • k8s-mod-n3
      • k8s-mod-n4
    #
    # Create minio-tenant LVM data services
    #
    echo && echo "******** SOE - START of execution ********" && echo \
    && ssh kos@k8s-mod-n1.cskylab.net \
      'sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s00-d00" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s00-d01" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s00-d02" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s00-d03"' \
    && ssh kos@k8s-mod-n2.cskylab.net \
      'sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s01-d00" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s01-d01" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s01-d02" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s01-d03"' \
    && ssh kos@k8s-mod-n3.cskylab.net \
      'sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s02-d00" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s02-d01" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s02-d02" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s02-d03"' \
    && ssh kos@k8s-mod-n4.cskylab.net \
      'sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s03-d00" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s03-d01" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s03-d02" \
    && sudo cs-lvmserv.sh -m create -qd "/srv/minio-tenant-s03-d03"' \
    && echo && echo "******** EOE - END of execution ********" && echo

    Activate Rsync mirroring and Restic backups in k8s-mod-n1

    • Go to k8s-mod-n1 folder in installation repository
    • Rename tpl-cs-cron_scripts file to cs-cron_scripts
    • Open console in root folder at your repository
    • Inject & deploy configuration to k8s-mod-n1 with:
    # Set environment variables
    export REPO_DIR="$HOME/mpb-1100"
    # Inject & deploy k8s-mod-n1 connfiguration
    echo && echo "******** SOE - START of execution ********" && echo \
    && cd ${REPO_DIR}/cs-mod/k8s-mod-n1/ \
      && pwd \
      && echo \
      && ./csinject.sh -qdm config \
    && echo && echo "******** EOE - END of execution ********" && echo

    License

    Copyright © 2021 cSkyLab.com ™

    Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.