Ubuntu server 22.04 Kubernetes node

This template contains the configuration files needed to run a Kubernetes node on Ubuntu Server 22.04 LTS.

Persistent Volumes on local storage are supported by LVM services.

Template override parameters

File _values-tpl.yaml contains template configuration parameters and their default values:

#
#   _values-tpl.yaml
#
#   cskygen template default values file
#
_tplname: ubt2204srv-k8s
_tpldescription: Ubuntu server 22.04 kubernetes node
_tplversion: 24-04-20
#
# Values to override
#
## Kubernetes version to install
k8s_version: "1.29.3-1.1"
## Machine related configuration values
machine:
  hostname: ubt2204srv-k8s
  domainname: cskylab.net
  localadminusername: kos
  localadminpassword: "NoFear21"
  timezone: "UTC"
  networkinterface: enp1s0
  ipaddress: 192.168.82.9
  netmask: 24
  gateway4: 192.168.82.1
  searchdomainnames:
    - cskylab.net
  nameservers:
    - 192.168.82.1
  ## Setup directory where configuration files will be injected
  setupdir: "/etc/csky-setup"
  systemlocale: "C.UTF-8"
  systemkeyboard: "us"
  
restic:
  ## Restic password is mandatory to access repository
  password: 'NoFear21'
  ## Restic repositories can be located in local paths, sftp paths and s3 buckets
  ## Local path example:
  # repo: '/srv/restic/mydir'
  ## S3 example:
  # repo: 's3:https://backup.cskylab.net/restic/mydir'
  ## sftp example:
  repo: 'sftp:kos@hostname.cskylab.net:/media/data/restic/mydir'
  
  ## S3 Bucket access and secret keys must be specified for S3 located repositories
  aws_access: 'restic_rw'
  aws_secret: 'iZ6Qpx1WiqmXXoXKxBxhiCMKWCsYOrgZKr'

Prerequisites

• Physical or virtual machine with two or more disks:
  • Disk 1: Ubuntu server 22.04 installed up and running.
  • Additional disks: Free and ready to be managed by LVM.
• Package openssh-server must be installed to allow remote ssh connections.

To run kubernetes node, (master or worker) the minimum recommended system requirements are:

• 2 CPU
• 4 GB RAM

Check kubernetes version

Before installation, you can choose the k8s version to install by changing in cs-deploy.sh script the variable k8s_version

To see available versions, connect to the machine and run:

# List available kubeadm versions
sudo apt update && sudo apt-cache madison kubeadm

NOTE: You must choose the kubernetes version, before running package installation with ./csinject.sh -d -m install script.

How-to guides

Inject & Deploy configuration

To install and configure the machine, open a terminal from the machine configuration directory in the management repository, and perform the following configuration steps:

1. SSH Authentication and sudoers file

# Run csinject.sh in [ssh-sudoers] execution mode
./csinject.sh -k

NOTE: If IP address has not been previously set in cloud-init or net-config, use -r IPaddress until network configuration files are deployed.

This step injects ssh key and sudoers file into the machine.

Required before other configuration options. Its purpose is to allow automated and passwordless logins by using ssh protocol.

If ssh key has not been injected before, you must provide the password for username {{ .machine.localadminusername }}@{{ .machine.hostname }} twice:

• First one to install ssh key (ssh-copy-id).
• Second one to deploy the sudoers file.

2. Network configuration

# Run csinject.sh to inject & deploy configuration in [net-config] deploy mode
./csinject.sh -d -m net-config

NOTE: If IP address has not been previously set in cloud-init or net-config, use -r IPaddress until network configuration files are deployed.

This step deploys network configuration files that allow the machine to operate with specific IP address and hostname. Cloud-init configuration will be disabled from the next start.

Reboot is recommended when finished.

3. Install packages, updates and configuration tasks

# Run csinject.sh to inject & deploy configuration in [install] deploy mode
./csinject.sh -d -m install

This step performs:

• Package installation
• Updates
• Configuration files deployment
• Configuration tasks

Required to run at least once in order to complete proper configuration. Reboot is recommended when finished.

4. Configuration tasks

# Run csinject.sh to inject & deploy configuration in [config] deploy mode (default)
./csinject.sh -d

When configuration needs to be changed, this mode redeploys all configuration files into the machine, executing again all configuration tasks.

5. Connect and operate

# Run csconnect.sh to establish a ssh session with sudoer (admin) user
./csconnect.sh

To run scripts and operate from inside the machine, establish an ssh connection with administrator (sudoer) user name {{ .machine.localadminusername }}@{{ .machine.hostname }}.

Storage services

Manage disk volume groups

Usage:

  sudo cs-volgroup.sh -h

List status:

  sudo cs-volgroup.sh -l

Create volume group + LVM thin pool:

# Create volume group with default variable values
  sudo cs-volgroup.sh -m create

If only one volume group is going to be created, you can edit the script and update in section Variable initialization the values for volume group name vg_name and block device names block_dev_names.

Otherwise, you can specify these values in the following way:

  # Create volume group with name "ssd"
  # from disk devices "/dev/sdb /dev/sdc /dev/sdd"
    sudo cs-volgroup.sh -m create -d "/dev/sdb /dev/sdc /dev/sdd" -v ssd

Delete volume group + LVM thin pool:

# Delete volume group with default variable values
  sudo cs-volgroup.sh -m delete
  # Delete volume group with name "ssd"
  # from disk devices "/dev/sdb /dev/sdc /dev/sdd"
    sudo cs-volgroup.sh -m delete -d "/dev/sdb /dev/sdc /dev/sdd" -v ssd

Before deleting volume groups, you must delete the LV's inside.

Manage Thin Provisioning LVM data services

Usage:

  sudo cs-lvmserv.sh -h

List status:

  sudo cs-lvmserv.sh -l

Create LV data service (Thin logical volume + directory mount):

  # Create thin logical volume data service "/srv/mydir"
  # in default volume group.
    sudo cs-lvmserv.sh -m create -d /srv/mydir

Create snapshot:

# Create snapshot of data service "/srv/mydir"
# in default volume group.
# Snapshot will be mounted read-only in /tmp/mydir_snap
  sudo cs-lvmserv.sh -m snap-create -d /srv/mydir

Remove snapshot:

# Remove snapshot of data service "/srv/mydir"
# in default volume group
  sudo cs-lvmserv.sh -m snap-remove -d /srv/mydir

Merge snapshot (Rollback to snapshot status):

# Merge snapshot of data service "/srv/mydir"
# in default volume group
  sudo cs-lvmserv.sh -m snap-merge -d /srv/mydir

Delete LV data service:

# Delete thin logical volume data service "/srv/mydir"
# in default volume group.
  sudo cs-lvmserv.sh -m delete -d /srv/mydir

Free space of unused blocks inside thin-pools:

# Free space inside thin-pools
  sudo cs-lvmserv.sh -m trim-space

Repair LVM thin-pool

1. Unmount all LVM in the volume group

2. List LVM to see vg and lv names

sudo lvscan

3. Deactivate volume group

sudo lvchange -an vg

4. Repair thin-pool

sudo lvconvert --repair  vg/tpool

5. Activate volume group

sudo lvchange -ay vg

6. Mount LVM and check data

To learn more see the following procedure: https://smileusd.github.io/2018/10/12/repair-thinpool/

Rsync data replication

Note: Prior to operate rsync with a remote host, you must insert the root public key for ssh authentication and passwordless login as sudoer user in the remote host. From a console inside the machine you must run sudo ssh-copy-id {{ .machine.localadminusername }}@hostname.domain.com

Usage:

  sudo cs-rsync.sh -h

RSync data from local directory TO remote directory (snapshot automatically created and removed):

# RSync data service "/srv/mydir"
# TO same remote directory at host "hostname.cskylab.net"
  sudo cs-rsync.sh -m rsync-to -d /srv/mydir \
  -t hostname.cskylab.net

RSync data FROM remote directory to local directory:

# RSync data service "/srv/mydir"
# FROM remote directory "/srv/mydir" at host "hostname.cskylab.net"
  sudo cs-rsync.sh -m rsync-from -d /srv/mydir \
  -t hostname.cskylab.net

Restic data backup and restore

The script cs-restic.sh is designed as a wrapper to execute restic in LVM data services with snapshot operations.

Usage:

  sudo cs-restic.sh -h

Credentials for default Restic environment and S3 bucket if used, are stored in the following variables:

# restic-environment
export RESTIC_REPOSITORY="{{ .restic.repo }}"
export RESTIC_PASSWORD="{{ .restic.password }}"
export AWS_ACCESS_KEY_ID="{{ .restic.aws_access }}"
export AWS_SECRET_ACCESS_KEY="{{ .restic.aws_secret }}"

Note: Prior to operate restic with a remote host in sftp mode, you must insert the root public key for ssh authentication and passwordless login as sudoer user in the remote host. From a console inside the machine you must run sudo ssh-copy-id {{ .machine.localadminusername }}@hostname.domain.com

Create repository:

To create Restic repository (Directory must exist):

  # Create repository
    sudo cs-restic.sh -m repo-init -r "{{ .restic.repo }}"

Backup LVM data service:

Data services must have been previously created with cs-lvmserv.sh. When making backup of a data service, the script automatically creates a snapshot and removes it when finished.

To backup a data service directory:

  # Backup data service /srv/mydir
    sudo cs-restic.sh -m restic-bck -d /srv/mydir \
      -r "{{ .restic.repo }}"

List snapshots in repository:

To list snapshots in a repository:

  # List snapshots in repository
    sudo cs-restic.sh -m restic-list -r "{{ .restic.repo }}"

Restore snapshot:

Data service directory must be empty. Otherwise you should use restic-mount and rsync data from mount point.

Restic restore does not show progress information in console. To see progress, open another terminal and run sudo du -sh /srv/mydir

To restore from specific snapshot E.G.:(2a3dff53):

  # Restore from specific snapshot
    sudo cs-restic.sh -m restic-res 2a3dff53 -d /srv/mydir \
      -r "{{ .restic.repo }}"

Mount restic repository snapshot:

Restic repository can be locally mounted, by default in /tmp/restic directory, in order to manually explore and copy files or to use the script cs-rsync.sh -m rsync-from to synchronize data from any snapshot in the repository to local data service directory.

To mount restic repository:

  # Mount repository in local directory /tmp/restic
    sudo cs-restic.sh -m restic-mount -r "{{ .restic.repo }}"

After a restic-mount operation, console will be freezed while repository is mounted. To access the contents of the repository, open another terminal and refer to the locally mounted directory /tmp/restic

Use of cs-rsync from mount point:

You can use the script cs-rsync.sh to restore a complete snapshot from mount point.

Examples:

  # RSync data service "/srv/mydir"
  # FROM Restic snapshot /tmp/restic/snapshots/latest
    sudo cs-rsync.sh -m rsync-from -r "/tmp/restic/snapshots/latest" \
      -d /srv/mydir
  # RSync data service "/srv/mydir"
  # FROM Restic snapshot /tmp/restic/snapshots/2021-04-03T21\:39\:08+02\:00
    sudo cs-rsync.sh -m rsync-from -r "/tmp/restic/snapshots/2021-04-03T21\:39\:08+02\:00" \
      -d /srv/mydir

Maintain Restic repository with forget option:

The default forget option provided in the script is --keep-hourly 6 --keep-daily 31 --keep-weekly 5 --keep-monthly 13 --keep-yearly 10. You can provide a specific forget option (-f) with an specific tag (-t).

It's not recommended to run restic prune with regular backups, since it can be a time-consuming process and it should be planned in a way that it doesn’t interfere with regular backups.

  # Forget snapshots in repository with default forget option and tag mydir
    sudo cs-restic.sh -m restic-forget -t "mydir" -r "{{ .restic.repo }}"

Initialize a new kubernetes cluster

To learn more about kubernetes cluster installation with kubeadm see https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/

Prepare your internal CA files

To use your internal CA, you must copy your certificate and key files to the configuration directory. The following files must exist in your configuration directory:

• ca.crt: CA certificate file
• ca.key: CA key file

NOTE: You must have the files into the k8s-master machine at install time, before initializing the kube with cs-k8init.sh script.

If files have been changed since machine installation, inject & deploy configuration by running:

# Inject & Deploy configuration
./csinject.sh -qd

Single control-plane K8s cluster

Follow these steps to install a single control-plane Kubernetes cluster with kubeadm.

To initialize a new cluster run:

# Initialize a new master node and kubernetes cluster":
sudo cs-k8init.sh -m init-master

Standalone K8s cluster

In standalone mode, master node can hold worloads for application deployment, and it is possible to deploy a single node cluster.

# Initialize a new kubestalone single node kubernetes cluster":
sudo cs-k8init.sh -m init-kubestalone

To display kubernetes cluster status execute:

# Display current cluster status:
sudo cs-k8init.sh -l

Credentials for k8s clusters administration

• Prepare your kubernetes credentials directory:

  If not previously created, you must have in your computer a special directory to hold kubernetes credentials files (kubeconfig files). Go to your home directory, and create it:

  # Create kubernetes credential directory for kubeconfig files
  mkdir ${HOME}/.kube

This directory will contain kubeconfig credential files for each k8s cluster you need to operate with.

• Copy kubeconfig file: You must copy from every k8s master node its credentials file .kube/config to your computer. Use an SCP command and name the config file according to the k8s cluster name, in the following form:

  # Example: Download from host k8s-mod-master.cskylab.com, the kubeconfig file for cluster k8s-mod
  scp kos@k8s-mod-master.cskylab.net:~/.kube/config ${HOME}/.kube/config-k8s-mod

• Edit kubeconfig file and customize names:

  Edit the kubeconfig file ${HOME}/.kube/config-<your_k8s_cluster_name> and change all the entries named kubernetes to your_k8s_cluster_name in the following way:

  Entry:	Change to:	Example
  kubernetes	<your_k8s_cluster_name>	Change kubernetes to k8s-model

• Change cluster administration context: To set the administration context to a specific cluster, you must export the $KUBECONFIG environment variable in the followinng way:

  ## $KUBECONFIG environment variable
  export KUBECONFIG=$HOME/.kube/config-k8s-mod

• Merging kubeconfig files:

  In addition to these single cluster kubeconfig files, you need to merge them into a global default kubeconfig file ${HOME}/.kube/config.

  Since kubeconfig files are structured YAML files, you can’t just append them to get one file. You must use kubectl to merge these files.

  To do so, take the following steps:

  # Backup your current default kubeconfig file
  cp -av ${HOME}/.kube/config ${HOME}/.kube/config.bak
  # Merge kubeconfig files format
  KUBECONFIG=file1:file2:file3 kubectl config view --merge --flatten > ${HOME}/.kube/config
  
  # Merging example for files config-k8s-mod and config-k8s-pro
  KUBECONFIG=${HOME}/.kube/config-k8s-mod:${HOME}/.kube/config-k8s-pro kubectl config view --merge --flatten > ${HOME}/.kube/config

Add node to k8s cluster

This procedure covers how to add a node to an existing cskylab kubernetes cluster.

Additional procedures can be found at:

• https://www.serverlab.ca/tutorials/containers/kubernetes/how-to-add-workers-to-kubernetes-clusters/
• https://kubernetes.io/docs/concepts/architecture/nodes/

Create a new token

Tokens only have a lifespan of 24-hours, preventing you from adding additional nodes after that time limit.

Create a new token using kubeadm. By using the –print-join-command argument, kubeadm will output the token and SHA hash required to securely communicate with the master.

# Create token and print join command
sudo kubeadm token create --print-join-command
# Verify created token
sudo kubeadm token list

Copy the join command to be used in next step.

Join the new node to the cluster

Use the kubeadm join command obtained with our new token to join the new node to the cluster.

Connect to the new node and execute the join command:

# Join the cluster (example)
sudo kubeadm join 192.168.1.130:6443 \
    --token qt57zu.wuvqh64un13trr7x \
    --discovery-token-ca-cert-hash sha256:5ad014cad868fdfe9388d5b33796cf40fc1e8c2b3dccaebff0b066a0532e8723

From the k8s master node or any other kubectl console, list your cluster’s nodes to verify the new node has successfully joined the cluster.

# List existing nodes
kubectl get nodes

Inject ssh keys between k8s nodes

In order to schedule cron-tab copies of local data services to other nodes, you must inject ssh keys from each node, to the other nodes in the cluster.

Note: You will be asked for the password of local admin user (See file secrets/admin-password).

Example: k8s-mod nodes

• Open terminals at each k8s node and connect inside the machine with ./csconnect.sh:
  • k8s-mod-n1
  • k8s-mod-n2
  • k8s-mod-n3
  • k8s-mod-n4

# Inject ssh keys from k8s-mod-n1
sudo ssh-copy-id kos@k8s-mod-n2.cskylab.net
sudo ssh-copy-id kos@k8s-mod-n3.cskylab.net
sudo ssh-copy-id kos@k8s-mod-n4.cskylab.net
# Inject ssh keys from k8s-mod-n2
sudo ssh-copy-id kos@k8s-mod-n1.cskylab.net
sudo ssh-copy-id kos@k8s-mod-n3.cskylab.net
sudo ssh-copy-id kos@k8s-mod-n4.cskylab.net
# Inject ssh keys from k8s-mod-n3
sudo ssh-copy-id kos@k8s-mod-n1.cskylab.net
sudo ssh-copy-id kos@k8s-mod-n2.cskylab.net
sudo ssh-copy-id kos@k8s-mod-n4.cskylab.net
# Inject ssh keys from k8s-mod-n4
sudo ssh-copy-id kos@k8s-mod-n1.cskylab.net
sudo ssh-copy-id kos@k8s-mod-n2.cskylab.net
sudo ssh-copy-id kos@k8s-mod-n3.cskylab.net

Remove node from k8s cluster

NOTE: Before running this procedure, you must check that the node to be removed is not running any persistent volume service in its local storage.

Drain the node

Remove all workloads in the node by running:

# Drain node <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Delete the node

NOTE: kubectl deletecommand deletes the node from cluster without confirmation prompt.

To delete the node from cluster:

# Delete node <node-name>
kubectl delete node <node-name>

After node removal, you can reset the machine to a state which was there before running kubeadm join. Connect to the removed node and execute:

# Reset the node
sudo kubeadm reset

k8s certificate management

Check certificate expiration

• To see certificate expiration, connect to k8s master node and run:

# Check k8s certificate expiration
sudo kubeadm certs check-expiration

Manual certificate renewal

kubeadm renews all the certificates during k8s master node upgrade.

You can renew your certificates manually using the existing CA.

• To renew certificates manually, connect to k8s master node and run:

kubeadm certs renew all

• Restart k8s master node after running the command.

k8s Raspberry Pi

Before create or joining a cluster in a Raspberry Pi, you must follow these steps:

Enable cgroups limit support

For the Raspberry Pi 4, make the following changes to the /boot/firmware/cmdline.txt file:

# Append the cgroups and swap options to the kernel command line
# Note the space before "cgroup_enable=cpuset", to add a space after the last existing item on the line
# Display file before change
cat /boot/firmware/cmdline.txt
# Execute the change
sudo sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 swapaccount=1/' /boot/firmware/cmdline.txt
# Display file after change
cat /boot/firmware/cmdline.txt
# Reboot the Raspberry Pi
sudo reboot

• Reboot the Raspberry Pi

Force delete namespace

To check what's going on at the status of the namespace you can check:

kubectl get ns {NAMESPACE} -o yaml

To force namespace deletion execute:

NAMESPACE={YOUR_NAMESPACE_TO_DELETE}
kubectl proxy &
kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize

Backup kubernetes master configuration and restore it to a new master node

This procedure is intended to perform a backup of kubernetes cluster confirguration that can be restored into a new master node machine.

Step 1: Backup /etc/kubernetes & take snapshot from etcd database

To perform a backup of kubernetes keys & etcd database execute from mcc, open terminal at configuration directory k8s-xxx (cluster level) and execute:

#
# Backup /etc/kubernetes & snapshot etcd database
#
echo \
&& echo "******** START of snippet execution ********" \
&& echo \
&& export BACKUP_DIR="_k8s-backup" \
&& sudo rm -r ./${BACKUP_DIR} \
; mkdir -pv ./${BACKUP_DIR}/etcd \
&& ssh {{ .machine.localadminusername }}@{{ .machine.hostname }}.{{ .machine.domainname }} \
  "echo \
&& sudo rm -r ./${BACKUP_DIR} \
; sudo mkdir -pv ./${BACKUP_DIR} \
&& sudo cp -av /etc/kubernetes/pki ./${BACKUP_DIR} \
&& sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save ./${BACKUP_DIR}/etcd-snapshot.db \
&& sudo chown -R kos:kos ./${BACKUP_DIR} \
&& echo "\
&& scp -rp {{ .machine.localadminusername }}@{{ .machine.hostname }}.{{ .machine.domainname }}:~/${BACKUP_DIR}/* ./${BACKUP_DIR}/ \
&& echo \
&& echo "******** END of snippet execution ********" \
&& echo

Step 2: Restore etcd snapshot locally

The etcd snapshot from previous step will be restored locally, in configuration management directory, running a docker container.

Note: In order to run the docker container, you must run this step from a local or a virtual mcc machine. Container based mcc may not run docker in docker.

#
# Restore etcd snapshot locally
#
echo \
&& echo "******** START of snippet execution ********" \
&& echo \
&& export BACKUP_DIR="_k8s-backup" \
&& docker run --rm \
    -v ./${BACKUP_DIR}:/backup \
    -v ./${BACKUP_DIR}/etcd:/var/lib/etcd \
    --env ETCDCTL_API=3 \
    'k8s.gcr.io/etcd-amd64:3.1.12' \
    /bin/sh -c "etcdctl snapshot restore '/backup/etcd-snapshot.db' ; mv /default.etcd/member/ /var/lib/etcd/" \
&& sudo chown -R ${USER}:${USER} ./${BACKUP_DIR} \
&& echo \
&& echo "******** END of snippet execution ********" \
&& echo

Step 3: Inject configuration into the new master node

After creating and installing the new master node, before executing kubeadm init you must copy the cluster keys & etcd database into the new master node.

From mcc, open terminal at configuration directory k8s-xxx (cluster level) and execute:

#
# Inject configuration into th new master node
#
echo \
&& echo "******** START of snippet execution ********" \
&& echo \
&& export BACKUP_DIR="_k8s-backup" \
&& ssh {{ .machine.localadminusername }}@{{ .machine.hostname }}.{{ .machine.domainname }} \
  "echo \
&& sudo rm -r ./${BACKUP_DIR} \
; sudo mkdir -pv ./${BACKUP_DIR} \
&& sudo mkdir -pv /var/lib/etcd \
&& sudo chown -R kos:kos ./${BACKUP_DIR} \
&& echo "\
&& scp -rp ./${BACKUP_DIR}/* {{ .machine.localadminusername }}@{{ .machine.hostname }}.{{ .machine.domainname }}:~/${BACKUP_DIR} \
&& ssh {{ .machine.localadminusername }}@{{ .machine.hostname }}.{{ .machine.domainname }} \
  "echo \
&& sudo cp -av ./${BACKUP_DIR}/pki /etc/kubernetes/ \
&& sudo chown -R root:root /etc/kubernetes \
&& sudo cp -av ./${BACKUP_DIR}/etcd /var/lib/ \
&& sudo chown -R root:root /var/lib/etcd \
&& echo "\
&& echo \
&& echo "******** END of snippet execution ********" \
&& echo

Step 4: Initialize the cluster

Initialize a the new master node following the procedures in section Initialize a new kubernetes cluster

After initialization, all previous cluster configuration should work, including admin access keys.

Reference:

Ref: https://labs.consol.de/kubernetes/2018/05/25/kubeadm-backup.html https://medium.com/@werkjober/how-to-backup-and-restore-etcd-snapshot-on-kubernetes-cluster-c612ce5b6147 https://k21academy.com/docker-kubernetes/etcd-backup-restore-in-k8s-step-by-step/

Utilities

Passwords and secrets

Generate passwords and secrets with:

# Screen
echo $(head -c 512 /dev/urandom | LC_ALL=C tr -cd 'a-zA-Z0-9' | head -c 16)
# File (without newline)
printf $(head -c 512 /dev/urandom | LC_ALL=C tr -cd 'a-zA-Z0-9' | head -c 16) > RESTIC-PASS.txt

Change the parameter head -c 16 according with the desired length of the secret.

Abridged ‘find’ command examples

When looking for files or directories, you can use the find command like in the following examples:

# Basic case-insensitive commands
# -------------------------------
find /tmp/restic/snapshots/latest/ -type f -iname "*foo*"   # find files under latest snapshot
find /tmp/restic/snapshots/latest/ -type d -iname "*foo*"   # find directories under latest snapshot
find /tmp/restic/snapshots/ -type f -name "foo.txt"         # find a specific file in all snapshots
# find files with different extensions
# ------------------------------------
find . -type f \( -name "*.c" -o -name "*.sh" \)                       # *.c and *.sh files
find . -type f \( -name "*cache" -o -name "*xml" -o -name "*html" \)   # three patterns
# find files that don't match a pattern (-not)
# --------------------------------------------
find . -type f -not -name "*.html"    # find all files not ending in ".html"
# Execute ls -lah with files found
# --------------------------------
find $PWD -type f -iname "*foo*" -exec ls -lah {} \;
# find files bigger than a size
# -----------------------------
find $PWD -type f -size +4G -exec ls -lah {} \;  # find all files bigger than 4GB

USB disk operations

Disk formatting and partitioning:

• Plug USB disk and find the block device:

  # List all disk devices and partitions
  sudo fdisk -l

• Execute fdisk in interactive mode:

  # fdisk interactive mode (Example for device /dev/sdc)
  sudo fdisk /dev/sdc
  
  Welcome to fdisk (util-linux 2.34).
  Changes will remain in memory only, until you decide to write them.
  Be careful before using the write command.
  Command (m for help): 

• Enter m to get a list of all available commands:

  Command (m for help): m
  Help:
    GPT
    M   enter protective/hybrid MBR
    Generic
    d   delete a partition
    F   list free unpartitioned space
    l   list known partition types
    n   add a new partition
    p   print the partition table
    t   change a partition type
    v   verify the partition table
    i   print information about a partition
    Misc
    m   print this menu
    x   extra functionality (experts only)
    Script
    I   load disk layout from sfdisk script file
    O   dump disk layout to sfdisk script file
    Save & Exit
    w   write table to disk and exit
    q   quit without saving changes
    Create a new label
    g   create a new empty GPT partition table
    G   create a new empty SGI (IRIX) partition table
    o   create a new empty DOS partition table
    s   create a new empty Sun partition table
  Command (m for help): 

• Enter g to create a new empty GPT partition table:

  Command (m for help): g
  Created a new GPT disklabel (GUID: FB0E06DB-479A-C842-A775-657D4D52BA41).
  Command (m for help): 

• Enter n to create a new partition and accept default values:

  Command (m for help): n
  Partition number (1-128, default 1): 
  First sector (2048-62545886, default 2048): 
  Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-62545886, default 62545886): 
  Created a new partition 1 of type 'Linux filesystem' and of size 29.8 GiB.
  Command (m for help): 

Note: You can change the partition type if needed, using option t change a partition type. (Example: 11 Microsoft basic data to support exfat file system.)

• Enter p to display the new partition table:

  Command (m for help): p
  Disk /dev/sdc: 29.84 GiB, 32023511040 bytes, 62545920 sectors
  Disk model: USB Flash Drive 
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disklabel type: gpt
  Disk identifier: FB0E06DB-479A-C842-A775-657D4D52BA41
  Device     Start      End  Sectors  Size Type
  /dev/sdc1   2048 62545886 62543839 29.8G Linux filesystem
  Command (m for help):   

• Enter w to save the changes:

  Command (m for help): w
  The partition table has been altered.
  Calling ioctl() to re-read partition table.
  Syncing disks.

Format partition & create file system:

• To format partition and create the file system, run:

  # Format partition & create file system (Example for partition /dev/sdc1)
  sudo mkfs.ext4 -L USBDisk /dev/sdc1

Note: You can use analog commands (i.e., mkfs.exfat or mkfs.ntfs) to create other types of file systems.

Repair file system:

You can use the fsck command to repair corrupted file systems.

• To check and repair the filesystem, unmount the partition and run:

  # Check and repair the file system (Example for partition /dev/sdc1)
  sudo fsck.ext4 /dev/sdc1

Note: You can use analog commands (i.e., fsck.exfat or fsck.ntfs) to check or repair other types of file systems.

USB Disk mount:

Disk must be previously partitioned and file system created.

To mount a USB disk in /media directory:

• Find the disk and its UUID

  # List all disk devices and partitions
  sudo fdisk -l
  
  # Find disk UUID
  sudo blkid

Output example for /dev/sdc1:

  Output
  /dev/XXX: xxxxxxxxxxxxxxxxxxxxxxxxx
  /dev/sdc1: UUID="74530e5a-836c-4cf1-963a-2743b296349b" TYPE="ext4" PARTUUID="e7d880eb-f6a6-8648-bb80-21d8dc72b6e0"

• To mount interactively for the current session use the mount command (Example for /dev/sdc1):

  # Mount disk interactively
  sudo mount /dev/sdc1 /media

• To mount at system startup if present, you must update /etc/fstab in the following way (Example for UUID=...):

  # Update fstab to mount disk at startup
  sudo echo "UUID=74530e5a-836c-4cf1-963a-2743b296349b  /media  auto nosuid,nodev,nofail 0 0" | sudo tee -a /etc/fstab
  
  # Mount interactively if not mounted at startup
  sudo mount -a

Create data directory:

Note: As a best practice, you should create a data directory with sudoer user ownership for remote access, in the mounted file system to be referenced in scripts instead of the mounting point. (i.e. You should reference /media/data instead of /media).

• To create data directory and set ownership:

  # Create data directory
  sudo mkdir /media/data
  sudo chown -R {{ .machine.localadminusername }}:{{ .machine.localadminusername }} /media/data

USB Disk unmount:

• To unmount from /mediaand get the disk ready to unplug:

  # Unmount disk interactively
  sudo umount /media

Reference

Scripts

cs-k8init

Purpose:
  Kubernetes cluster initialization.
  Use this script to initialize a kubernetes cluster with kubeadm.
Usage:
  sudo cs-k8init.sh [-l] [-m <execution_mode>] [-h] [-q]
Execution modes:
  -l  [list-status]       - Display current cluster status.
  -m  <execution_mode>    - Valid modes are:
      [init-master]       - Initialize cluster master node.
      [init-kubestalone]  - Initialize kubestalone single node cluster.
Options and arguments:  
  -h  Help
  -q  Quiet (Nonstop) execution.
Examples:
  # Initialize a new master node and kubernetes cluster":
    sudo cs-k8init.sh -m init-master
  # Initialize a new kubestalone single node kubernetes cluster":
    sudo cs-k8init.sh -m init-kubestalone
  # Display current cluster status:
    sudo cs-k8init.sh -l

Tasks performed:

cs-volgroup

Purpose:
  Disk volume group management.
  Use this script to create and delete volume groups
  from one or several disk devices.
  Each volume group will have a thin-pool associated to hold
  data services that will be created with the script "cs-lvmserv.sh".
Usage:
  sudo cs-volgroup.sh [-l] [-m <execution_mode>] [-d  <block_dev_names>]
                      [-v <vg_name>] [-h] [-q]
Execution modes:
  -l  [list-status]     - List volume group and disk status.
  -m  <execution_mode>  - Valid modes are:
  
      [create]          - Create volume group and thin pool.
      [delete]          - Delete volume group and erase disks.
Options and arguments:  
  -d  <block_dev_names> - Disk device names to be added into volume group.
                          (Array quoted list and space separated values)
                          (Optional. Default value)
  -v  <vg_name>         - Volume group name. (Optional. Default value)
  -h  Help
  -q  Quiet (Nonstop) execution.
Examples:
  # List current status
    sudo cs-volgroup.sh -l
  # Create volume group with default variable values 
    sudo cs-volgroup.sh -m create
  # create volume group with name "ssd"
  # from disk devices "/dev/sdb /dev/sdc /dev/sdd"
    sudo cs-volgroup.sh -m create -d "/dev/sdb /dev/sdc /dev/sdd" -v ssd

Tasks performed:

cs-lvmserv

Purpose:
  LVM data services.
  Use this script to create or delete directory data services with independent 
  thin logical volumes suitable for snapshot operations.
  A volume group and thin-pool must have been created before using the 
  script "cs-volgroup.sh".
Usage:
  sudo cs-lvmserv.sh  [-l] [-m <execution_mode>]
                      [-d <data_path>] [-v <vg_name>] [-h] [-q]
Execution modes:
  -l  [list-status]     - List status of existing logical volumes.
  -m  <execution_mode>  - Valid modes are:
  
      [create]          - Create data service.
                          (Thin logical volume + directory mount)
      [delete]          - Delete data service.
                          (Thin logical volume + directory mount)
      [snap-create]     - Create snapshot and mount to read-only directory.
      [snap-remove]     - Remove snapshot and unmount from read-only directory.
      [snap-merge]      - Roll back logical volume to snapshot status
                           and discard all changes.
      [trim-space]      - Free space inside thin-pools
                          discarding unused blocks on all mounted filesystems.
Options and arguments:  
  -d  <data_path>       - Data service directory path.
                          (Thin LV will be mounted on it)
  -v  <vg_name>         - Volume group name. (Default value)
  -h  Help
  -q  Quiet (Nonstop) execution.
Examples:
  # List logical volume status
    sudo cs-lvmserv.sh -l
  # Create thin logical volume data service "/srv/gitlab-postgresql"
  # in default volume group.
    sudo cs-lvmserv.sh -m create -d /srv/gitlab-postgresql
  # Create snapshot of data service "/srv/gitlab-postgresql"
  # in default volume group.
  # Snapshot will be mounted read-only in /tmp/gitlab-postgresql_snap
    sudo cs-lvmserv.sh -m snap-create -d /srv/gitlab-postgresql
  # Remove snapshot of data service "/srv/gitlab-postgresql"
  # in default volume group
    sudo cs-lvmserv.sh -m snap-remove -d /srv/gitlab-postgresql
  # Merge snapshot of data service "/srv/gitlab-postgresql"
  # in default volume group
    sudo cs-lvmserv.sh -m snap-merge -d /srv/gitlab-postgresql
  # Delete thin logical volume data service "/srv/gitlab-postgresql"
  # in default volume group
    sudo cs-lvmserv.sh -m delete -d /srv/gitlab-postgresql
  # Free space inside thin-pools
    sudo cs-lvmserv.sh -m trim-space

Tasks performed:

cs-rsync

Note: Prior to operate rsync with a remote host, you must insert the root public key for ssh authentication and passwordless login as sudoer user in the remote host. From a console inside the machine you must run sudo ssh-copy-id {{ .machine.localadminusername }}@hostname.domain.com

Purpose:
  RSync copies for LVM data services.
  Use this script to perform RSync operations with data services supported by
  thin-logical volumes created with the script "cs-lvmserv.sh".
  Snapshots will be created automatically for rsync-to copies 
  and removed when finished.
Usage:
  sudo cs-rsync.sh  [-m <execution_mode>] [-d <data_path>] [-v <vg_name>] 
                    [-r <remote_data_path>] [-t <hostname.domain.com>] [-h] [-q]
Execution modes:
  -m  <execution_mode>  - Valid modes are:
      [rsync-to]        - RSync data from local directory TO remote directory.
      [rsync-from]      - RSync data FROM remote directory to local directory.
Options and arguments:  
  -d  <data_path>           - Local data service directory path.
  -v  <vg_name>             - Volume group name. (Default value)
  -r  <remote_data_path>    - Remote directory path. (Default is same as local)
  -t  <hostname.domain.com> - Backup host (Default value)
  -h  Help
  -q  Quiet (Nonstop) execution.
Examples:
  # RSync data service "/srv/gitlab-postgresql"
  # TO same remote directory at host "bckpoint.cskylab.com"
    sudo cs-rsync.sh -m rsync-to -d /srv/gitlab-postgresql \
    -t bckpoint.cskylab.com
  # RSync data service "/srv/gitlab-postgresql"
  # FROM remote directory "/srv/gitlab-postgresql" at host "bckpoint.cskylab.com"
    sudo cs-rsync.sh -m rsync-from -d /srv/gitlab-postgresql \
    -t bckpoint.cskylab.com

Tasks performed:

cs-restic

Purpose:
  Restic backup for LVM data services.
  Use this script to perform Restic operations with data services supported by
  thin-logical volumes created with the script "cs-lvmserv.sh".
  Snapshots will be created automatically for backups and removed when finished.
Usage:
  sudo cs-restic.sh [-m <execution_mode>] [-d <data_path>] [-v <vg_name>] [-t <tag_name>] 
                    [-r <RESTIC_REPOSITORY>] [-i <restic_snap_id>]
                    [-p <RESTIC_PASSWORD>] [-a <AWS_ACCESS_KEY_ID>]
                    [-k <AWS_SECRET_ACCESS_KEY>] [-f <forget_options>] [-h] [-q]
Execution modes:
  -m  <execution_mode>   - Valid modes are:
  
      [restic-bck]       - Backup to local or remote repository.
      [restic-list]      - List snapshots in repository.
      [restic-mount]     - Mount repository to /mnt directory.
      [restic-res]       - Restore from repository to directory data service.
                           (Directory must be empty).
      [repo-init]        - Initialize repository (Directory or bucket must exist).
      [restic-forget]    - Maintain repository and remove snapshots with forget option.
Options and arguments:  
  -d  <data_path>             - Local data service directory path. (Mandatory)
  -v  <vg_name>               - Volume group name. (Default value)
  -t  <tag_name>              - Tag name for the snapshot. (Default value)
  -r  <RESTIC_REPOSITORY>     - Restic repository (Default value)
  -i  <restic_snap_id>        - Snapshot ID to restore (Default latest)
  -p  <RESTIC_PASSWORD>       - Restic password (Default value)
  -a  <AWS_ACCESS_KEY_ID>     - S3 access key (Default value)
  -k  <AWS_SECRET_ACCESS_KEY> - S3 secret key (Default value)
  -f  <forget_options>        - Options to execute (Default value)
  -h  Help
  -q  Quiet (Nonstop) execution.
Examples:
  # Backup data service "/srv/gitlab-postgresql"
  # TO local repository "/srv/restic/gitlab-postgresql/" 
    sudo cs-restic.sh -m restic-bck -d /srv/gitlab-postgresql \
                                  -r "/srv/restic/gitlab-postgresql/"
  # Backup data service "/srv/gitlab-postgresql"
  # TO remote sftp repository "sftp:kos@backup.cskylab.com:/srv/restic/gitlab-postgresql/" 
    sudo cs-restic.sh -m restic-bck -d /srv/gitlab-postgresql \
                                  -r "sftp:kos@backup.cskylab.com:/srv/restic/gitlab-postgresql/"
  # Backup data service "/srv/gitlab-postgresql" tagged "gitlab-postgresql"
  # TO shared repository in MinIO bucket "s3:https://minio-standalone.cskylab.com/restic-test/" 
    sudo cs-restic.sh -m restic-bck -d /srv/gitlab-postgresql \
                                  -t gitlab-postgresql -r "s3:https://minio-standalone.cskylab.com/restic-test/"
  # List snapshots in repository "s3:https://minio-standalone.cskylab.com/restic-test/" 
    sudo cs-restic.sh -m restic-list -r "s3:https://minio-standalone.cskylab.com/restic-test/"
  # Restore data service "/srv/gitlab-postgresql"
  # FROM latest snapshot in repository "/srv/restic/gitlab-postgresql" 
    sudo cs-restic.sh -m restic-res -d /srv/gitlab-postgresql \
                                  -r "/srv/restic/gitlab-postgresql"
  # Create repository in directory "/srv/restic/gitlab-postgresql"
    sudo cs-restic.sh -m repo-init -r "/srv/restic/gitlab-postgresql"
  # Maintain repository "/srv/restic/gitlab-postgresql" applying host, tag and forget options
    sudo cs-restic.sh -m restic-forget -r "/srv/bck/gitlab-postgresql" \
              -t "gitlab-postgresql" -f "--keep-hourly 6 --keep-daily 31 --keep-weekly 5 --keep-monthly 13 --keep-yearly 10"
  # Mount repository "/srv/bck/gitlab-postgresql"   
  # in /mnt directory
    sudo cs-restic.sh -m restic-mount -r "s3:https://minio-standalone.cskylab.com/restic-test/"

Tasks performed:

cs-deploy

Purpose:
  Machine installation and configuration deployment.
  This script is usually called by csinject.sh when executing Inject & Deploy
  operations. Exceptionally, it can also be run manually from inside the machine.
Usage:
  sudo cs-deploy.sh [-m <execution_mode>] [-h] [-q]
Execution modes:
  -m  <execution_mode>  - Valid modes are:
  
      [net-config]      - Network configuration. (Reboot when finished).
      [install]         - Package installation, updates and configuration tasks (Reboot when finished).
      [config]          - Redeploy config files and perform configuration tasks (Default mode).
Options and arguments:  
  -h  Help
  -q  Quiet (Nonstop) execution.
Examples:
  # Deploy configuration in [net-config] mode:
    sudo cs-deploy.sh -m net-config
  # Deploy configuration in [install] mode:
    sudo cs-deploy.sh -m install
  # Deploy configuration in [config] mode:
    sudo cs-deploy.sh -m config

Tasks performed:

cs-inject

Note: This script runs from the "DevOps Computer", opening a terminal from the machine configuration directory in the management repository,.

Purpose:
  Inject & Deploy configuration files into remote machine.
  This script runs from the management (DevOps) computer, copying all configuration
  files to the remote machine, and calling the script 'cs-deploy.sh' to run from
  inside the remote machine if 'deploy' mode [-d] is selected.
Usage:
  ./csinject.sh [-k] [-i] [-d] [-m <deploy_mode>] [-u <sudo_username>] [-r <remote_machine>] [-h] [-q]
Execution modes:
  -k  [ssh-sudoers] - install ssh key and sudoers file into the machine. Required before other actions.
  -i  [inject]      - Inject only. Inject configuration files into the machine for manual deployment.
  -d  [deploy]      - Inject & Deploy configuration. Calls 'cs-deploy.sh' to run from inside the machine.
Options and arguments:  
  -m  <deploy_mode>       - Deploy mode passed to 'cs-deploy.sh'. Valid modes are:
      [net-config]        - Network configuration. (Reboot when finished).
      [install]           - Package installation, updates and configuration tasks (Reboot when finished).
      [config]            - Redeploy config files and perform configuration tasks (Default mode).
  
  -u  <sudo_username>     - Remote administrator (sudoer) user name (Default value).
  -r  <remote_machine>    - Machine hostname or IPAddress (Default value).
  -h  Help
  -q  Quiet (Nonstop) execution.
Examples:
  # Copy ssh key and sudoers file into the machine:
    ./csinject.sh -k
  # Inject & Deploy configuration in [net-config] mode:
    ./csinject.sh -dm net-config
  # Inject & Deploy configuration in [install] mode:
    ./csinject.sh -dm install
  # Inject & Deploy configuration in [config] mode (default):
    ./csinject.sh -d

Tasks performed:

cs-connect

Note: This script runs from the "DevOps Computer", opening a terminal from the machine configuration directory in the management repository,.

Purpose:
  SSH remote connection.
  Use this script to remote login into the machine and establish a ssh session.
Usage:
  csconnect.sh [-u <sudo_username>] [-r <remote_machine>] [-h]
Options and arguments:  
  -u  <sudo_username>   - Remote user name (Default value).
  -r  <remote_machine>  - Machine hostname or IPAddress (Default value).
  -h  Help
Examples:
  # Connect to the machine with default values
    ./csconnect.sh
  # Connect to IPAddress with specific user
    ./csconnect.sh -u sudo_username -r 192.168.2.99

Tasks performed:

cs-helloworld

Purpose:
  Sequential block script model.
  Use this script as a model or skeleton to write other configuration scripts.
Usage:
  sudo cs-helloworld.sh [-l] [-m <execution_mode>] [-n <name>] [-h] [-q]
Execution modes:
  -l  [list-status]     - List current status.
  -m  <execution_mode>  - Valid modes are:
      [install]         - Install.
      [remove]          - Remove.
      [update]          - Update and reconfigure.
Options and arguments:  
  -n <name>             - Name of the person to report status.
                          (Optional in list-status. Default value) 
  -h  Help
  -q  Quiet (Nonstop) execution.
Examples:
  # Mode "install":
    sudo cs-helloworld.sh -m install
  # Mode "remove":
    sudo cs-helloworld.sh -m remove
  # Mode "list-status":
    sudo cs-helloworld.sh -l
  # Mode "list-status" with special name to report:
    sudo cs-helloworld.sh -l -n Bond

Tasks performed:

License

Copyright © 2021 cSkyLab.com ™

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.