Longhorn provides native block storage for Kubernetes clusters and can be used with K3s. The K3s worker nodes have a secondary disk attached which will be used by Longhorn to provision storage. Disks attached to the nodes must be prepped. 

First, we will wipe the second disk attached to the node.

worker-01:~# wipefs -a /dev/<second_disk>

Now, format the disk as ext4.

worker-01:~# mkfs.ext4 /dev/<second_disk>

Create a directory to mount the disk. Longhorn will use this directory for data storage.

worker-01:~# mkdir /longhorn-storage

Mount the disk, then add it to /etc/fstab to persist reboot.

worker-01:~# mount /dev/<second_disk> /longhorn-storage/
worker-01:~# echo UUID\=$(findmnt -n -o UUID,TARGET,FSTYPE,OPTIONS /longhorn-storage) 0 0 | tee -a /etc/fstab

open-iscsi and a NFSv4 client is required for Longhorn.

worker-01:~# apt install -y nfs-common open-iscsi

Now, we are ready to install Longhorn using Helm. Additional configuration flags, such as, defaultDataPath can be passed to Helm as described in customize default settings.

helm repo add longhorn https://charts.longhorn.io; helm repo update

helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --set defaultSettings.defaultDataPath="/longhorn-storage"

We can access the Longhorn UI by creating a service from the control node.

control-01:~$ kubectl apply -f longhorn-svc-lb.yaml

longhorn-svc-lb.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: longhorn-svc-lb
  namespace: longhorn-system
spec:
  selector:
    app: longhorn-ui
  type: LoadBalancer
  loadBalancerIP: 10.33.0.210
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: http

#kubernetes #k3s #k8s #longhorn #storage #weekendproject

MetalLB is used to handle load balancing services for bare metal installation of Kubernetes, with LoadBalancer declarative providing network connectivity into the cluster. To get a better understanding of how modular and flexible Kubernetes can be, ServiceLB which ships as the default service load balance used by K3s, can be disabled during installation or through configuration. Once disabled, MetalLB can be setup.

Start with disabling ServiceLB by adding a statement to config.yaml

echo disable:
  - \"servicelb\" >> /etc/rancher/k3s/config.yaml

Using Helm, install MetalLB.

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb

IPAddressPool configures the addresses available for MetalLB to assign to Services deployed in K3s. The IP address selected from the pool effectively act as the “external” address for the service in the cluster.

---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: metallb-address-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.33.0.200-10.33.0.254

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: metallb-l2-adver
  namespace: metallb-system
spec:
  ipAddressPools:
  - default

Now, MetalLB will automatically assign the next available IP address to any Service with configuration type: LoadBalancer

#kubernetes #k3s #k8s #metallb #weekendproject

Building a Kubernetes cluster is a great way to start learning about the power, functionality and purpose this technology provides in the modern containerization stack.

K3s is a light-weight Kuberetes compliant distribution which can run well on low powered devices and can be used on a single node cluster or it can scale up to a multi-node cluster.

This project will start with a 4 node cluster, 1 control and 3 workers. Nodes are virtual machines with 8GB of RAM and 2 – 32GB disks. The first disk will have Debian installed while the second disk will be used for cluster storage.

K3s can easily be installed on the control node with the following command, as described in the documentation

control-01:~$ curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 --disable servicelb --token super_secret_password --node-taint CriticalAddonsOnly=true:NoExecute --bind-address 10.33.0.100

A couple of additional flags are passed to the install script then described in the K3s Quick start guide. Detail on the configuration options can be found on on https://docs.k3s.io/cli/server.

Briefly, —write-kubeconfig-mode 644 will allow non-root users to read /etc/rancher/k3s/k3s.yaml, the k3s configuration file.

—disable servicelb is used to disable the ServiceLB package. Later on, a different load balance manager, MetalLB will be installed.

—token supersecretpassword defines the password that will be used to connect agent nodes to the server.

—node-taint CriticalAddonsOnly=true:NoExecute creates a node taint that prevents pods to running on the control-plane node unless a tolerance value of “CriticalAddonsOnly” exits. A further explanation would be,

By default, server nodes will be schedulable and thus your workloads can get launched on them. If you wish to have a dedicated control plane where no user workloads will run, you can use taints. https://docs.k3s.io/datastore/ha#2-launch-server-nodes

Setup an env variable on control node to simplify using kubectl.

control-01:~$ echo "KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /etc/environment

Install K3s as an Agent on the worker nodes, worker01-03. K3S_URL is the IP of the control node.

worker-01:~$ curl -sfL https://get.k3s.io | K3S_URL=https://10.33.0.100:6443 K3S_TOKEN=super_secret_password sh -

Label each worker to organize the nodes in the cluster.

control-01:~$ kubectl label nodes k3s-worker-01 kubernetes.io/role=worker

The cluster should be operational at this point. We can check the status of the nodes with kubectl get nodes --show-labels.

#kubernetes #k3s #k8s #weekendproject