Computers Can Be Fun

Logging must be configured first to collect telemetry on the K3s Control host. Detailed configuration for setting up API Server logging is provided at https://docs.k3s.io/security/hardening-guide#api-server-audit-configuration.

A logging directory must be created first.

mkdir -p -m 744 /var/lib/rancher/k3s/server/logs

Next, a default audit policy, audit.yaml should be created in /var/lib/rancher/k3s/server

A simple policy manifests where only metadata is logged will look like this:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata

Additional server options are added to the k3s systemd service file.

...
ExecStart=/usr/local/bin/k3s \
    server \
    ...
    '--kube-apiserver-arg=audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' \
    '--kube-apiserver-arg=audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml' \

We should see an audit.log file in /var/lib/rancher/k3s/server/logs/ once the K3s Server service is restarted.

Collect Logs With Splunk OTEL

We are now ready to deploy an updated Splunk OTEL Collector configuration for monitoring the K3s server audit log. This is done by adding extraFileLogs and agent stanzas to the my_customized_values.yaml file used to initially deploy the splunk-otel-collector

logsCollection:
  ...
  extraFileLogs:
    filelog/audit-log:
      include: [/var/lib/rancher/k3s/server/logs/audit.log]
      start_at: beginning
      include_file_path: true
      include_file_name: false
      resource:
        com.splunk.source: /var/lib/rancher/k3s/server/logs/audit.log
        host.name: 'EXPR(env("K8S_NODE_NAME"))'
        com.splunk.sourcetype: kube:apiserver-audit
 agent:
  extraVolumeMounts:
    - name: audit-log
      mountPath: /var/lib/rancher/k3s/server/logs
  extraVolumes:
    - name: audit-log
      hostPath:
        path: /var/lib/rancher/k3s/server/logs

Let's review the important parts of this manifest. First, extraFileLogs defines a new set of logs in the DaemonSet's container to collect through include directive. The resource directive is used to set the various internal Splunk related values for host, source, and sourcetype.

Next, the agent sections configures the DaemonSet to mount the the K3s server logs directory inside the container running the OTEL collector. agent.extraVolumes.hostPath points to the logging path on the Control Node.

Then we can run helm to apply the new values configuration.

helm upgrade -n infra splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector -f my_customized_values.yaml

#splunk #k3s #otel #logging #apiserver

In a Kubernetes cluster, the container logs are kept locally on the worker node where the container is running. To successfully log this information to a central locations, we must collect these log files from each node.

One method is to use a node-level logging agent configured as a DaemonSet with access to the log directory. A DaemonSet are Pods that run on every node in the cluster. This provides the perfect solution for when you want the same tooling deployed across the cluster. Using a DaemonSet allows us to easily collect container logs from Pods on each worker node. In this configuration, I will be using Splunk's distribution of OpenTelemetry (OTEL) Collector to send logs to a centralized Splunk index.

Deploying Splunk OTEL Collector as a DaemonSet on K3s is straight forward using a Helm chart to configure the application. Details can be found on https://github.com/signalfx/splunk-otel-collector-chart.

helm install splunk-otel-collector --values my_customized_values.yaml splunk-otel-collector-chart/splunk-otel-collector

Make sure to setup a HTTP Event Collector (HEC) input on your Splunk instance first. More information on HEC setup can be found in Splunk's documentation: https://docs.splunk.com/Documentation/Splunk/9.0.4/Data/UsetheHTTPEventCollector.

Now, the interesting configurations happen in the my_customized_values.yaml. The endpoint parameter points to your Splunk indexer.

clusterName: "k3s-cluster"
splunkPlatform:
  endpoint: "https://10.0.90.102:8088/services/collector"
  token: "YOUR_HEC_TOKEN"
  index: "kubernetes"
  insecureSkipVerify: true

logsEngine: otel
tolerations:
  - key: CriticalAddonsOnly
    operator: Exists

logsCollection:
  containers:
    # Set useSplunkIncludeAnnotation flag to `true` to collect logs from pods with `splunk.com/include: true` annotation and ignore others.
    # All other logs will be ignored.
    useSplunkIncludeAnnotation: true

Tolerations are configured to allow the DaemonSet to run on the K3s Control node. This allows us to collect log from those containers running on the control plane, including Traefik, and later, audit logs from the API Server. We configure the logsCollection.containers section with useSplunkIncludeAnnotation: true to have more control over which logs are collected. This setting configures the OTEL collector to look for a specific resource annotation for logs to include in collection. By default the Splunk OTEL Collector collects all container logs unless an exclude annotation is found on a resource.

For the most part, I've taken to applying the Splunk Include annotation to namespaces, allowing me to collect logging from pods running application workloads. This allows me slowly ramp up log collection across the cluster depending on requirements need for monitoring.

Create a patch file to be applied:

# Deployment Template spec
spec:
  template:
    metadata:
      annotations:
        splunk.com/include: "true"

Then apply the patch the Namespace, Deployment or StatefulSet you would like to include in logging, for example, Traefik.

kubectl -n kube-system patch deployments.apps traefik --patch-file splunk-otel-patch.yaml

Traefik logs should now be flowing into the kubernetes index in Splunk.

In the same way you would to assign an IP address to a service in a traditional data center, Kubernetes Pod must also have a known address within the cluster. This is created by using a Service resource.

BookStack will have two defined Services, one for the front end, bookstack-svc and a second one for bookstack-db-svc, the SQL database.

bookstack-svc.yaml manifest

---
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: bookstack
    partOf: bookstack
  name: bookstack-svc
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: bookstack
    partOf: bookstack

Key lines here come in the spec: ... section. MetalLB will assign the next available IP from its block when type: LoadBalancer is set. This gives the Services an address external to the cluster.

Next, port: 80 and targetPort: 80 indicates which port the Services exposes and on which port the Pod is listening, respectively.

Finally, the **selector: ... ** crucially describes what Pod(s) should be matched to this services based on a label, a label of app: bookstack.

bookstack-db-svc.yaml manifest

apiVersion: v1
kind: Service
metadata:
  labels:
    partOf: bookstack
  name: bookstack-db-svc
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: mysql
      port: 3306
      protocol: TCP
      targetPort: 3306
  selector:
    app: mariadb
    partOf: bookstack

The Service for bookstacks-db-svc uses type: ClusterIP which assigns an IP address from the clusters pool to allow internal network traffic only.

Deploy these manifests to K3s with: kubectl apply -f bookstack-db-svc.yaml -f bookstack-svc.yaml

We can access BookStack through the IP address MetalLB assigned to bookstack-svc displayed as the External-IP. kubectl get svc bookstack-svc

Longhorn provides native block storage for Kubernetes clusters and can be used with K3s. The K3s worker nodes have a secondary disk attached which will be used by Longhorn to provision storage. Disks attached to the nodes must be prepped. 

First, we will wipe the second disk attached to the node.

worker-01:~# wipefs -a /dev/<second_disk>

Now, format the disk as ext4.

worker-01:~# mkfs.ext4 /dev/<second_disk>

Create a directory to mount the disk. Longhorn will use this directory for data storage.

worker-01:~# mkdir /longhorn-storage

Mount the disk, then add it to /etc/fstab to persist reboot.

worker-01:~# mount /dev/<second_disk> /longhorn-storage/
worker-01:~# echo UUID\=$(findmnt -n -o UUID,TARGET,FSTYPE,OPTIONS /longhorn-storage) 0 0 | tee -a /etc/fstab

open-iscsi and a NFSv4 client is required for Longhorn.

worker-01:~# apt install -y nfs-common open-iscsi

Now, we are ready to install Longhorn using Helm. Additional configuration flags, such as, defaultDataPath can be passed to Helm as described in customize default settings.

helm repo add longhorn https://charts.longhorn.io; helm repo update

helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --set defaultSettings.defaultDataPath="/longhorn-storage"

We can access the Longhorn UI by creating a service from the control node.

control-01:~$ kubectl apply -f longhorn-svc-lb.yaml

longhorn-svc-lb.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: longhorn-svc-lb
  namespace: longhorn-system
spec:
  selector:
    app: longhorn-ui
  type: LoadBalancer
  loadBalancerIP: 10.33.0.210
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: http

#kubernetes #k3s #k8s #longhorn #storage #weekendproject

MetalLB is used to handle load balancing services for bare metal installation of Kubernetes, with LoadBalancer declarative providing network connectivity into the cluster. To get a better understanding of how modular and flexible Kubernetes can be, ServiceLB which ships as the default service load balance used by K3s, can be disabled during installation or through configuration. Once disabled, MetalLB can be setup.

Start with disabling ServiceLB by adding a statement to config.yaml

echo disable:
  - \"servicelb\" >> /etc/rancher/k3s/config.yaml

Using Helm, install MetalLB.

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb

IPAddressPool configures the addresses available for MetalLB to assign to Services deployed in K3s. The IP address selected from the pool effectively act as the “external” address for the service in the cluster.

---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: metallb-address-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.33.0.200-10.33.0.254

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: metallb-l2-adver
  namespace: metallb-system
spec:
  ipAddressPools:
  - default

Now, MetalLB will automatically assign the next available IP address to any Service with configuration type: LoadBalancer

#kubernetes #k3s #k8s #metallb #weekendproject

Building a Kubernetes cluster is a great way to start learning about the power, functionality and purpose this technology provides in the modern containerization stack.

K3s is a light-weight Kuberetes compliant distribution which can run well on low powered devices and can be used on a single node cluster or it can scale up to a multi-node cluster.

This project will start with a 4 node cluster, 1 control and 3 workers. Nodes are virtual machines with 8GB of RAM and 2 – 32GB disks. The first disk will have Debian installed while the second disk will be used for cluster storage.

K3s can easily be installed on the control node with the following command, as described in the documentation

control-01:~$ curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 --disable servicelb --token super_secret_password --node-taint CriticalAddonsOnly=true:NoExecute --bind-address 10.33.0.100

A couple of additional flags are passed to the install script then described in the K3s Quick start guide. Detail on the configuration options can be found on on https://docs.k3s.io/cli/server.

Briefly, —write-kubeconfig-mode 644 will allow non-root users to read /etc/rancher/k3s/k3s.yaml, the k3s configuration file.

—disable servicelb is used to disable the ServiceLB package. Later on, a different load balance manager, MetalLB will be installed.

—token supersecretpassword defines the password that will be used to connect agent nodes to the server.

—node-taint CriticalAddonsOnly=true:NoExecute creates a node taint that prevents pods to running on the control-plane node unless a tolerance value of “CriticalAddonsOnly” exits. A further explanation would be,

By default, server nodes will be schedulable and thus your workloads can get launched on them. If you wish to have a dedicated control plane where no user workloads will run, you can use taints. https://docs.k3s.io/datastore/ha#2-launch-server-nodes

Setup an env variable on control node to simplify using kubectl.

control-01:~$ echo "KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /etc/environment

Install K3s as an Agent on the worker nodes, worker01-03. K3S_URL is the IP of the control node.

worker-01:~$ curl -sfL https://get.k3s.io | K3S_URL=https://10.33.0.100:6443 K3S_TOKEN=super_secret_password sh -

Label each worker to organize the nodes in the cluster.

control-01:~$ kubectl label nodes k3s-worker-01 kubernetes.io/role=worker

The cluster should be operational at this point. We can check the status of the nodes with kubectl get nodes --show-labels.

#kubernetes #k3s #k8s #weekendproject

Finally!

It took some work, but Writefreely is finally running on my K3s cluster.

My starting point was with configurations in writefreely-docker which provided an excellent source of knowledge around building the docker container. Once built, I published the image to a local container repository for K3s to pull from.

I deployed a Pod to keep things simple. The key here is to make sure the securityContext object has fsGroup and runAsUser is set to to the same user defined in the image build process. In this case, the user is daemon (uid=2).

apiVersion: v1
kind: Pod
...
spec:
  securityContext:
    runAsUser: 2
    runAsGroup: 2
    fsGroup: 2
  containers:
    ...

ConfigMap can be used to inject Writefreely's config.ini into the Pod. The container image expects the configuration to be in in /config/config.ini and a directory for the templates and database in /data/. The data directory should be a PersistentVolume.

  containers:
    ...
      volumeMounts:
      - name: data
        mountPath: /data
      - name: config
        mountPath: /config/config.ini
        subPath: config.ini
    ...

The container expects environment variables to configure the admin user on first deployment.

...
containers:
    - name: writefreely
      image: containers.internal.hrck.net/writeas/writefreely:latest
      ...
      env:
        - name: USERNAME
          value: "alex"
        - name: PASSWORD
          value: "<value of admin user's password>"
     ...

Full build configuration and Kubernetes manifests can be found on Github

#kubernetes #k3s #k8s