kubernetes/README.md

# kubernetes

Kubernetes is an open‑source platform that automates the deployment, scaling, and management of containerized applications. It acts as an orchestrator, ensuring your containers run reliably across clusters of machines, handling networking, storage, and updates without downtime.

<https://kubernetes.io>

- [kubernetes](#kubernetes)
  - [kubectl](#kubectl)
    - [Env variables](#env-variables)
    - [Resources](#resources)
  - [Namespaces](#namespaces)
    - [Create namespace](#create-namespace)
    - [namespace stuck on delete](#namespace-stuck-on-delete)
  - [Pods](#pods)
    - [Create an pod](#create-an-pod)
    - [Get Pod](#get-pod)
    - [delete Pod](#delete-pod)
    - [Failed pods](#failed-pods)
    - [delete all pods with prefix](#delete-all-pods-with-prefix)
    - [get not ready pods](#get-not-ready-pods)
    - [OOMKilled](#oomkilled)
    - [network testing](#network-testing)
    - [Attach to an pod](#attach-to-an-pod)
    - [Run command on pod](#run-command-on-pod)
    - [Keep pod Alive](#keep-pod-alive)
  - [Persistent volumes](#persistent-volumes)
    - [create PersistentVolumes - host path](#create-persistentvolumes---host-path)
    - [find persistent volume used pvc](#find-persistent-volume-used-pvc)
    - [Patch pv - change to retain policy](#patch-pv---change-to-retain-policy)
    - [Patch pv - remove finalizers](#patch-pv---remove-finalizers)
    - [check orphans](#check-orphans)
    - [persistent volume claims](#persistent-volume-claims)
    - [Create an pod with PVC](#create-an-pod-with-pvc)
  - [Services Accounts](#services-accounts)
  - [Tokens](#tokens)
  - [cluster role binding](#cluster-role-binding)
  - [configMap](#configmap)
  - [Secrets](#secrets)
    - [Manifest - Opaque / Base64](#manifest---opaque--base64)
    - [Manifest - StringData](#manifest---stringdata)
    - [Inline with heredoc and environment variables](#inline-with-heredoc-and-environment-variables)
    - [substr](#substr)
    - [using as env](#using-as-env)
    - [mounting](#mounting)
  - [annotations](#annotations)
  - [nodes](#nodes)
    - [cordon](#cordon)
  - [taints](#taints)
    - [add taint](#add-taint)
    - [remove taint](#remove-taint)
    - [control plane - NoSchedule](#control-plane---noschedule)
    - [Official \*.kubernetes.io taints](#official-kubernetesio-taints)
  - [statefulset](#statefulset)
    - [statefulset - Set Replicas](#statefulset---set-replicas)
  - [Deployment](#deployment)
    - [Deployment - Set Replicas](#deployment---set-replicas)
    - [Deployment - rollout restart](#deployment---rollout-restart)
  - [Daemonset](#daemonset)
    - [Daemonset - rollout restart](#daemonset---rollout-restart)
  - [services](#services)
  - [certs](#certs)
    - [list all certs](#list-all-certs)
    - [get cert end date](#get-cert-end-date)
  - [service accounts](#service-accounts)
  - [Services DNS Name](#services-dns-name)
  - [core-dns](#core-dns)
    - [Overrides](#overrides)
    - [Remove warning from logs](#remove-warning-from-logs)
  - [Custom Resource Definitions](#custom-resource-definitions)
  - [k3s](#k3s)
    - [Install / Setup](#install--setup)
    - [prune old images](#prune-old-images)
    - [kill all](#kill-all)
    - [Prune ephemeral containerd data](#prune-ephemeral-containerd-data)
    - [Graceful shutdown](#graceful-shutdown)
    - [check system logs](#check-system-logs)
    - [Workarounds \& Fixes](#workarounds--fixes)
    - [klipper-lb](#klipper-lb)
      - [troubleshooting](#troubleshooting)
    - [Containerd state](#containerd-state)
  - [host cli](#host-cli)
    - [host cli - check port usage](#host-cli---check-port-usage)
    - [kill all connections](#kill-all-connections)
  - [cert-manager](#cert-manager)
    - [Removing cert‑manager Metadata from Secrets](#removing-certmanager-metadata-from-secrets)
  - [hosting](#hosting)
    - [misc](#misc)

## kubectl

kubectl is the command‑line tool used to interact with Kubernetes clusters. Think of it as the “remote control” for Kubernetes: it lets you deploy applications, inspect resources, and manage cluster operations directly from your terminal.

### Env variables

``` bash
export KUBECONFIG=/path/to/kubeconfig
```

### Resources

**List all resources:**

```bash
kubectl get all -n kube-system | grep traefik
```

## Namespaces

### Create namespace

Using cli

``` bash
 kubectl create namespace tests 
```

Or using yaml

``` yaml
apiVersion: v1
kind: Namespace
metadata:
  name: namespace-name
  labels:
    name: namespace-name
```

### namespace stuck on delete

**nuke finalizers:**

``` bash
NAMESPACE="NAMESPACE_NAME"

kubectl get namespace ${NAMESPACE} -o json \
  | jq '.spec.finalizers = []' \
  | kubectl replace --raw /api/v1/namespaces/${NAMESPACE}/finalize -f -
```

## Pods

### Create an pod

**Create an ubuntu pod for tty access example:**

``` bash
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-test
  namespace: tests
spec:
  containers:
    - name: ubuntu-test
      image: ubuntu
      # In Kubernetes, the pod stays alive as long as PID 1 is running.
      # so with this options:
      # - It does not exit automatically.
      # - It waits for user input forever.
      # - It behaves like an interactive shell session.
      command: ["sh"]  # PID 1 = interactive shell
      stdin: true # keep STDIN open
      tty: true # allocate a terminal
```

or using only CLI (auto removes on exiting).

``` bash
kubectl run ubuntu-test \
  --image=ubuntu \
  --restart=Never \
  --rm \
  -it -- bash
```

``` bash
kubectl run hello-world-test --image=ubuntu --restart=Never --rm 
```

``` bash
kubectl run hello-world-test --image=hello-world
```

*Example with busybox, good for network testing.*

``` bash
kubectl run net-test \
  --image=busybox \
  --restart=Never \
  --rm -it -- sh
```

*Example with curl, good for network testing.*

``` bash
kubectl run curl-test \
  --image=curlimages/curl \
  --restart=Never \
  --rm -it -- sh
```

**Create an ubuntu pod with and execute command:**

``` bash
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-ls-test
  namespace: tests
spec:
  restartPolicy: Never # executes only one time, no retry on error 
  
#
# nodeName: "serverExample01" # restrict to an specific node
#
  containers:
    - name: ubuntu-seaweedfs-test
      image: ubuntu
      command: ["bash", "-c"]
      args: 
        - "ls -lah /data"

      volumeMounts:
      - name: data
        mountPath: /data

  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-pvc
 ```

**Create an benchmark pod with hostPath volume:**

```yaml
kind: Pod
apiVersion: v1
metadata:
  name: disk-benchmark
  namespace: storage-limbosolutions-com
spec:
  #nodeName: myNode
  containers:
    - name: diskmark
      image: e7db/diskmark
      volumeMounts:
      - mountPath: "/disk"
        name: disk
  volumes:
    - name: disk
      hostPath:
        path: /storage/bench-test
        type: DirectoryOrCreate
```

**Example using yaml and hostNetwork:**

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: dns-test
  namespace: tests
spec:
  hostNetwork: true
  containers:
  - name: dns-test
    image: busybox
    command: ["sh"]
    stdin: true
    tty: true
```

### Get Pod

**Get pod name by label ap:**

```bash
POD_NAME=$(kubectl get pod -l app=myAppName -n appNamespace -o jsonpath='{.items[0].metadata.name}')
echo $POD_NAME
```

**Get pod name by text on description, for example find by ip:**

``` bash
 kubectl get pods -A -o wide | grep 10.0.3.224
```

**Get all pods Shows NAMESPACE, NAME, READY, STATUS, RESTARTS, AGE.**

``` bash
 kubectl get pods -A
```

**Get all pods on specific node.**

``` bash
 kubectl get pods -A -o wide --field-selector spec.nodeName=${NODE_NAME} 
```

### delete Pod

``` bash
kubectl delete pod -n appNamespace -l app=myAppName
```

### Failed pods

List all.

``` bash
kubectl get pods --all-namespaces --field-selector=status.phase=Failed --no-headers | wc -l
```

**Delete all failed pods.**

``` bash
count=0
while true; do
  pods=$(kubectl get pods --all-namespaces --field-selector=status.phase=Failed \
    -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \
    | head -n 500)

  [ -z "$pods" ] && break

  echo "Batch $((++count)): deleting up to 500 pods"

  while read ns name; do
    kubectl delete pod "$name" -n "$ns"
  done <<< "$pods"
done
```

### delete all pods with prefix

``` bash
NAMESPACE=monitoring
kubectl delete pod -n ${NAMESPACE} \
  $(kubectl get pods -n ${NAMESPACE} \
    -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' \
    | grep '^alloy')
```

### get not ready pods

``` bash
kubectl get pods -A -o json \
| jq -r '
  .items[]
  | select(.status.containerStatuses != null)
  | . as $pod
  | $pod.status.containerStatuses[]
  | select(.ready == false)
  | [
      $pod.metadata.namespace,
      $pod.metadata.name,
      (.state.waiting.reason // .state.terminated.reason // "-")
    ]
  | @tsv
'
```

### OOMKilled

**list all OOMKilled:**

``` bash
kubectl get events --all-namespaces | grep -i "OOMKilled"
```

``` bash
kubectl get pods --all-namespaces \
-o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}{end}' \
| grep OOMKilled
```

On cluster node

```bash
sudo dmesg -T | grep -i kill
sudo dmesg -T | grep -Ei "oom|kill"
sudo journalctl -k | grep -Ei "oom|kill"
sudo journalctl -k --since "1 hour ago" | grep -Ei "oom|kill"
sudo journalctl -k -o short-iso | grep -Ei "oom|kill"
```

### network testing

``` bash
kubectl run  -i --tty dns-test --namespace tests --image=busybox --restart=Never -- 
kubectl delete pod dns-test --namespace tests || 0
```

- Execute command inside pod.

``` bash
nslookup google.com
```

### Attach to an pod

Attach connects your terminal to the main process of the container (PID 1), or another running process if specified.

Use it when you want to:

- see the raw output of the main process
- want to send input directly to the main process

``` bash
kubectl attach -it myPodName -n appNamespace
```

``` bash
POD_NAME=$(kubectl get pod -l app=myAppName -n appNamespace -o jsonpath='{.items[0].metadata.name}')
kubectl attach -it ${POD_NAME} -n appNamespace
```

### Run command on pod

``` bash
# sh
MY_APP_NAME=???
NAMESPACE=???
POD_NAME=$(kubectl get pod -l app=$MY_APP_NAME -n $NAMESPACE -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n $NAMESPACE -it ${POD_NAME} -- sh
```

### Keep pod Alive

```yaml
command: ["tail", "-f", "/dev/null"]
```

```yaml
...
command: ["bash", "-c"]
tty: true 
args:
  - |
      while true; do
        sleep 100
      done     
...
```

``` bash
# bash
POD_NAME=$(kubectl get pod -l app=myAppName -n appNamespace -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it ${POD_NAME} -- bash 
```

``` bash
# execute an command like ls
POD_NAME=$(kubectl get pod -l app=myAppName -n appNamespace -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it ${POD_NAME} -- ls /
```

## Persistent volumes

### create PersistentVolumes - host path

``` yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-host-folder
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /path_on_host
    type: DirectoryOrCreate
```

### find persistent volume used pvc

``` bash
NAMESPACE=???
PVC_NAME=???
PV_NAME=$(kubectl get pvc $PVC_NAME -n $NAMESPACE -o jsonpath='{.spec.volumeName}')
echo "${PV_NAME}"
```

### Patch pv - change to retain policy

``` bash
PV_NAME="???"
kubectl patch pv $PV_NAME \
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
```

### Patch pv - remove finalizers

``` bash
PV_NAME="???"
kubectl patch pv $PV_NAME \
  -p '{"metadata":{"finalizers": null}}'
```

### check orphans

**Execute on host:**

``` bash
#/bin/bash
STORAGE_DIR="/var/lib/rancher/k3s/storage"
# clean if already been used
unset PV_PATHS
declare -A PV_PATHS
ORPHANS=()

for name in $(kubectl get pv -o jsonpath='{.items[*].metadata.name}'); do
    path=$(kubectl get pv ${name} -o jsonpath='{.spec.local.path}' 2>/dev/null)
    # check if path is no empty
    [[ -n "${path}" ]] && PV_PATHS["${path}"]="1"
done

for dir in "$STORAGE_DIR"/*; do
    [ -d "$dir" ] || continue
    if [[ -z "${PV_PATHS[$dir]}" ]]; then
        ORPHANS+=("$dir")
    fi
done

if [[ ${#ORPHANS[@]} -eq 0 ]]; then
    echo "No orphans found."
else
    echo "orphans:"
    printf "%s\n" "${ORPHANS[@]}"
fi
```

### persistent volume claims

``` yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-host-folder
spec:
  volumeName: pv-host-folder
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
```

### Create an pod with PVC

**Create an ubuntu pod for tty access example:**

``` bash
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-test
  namespace: tests
spec:
  #### deploy to an specific node
  nodeName: chimera-gluten
  containers:
    - name: ubuntu-test
      image: ubuntu
      # In Kubernetes, the pod stays alive as long as PID 1 is running.
      # so with this options:
      # - It does not exit automatically.
      # - It waits for user input forever.
      # - It behaves like an interactive shell session.
      command: ["sh"]  # PID 1 = interactive shell
      stdin: true # keep STDIN open
      tty: true # allocate a terminal

      volumeMounts:
        - name: data
          mountPath: /data

  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-pvc
```

## Services Accounts

``` bash
kubectl -n headlamp create serviceaccount $ACCOUNT_NAME
```

**List all:**

```bash
kubectl get serviceAccount --all-namespaces
```

**Get Service Account Token:**

```bash
kubectl get secret <secret_name> -o jsonpath='{.data.token}' | base64 -d
```

```bash
kubectl get secret <secret_name> -o jsonpath='{.data.token}' | base64 -d > ./service-account-secret-base64
```

**Get Cluster certificate Base64:**

```bash
kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' 
```

## Tokens

``` bash
kubectl create token $SERVICE_ACCOUNT_NAME -n $NS
```

``` yaml
apiVersion: v1
kind: Secret
metadata:
  name: name
  namespace: ns
  annotations:
    kubernetes.io/service-account.name: "service-account-name"
type: kubernetes.io/service-account-token
```

## cluster role binding

``` bash
kubectl create clusterrolebinding $NAME --serviceaccount=$NS:$SERVICE_ACCOUNT 
```

## configMap

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config
data:
  app.conf: |
    key=value
  nested/file.txt: |
    hello
```

## Secrets

### Manifest - Opaque / Base64

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: secret-name
  namespace: namespace-name
type: Opaque
data:
  SERVER_ADDRESS: MTI3LjAuMC4x # 127.0.0.1 BASE64
```

### Manifest - StringData

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: secret-name
  namespace: namespace-name
stringData:
  SERVER_ADDRESS: 127.0.0.1
```

### Inline with heredoc and environment variables

``` bash
SERVER_ADDRESS=127.0.0.1
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: secret-name
  namespace: namespace-name
stringData:
  SERVER_ADDRESS: ${SERVER_ADDRESS}
EOF
```

### substr

**yaml secret template:**

``` yaml
# ./secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: secret-name
  namespace: namespace-name
stringData:
  SERVER_ADDRESS: ${SERVER_ADDRESS}
```

``` bash
export SERVER_ADDRESS="127.0.1"
envsubst < ./secret.yaml | kubectl apply -f -
```

**env file and envsubst:**

``` bash
#---
# ./.env
# content:
# SERVER_ADDRESS=127.0.0.1
#---
set -a
source ./.env
set +a
envsubst < ./secret.yaml | kubectl apply -f -

```

### using as env

``` yaml
containers:
  image: myimage
  env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: my-secret
        key: PASSWORD
```

### mounting

``` yaml
containers:
  - name: test
    volumeMounts:
      - name: secret-name
        mountPath: /secret
        subPath: mySecret
      
volumes:
  - name: secret-name
    secret:
      secretName: secret-name
      # required if secret is ah ssh identity
      # defaultMode: 0400

```

## annotations

``` bash
kubectl annotate secret test-your-domain-tls \
  -n test-namespace \
  replicator.v1.mittwald.de/replicate-to="namespace-1,namespace-2" --overwrite
```

## nodes

**Get nodes info:**

```bash
kubectl get nodes -o wide
```

**remove annotation:**

``` bash
kubectl annotate node <NODE_NAME> <ANNOTATION_NAME>-
```
### cordon

``` bash
NODE="????"
kubectl cordon ${NODE}
```

Marks a node as unschedulable.

- No new pods will be scheduled on that node
- Existing pods are not affected
- Even after a reboot, existing pods return to the same node
- Used for temporary maintenance (updates, debugging, draining prep)
- Kubernetes automatically adds the taint:
  `node.kubernetes.io/unschedulable:NoSchedule`

``` bash
NODE="???"
kubectl uncordon ${NODE}
```

Reverses the cordon.

- The node becomes schedulable again
- New pods can land on it
- Existing pods remain untouched

## taints

**get node taints:**

``` bash
kubectl describe node <NODE_NAME> | grep taint
```

### add taint

``` bash
NODE="????"
TAINT="infra.mydomain.com/dedicated=role:NoSchedule"
kubectl taint nodes ${NODE} ${TAINT}
```

### remove taint

``` bash
NODE="chimera-deepstate"
TAINT="infra.mydomain.com/dedicated=role:NoSchedule"
kubectl taint nodes ${NODE} ${TAINT}-
```

### control plane - NoSchedule

``` bash
NODE="????"
kubectl taint nodes ${NODE} node-role.kubernetes.io/control-plane=:NoSchedule
```

### Official *.kubernetes.io taints

**Node condition taints (automatic):**

- node.kubernetes.io/not-ready - Node is NotReady
- node.kubernetes.io/unreachable - Node unreachable
- node.kubernetes.io/out-of-disk - Node out of disk
- node.kubernetes.io/memory-pressure - Memory pressure
- node.kubernetes.io/disk-pressure - Disk pressure
- node.kubernetes.io/network-unavailable- Network unavailable
- node.kubernetes.io/unschedulable - Node was cordoned
- node.kubernetes.io/ready - Node is ready (rarely used as taint)

**Eviction taints (used by kubelet):**

- node.kubernetes.io/pid-pressure - Too many processes
- node.kubernetes.io/unschedulable - Node cordoned
- node.kubernetes.io/taint-effect-no-execute - NoExecute taints

**Role taints (official, safe to use):**

- node-role.kubernetes.io/control-plane - Control-plane node
- node-role.kubernetes.io/master - Legacy control-plane

Everything else in *.kubernetes.io is reserved and should not be used.


## statefulset

### statefulset - Set Replicas

```bash
kubectl patch statefulset <statefulset-name>  \
  -p '{"spec":{"replicas":0}}'
```

## Deployment

### Deployment - Set Replicas

``` bash
DEPLOYMENT_NAME="???"
# example with 0 to "disable" deployment
kubectl scale deployment ${DEPLOYMENT_NAME} --replicas=0
```

```bash
kubectl patch deployment <deployment-name> \
  -p '{"spec":{"replicas":0}}'
```

### Deployment - rollout restart

```bash
NAME="???"
NAMESPACE="???"
kubectl rollout restart deployment $NAME -n $NAMESPACE
```

## Daemonset

### Daemonset - rollout restart

```bash
NAME="???"
NAMESPACE="???"
kubectl rollout restart daemonset $NAME -n $NAMESPACE
```

## services

**nuke finalizers:**

``` bash
SERVICE="traefik"
NAMESPACE="traefik-public"
kubectl patch service ${SERVICE} \
  -n ${NAMESPACE} \
  --type=merge \
  -p '{"metadata":{"finalizers":[]}}'
```

## certs

### list all certs

```bash
kubectl get cert -n default
```

### get cert end date

``` bash
kubectl get secret certificate-name-tls -o "jsonpath={.data['tls\.crt']}" | base64 --decode | openssl x509 -enddate -noout
```

## service accounts

**Get service account token:**

```bash
kubectl get secret continuous-deploy -o jsonpath='{.data.token}' | base64 -d
```

## Services DNS Name

Kubernetes automatically provides DNS names for Services and Pods, and CoreDNS serves these records. This allows workloads to communicate using stable, predictable names instead of changing IP addresses.

```text
<service-name>.<namespace>.svc.<cluster-domain>
<SERVICE_NAME>.<NAMESPACE>.svc.cluster.local
```

## core-dns

### Overrides

``` yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  empty.server: |
    # empty file, to remove warning: No files matching import glob pattern: /etc/coredns/custom/*.server
  split_dns.override: |
    rewrite name exact requested.domain.tldr target.domain.tldr
    # wild card. works like *.requested.domain.tldr 
    rewrite name regex (.*)\.requested\.domain\.tldr target.domain.tldr
```

### Remove warning from logs

```log
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
```

1. Apply on kubernetes

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  log.override: |
    #
  stub.server: |
    #
```

## Custom Resource Definitions

- **Definition:** A Custom Resource Definition (CRD) is an extension of the Kubernetes API.

- **Purpose:** They allow you to define new resource kinds (e.g., Database, Backup, FooBar) that behave like native Kubernetes objects.

- **Analogy:** By default, Kubernetes understands objects like Pods and Services. With CRDs, you can add your own object types and manage them with kubectl just like built‑in resources

**List traefik CRDS:**

```bash
kubectl get crds | grep traefik
```


## k3s

K3s is a lightweight, certified Kubernetes distribution designed to run in resource‑constrained environments such as edge devices, IoT appliances, and small servers. It simplifies installation and operation by packaging Kubernetes into a single small binary, while still being fully compliant with the Kubernetes API.

🌐 What K3s Is

- Definition: K3s is a simplified Kubernetes distribution created by Rancher Labs (now part of SUSE) and maintained under the CNCF.
- Purpose: It’s built for environments where full Kubernetes (K8s) is too heavy — like Raspberry Pis, edge servers, or CI pipelines.
- Size: The entire distribution is packaged into a binary under ~70MB.

### Install / Setup

**Default master installation:**

``` bash
curl -sfL https://get.k3s.io | sh -
```

Install specific version and disable:

- flannel (alternative example calico, cilium)
- servicelb (alternative example metallb, cilium)
- traefik (then install using helm chart or custom manifests for better control)

```bash
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.33.3+k3s1 INSTALL_K3S_EXEC="--flannel-backend=none \
--disable-network-policy \
--disable=servicelb \
--disable=traefik" \
 sh -
```

### prune old images

prune orphan images, execute on kubernetes host node

```bash
crictl rmi --prune
```

```bash
sudo ctr --address /run/k3s/containerd/containerd.sock images prune --all
```

### kill all

``` bash
k3s-killall.sh 
```

If k3s using mount. Force kill by mount

```bash
kill -9 $(lsof +f -- /mount | awk 'NR>1 {print $2}')
```

### Prune ephemeral containerd data

Useful when testing image‑cache behavior or forcing a full image repull on the next reconciliation.
This removes all containerd state, including images, snapshots, and metadata.

``` bash
# Stop k3s service
systemctl stop k3s

# Remove all containerd data (images, snapshots, metadata)
rm -rf /var/lib/rancher/k3s/agent/containerd

# Start k3s service
systemctl start k3s
```

After this, k3s will start with a completely clean containerd state, and all images will be pulled again as workloads reconcile.

### Graceful shutdown

``` yaml
# /etc/rancher/k3s/kubelet.config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 45s
shutdownGracePeriodCriticalPods: 15s
```

``` yaml
# /etc/rancher/k3s/config.yaml
kubelet-arg: "config=/etc/rancher/k3s/kubelet.config"
```

### check system logs

```bash
sudo journalctl -u k3s-agent --since "1h ago" --reverse --no-pager | more
sudo journalctl -u k3s-agent --since "1 hour ago" --reverse | grep -i "Starting k3s-agent.service" 
sudo journalctl -u k3s --reverse | grep -i "Starting k3s.service"
```

*Example: [test-services.services.svc.cluster.local](test-services.services.svc.cluster.local).*

### Workarounds & Fixes

### klipper-lb

Klipper‑LB is the tiny, built‑in load balancer that k3s uses to give each agent a local, stable endpoint for talking to the Kubernetes API server. Instead of exposing a full external load balancer, k3s runs this lightweight component on 127.0.0.1:6444, and it simply forwards traffic from the agent to the control‑plane node (or rotates between multiple servers in an HA setup). It exists to make k3s simpler to deploy—no extra software, and no external LB. startup even though the cluster continues working normally.

#### troubleshooting

**log: warning - Error starting load balancer: listen tcp 127.0.0.1:6444: bind: address already in use.**

``` bash
rm -rf /var/lib/rancher/k3s/agent/etc/klipper-lb 
systemctl restart k3s-agent
```

### Containerd state

This procedure simulates a fresh node joining the cluster.
It deletes all containerd runtime state but does not remove the node from the cluster.

1. Stop k3s (server or agent)
1. Delete containerd state

    ```bash
    sudo rm -rf /var/lib/rancher/k3s/agent/containerd
    sudo rm -rf /var/lib/containerd
    ```

1. Start k3s

 **What this does:**

- Removes all images, snapshots, and container metadata
- Forces k3s to repull every image through CRI → mirrors → Harbor
- Simulates a fresh node rebuild
- Node identity, certificates, and cluster membership remain intact
- Workloads are rescheduled normally
- This is the correct method to validate offline rebuild capability and ensure Harbor mirrors are complete.

If all container images are provided locally (Example: through Harbor proxy caches), then the entire containerd image store becomes fully ephemeral.
This means:

- /var/lib/rancher/k3s/agent/containerd
- /var/lib/containerd

contain no unique or irreplaceable data. so it can be ignored on backups.

## host cli

### host cli - check port usage

``` bash
# example: port 32329
ss -ltnp | grep 32329

```

### kill all connections

``` bash
ss -K dst SERVER_IP:6443
```

## cert-manager

clear stale ACME challenges and orders created before DNS fixes.  
These objects are temporary and safe to delete — they do NOT remove or affect existing valid certificates.  
After cleanup, cert-manager automatically recreates fresh challenges using the corrected DNS configuration.

``` bash
kubectl delete challenge -A --all
kubectl delete order -A --all

```

### Removing cert‑manager Metadata from Secrets

When migrating clusters or taking manual control of TLS certificates, you may need to fully detach a Secret from cert‑manager. Cert‑manager uses labels and annotations to track ownership, ACME challenge state, and renewal configuration. If these remain, cert‑manager may attempt to “adopt” or overwrite the Secret.

This guide shows how to safely remove all cert‑manager metadata so the Secret becomes unmanaged.

**View Secrets Managed by cert‑manager:**

``` bash
kubectl get secrets -A --show-labels | grep cert-manager
```

This lists Secrets that contain cert‑manager labels or annotations.

**Remove cert‑manager Labels and Annotations:**

``` bash
SECRET_NAME=chimera-limbosolutions-com-tls
NAMESPACE=ignition-provisioner

# Remove cert-manager annotations
kubectl annotate secret ${SECRET_NAME} -n ${NAMESPACE} \
  cert-manager.io/alt-names- \
  cert-manager.io/common-name- \
  cert-manager.io/ip-sans- \
  cert-manager.io/issuer-group- \
  cert-manager.io/issuer-kind- \
  cert-manager.io/issuer-name- \
  cert-manager.io/uri-sans- \
  cert-manager.io/certificate-name- \
  acme.cert-manager.io/http-domain- \
  acme.cert-manager.io/dns-domain- \
  kubectl.kubernetes.io/last-applied-configuration-

# Remove cert-manager controller labels
kubectl label secret ${SECRET_NAME} -n ${NAMESPACE} \
  controller.cert-manager.io/fao- \
  controller.cert-manager.io/owner-kind- \
  controller.cert-manager.io/owner-name- \
  controller.cert-manager.io/owner-group-
```

After this cleanup, the Secret is fully detached from cert‑manager and will no longer be renewed, validated, or overwritten.

**Verify Cleanup:**

``` bash
kubectl get secrets -A --show-labels | grep cert-manager
```

If the Secret no longer appears, it is now unmanaged.

## hosting

### misc

**Error: failed to create fsnotify watcher: too many open files.**

When Alloy configured to collect logs (on_logs mode), the node can start to hit Linux inotify limits. Inotify is the kernel subsystem responsible for watching file changes, and Alloy relies heavily on it to monitor:

- container log files
- rotated log files
- symlinks under /var/log/containers
- pod log directories under /var/log/pods

In clusters with many pods, sidecars, and continuous log rotation, Alloy can easily require hundreds of thousands of inotify watches.

The default Linux limits are extremely low for modern Kubernetes workloads. Once the node reached these limits, several components started failing to create watchers:

- Alloy log pipeline
- K3s API server (which also uses file watchers internally)
- containerd and kubelet log watchers
- These failures produced high‑volume error logs such as:

``` bash
fsnotify watcher: too many open files
http: superfluous response.WriteHeader call
```

Increasing the kernel limits resolved the issue immediately.

``` bash
cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instances
```

**Increase Values example:**

``` bash
# /etc/sysctl.conf
fs.inotify.max_user_watches=1048576"
fs.inotify.max_user_instances=1024" 

```

``` bash
sudo sysctl -p
```

**If error continues try with this values:**

``` bash
# /etc/sysctl.conf
fs.inotify.max_user_watches=2097152
fs.inotify.max_user_instances=8192
fs.inotify.max_queued_events=65536
```

If the error persists after a reboot, and running the command `sysctl -p` resolves the issue, create the following file:

``` bash
# /etc/sysctl.d/99-zzz-override-inotify.conf 
fs.inotify.max_user_watches=2097152
fs.inotify.max_user_instances=8192
fs.inotify.max_queued_events=65536
```

Note: In my case, I already have a file named `/etc/sysctl.d/99-zzz-override_cilium.conf`, which despite not affecting the same parameters, might still be influencing something. What is certain is that after configuring this file, the error has stopped occurring.

If you use this method, the file `/etc/sysctl.conf` can be removed.