diff --git a/README.md b/README.md index 827d872..0555242 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ Kubernetes is an open‑source platform that automates the deployment, scaling, and management of containerized applications. It acts as an orchestrator, ensuring your containers run reliably across clusters of machines, handling networking, storage, and updates without downtime. + + - [kubernetes](#kubernetes) - [kubectl](#kubectl) - [Env variables](#env-variables) @@ -42,12 +44,12 @@ Kubernetes is an open‑source platform that automates the deployment, scaling, - [mounting](#mounting) - [annotations](#annotations) - [nodes](#nodes) + - [cordon](#cordon) - [taints](#taints) - [add taint](#add-taint) - [remove taint](#remove-taint) - [control plane - NoSchedule](#control-plane---noschedule) - [Official \*.kubernetes.io taints](#official-kubernetesio-taints) - - [cordon](#cordon) - [statefulset](#statefulset) - [statefulset - Set Replicas](#statefulset---set-replicas) - [Deployment](#deployment) @@ -70,9 +72,9 @@ Kubernetes is an open‑source platform that automates the deployment, scaling, - [prune old images](#prune-old-images) - [kill all](#kill-all) - [Prune ephemeral containerd data](#prune-ephemeral-containerd-data) + - [Graceful shutdown](#graceful-shutdown) - [check system logs](#check-system-logs) - [Workarounds \& Fixes](#workarounds--fixes) - - [Failed unmounting var-lib-rancher.mount on reboot](#failed-unmounting-var-lib-ranchermount-on-reboot) - [klipper-lb](#klipper-lb) - [troubleshooting](#troubleshooting) - [Containerd state](#containerd-state) @@ -773,6 +775,32 @@ kubectl get nodes -o wide ``` bash kubectl annotate node - ``` +### cordon + +``` bash +NODE="????" +kubectl cordon ${NODE} +``` + +Marks a node as unschedulable. + +- No new pods will be scheduled on that node +- Existing pods are not affected +- Even after a reboot, existing pods return to the same node +- Used for temporary maintenance (updates, debugging, draining prep) +- Kubernetes automatically adds the taint: + `node.kubernetes.io/unschedulable:NoSchedule` + +``` bash +NODE="???" +kubectl uncordon ${NODE} +``` + +Reverses the cordon. + +- The node becomes schedulable again +- New pods can land on it +- Existing pods remain untouched ## taints @@ -831,32 +859,6 @@ kubectl taint nodes ${NODE} node-role.kubernetes.io/control-plane=:NoSchedule Everything else in *.kubernetes.io is reserved and should not be used. -### cordon - -``` bash -NODE="????" -kubectl cordon ${NODE} -``` - -Marks a node as unschedulable. - -- No new pods will be scheduled on that node -- Existing pods are not affected -- Even after a reboot, existing pods return to the same node -- Used for temporary maintenance (updates, debugging, draining prep) -- Kubernetes automatically adds the taint: - `node.kubernetes.io/unschedulable:NoSchedule` - -``` bash -NODE="???" -kubectl uncordon ${NODE} -``` - -Reverses the cordon. - -- The node becomes schedulable again -- New pods can land on it -- Existing pods remain untouched ## statefulset @@ -985,7 +987,6 @@ data: # ``` - ## Custom Resource Definitions - **Definition:** A Custom Resource Definition (CRD) is an extension of the Kubernetes API. @@ -1075,6 +1076,21 @@ systemctl start k3s After this, k3s will start with a completely clean containerd state, and all images will be pulled again as workloads reconcile. +### Graceful shutdown + +``` yaml +# /etc/rancher/k3s/kubelet.config +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +shutdownGracePeriod: 45s +shutdownGracePeriodCriticalPods: 15s +``` + +``` yaml +# /etc/rancher/k3s/config.yaml +kubelet-arg: "config=/etc/rancher/k3s/kubelet.config" +``` + ### check system logs ```bash @@ -1087,65 +1103,6 @@ sudo journalctl -u k3s --reverse | grep -i "Starting k3s.service" ### Workarounds & Fixes -#### Failed unmounting var-lib-rancher.mount on reboot - -When running K3s with /var/lib/rancher on a separate disk. - -K3s and containerd often leave behind mount namespaces and overlay layers that block clean unmounting during shutdown. -This causes slow reboots and errors like: - -``` bash -Failed unmounting var-lib-rancher.mount -``` - -1. Create the cleanup service - - ``` bash - nano /etc/systemd/system/rancher-cleanup.service - ``` - - Paste: - - ``` bash - - [Unit] - DefaultDependencies=no - Before=shutdown.target - - [Service] - Type=oneshot - ExecStart=/bin/sh -c '/bin/umount -l /var/lib/rancher || true' - - [Install] - WantedBy=shutdown.target - - ``` - - Why this works - - - DefaultDependencies=no ensures the service runs early. - - Before=umount.target guarantees it executes before systemd tries to unmount anything. - - umount -l detaches the filesystem immediately, even if containerd still holds namespaces. - - || true prevents harmless “not mounted” errors from blocking shutdown. - -1. Reload systemd - - ``` bash - systemctl daemon-reload - ``` - -1. Enable the cleanup service - - ```bash - systemctl enable rancher-cleanup.service - ``` - -1. Reboot to test: - - ``` bash - reboot - ``` - ### klipper-lb Klipper‑LB is the tiny, built‑in load balancer that k3s uses to give each agent a local, stable endpoint for talking to the Kubernetes API server. Instead of exposing a full external load balancer, k3s runs this lightweight component on 127.0.0.1:6444, and it simply forwards traffic from the agent to the control‑plane node (or rotates between multiple servers in an HA setup). It exists to make k3s simpler to deploy—no extra software, and no external LB. startup even though the cluster continues working normally. @@ -1326,3 +1283,16 @@ fs.inotify.max_user_watches=2097152 fs.inotify.max_user_instances=8192 fs.inotify.max_queued_events=65536 ``` + +If the error persists after a reboot, and running the command `sysctl -p` resolves the issue, create the following file: + +``` bash +# /etc/sysctl.d/99-zzz-override-inotify.conf +fs.inotify.max_user_watches=2097152 +fs.inotify.max_user_instances=8192 +fs.inotify.max_queued_events=65536 +``` + +Note: In my case, I already have a file named `/etc/sysctl.d/99-zzz-override_cilium.conf`, which despite not affecting the same parameters, might still be influencing something. What is certain is that after configuring this file, the error has stopped occurring. + +If you use this method, the file `/etc/sysctl.conf` can be removed.