261 lines
5.6 KiB
Markdown
261 lines
5.6 KiB
Markdown
# Casa k3s cluster
|
|
|
|
**k3s version:** v1.35.2+k3s1
|
|
|
|
**Table of Contents:**
|
|
|
|
- [Common](#common)
|
|
- [OCI Mirrors Cache](#oci-mirrors-cache)
|
|
- [Master Node](#master-node)
|
|
- [Master Node - proxmox vm](#master-node---proxmox-vm)
|
|
- [Master Node - network configuration](#master-node---network-configuration)
|
|
- [Master Node - k3s setup](#master-node---k3s-setup)
|
|
- [minion01 - worker node](#minion01---worker-node)
|
|
- [Minion01 - proxmox vm](#minion01---proxmox-vm)
|
|
- [casa-minion-01 - k3s - setup](#casa-minion-01---k3s---setup)
|
|
|
|
## Common
|
|
|
|
**Disable swap:**
|
|
|
|
``` bash
|
|
swapoff -a
|
|
Edit /etc/fstab and comment out any swap entries:
|
|
# /swapfile none swap sw 0 0
|
|
```
|
|
|
|
**Other Packages:**
|
|
|
|
``` bash
|
|
sudo apt update -y && \
|
|
sudo apt install -y \
|
|
curl \
|
|
btop
|
|
|
|
```
|
|
|
|
**Update system:**
|
|
|
|
``` bash
|
|
sudo apt update -y && \
|
|
sudo apt upgrade -y && \
|
|
sudo apt auto-remove -y
|
|
```
|
|
|
|
**name servers:**
|
|
|
|
``` bash
|
|
cat <<EOF > /etc/resolv.conf
|
|
domain dev.lan
|
|
search dev.lan. lan.
|
|
nameserver 192.168.14.254
|
|
EOF
|
|
```
|
|
|
|
### OCI Mirrors Cache
|
|
|
|
``` bash
|
|
# /etc/rancher/k3s/registries.yaml
|
|
|
|
mirrors:
|
|
# --- Docker Hub (all aliases → docker-mirror) ---
|
|
docker.io:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/docker-mirror
|
|
registry-1.docker.io:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/docker-mirror
|
|
index.docker.io:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/docker-mirror
|
|
|
|
# --- GHCR ---
|
|
ghcr.io:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/ghcr-mirror
|
|
|
|
# --- Quay.io ---
|
|
quay.io:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/quay-mirror
|
|
|
|
# --- Kubernetes official registry ---
|
|
registry.k8s.io:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/k8s-mirror
|
|
|
|
# --- Google Container Registry (your "grc-mirror") ---
|
|
gcr.io:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/grc-mirror
|
|
|
|
# --- AWS Public ECR ---
|
|
public.ecr.aws:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/aws-mirror
|
|
|
|
# --- Microsoft Container Registry ---
|
|
mcr.microsoft.com:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/mcr-proxy
|
|
|
|
# --- GitLab Container Registry ---
|
|
registry.gitlab.com:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/gitlab-mirror
|
|
|
|
# --- Bitnami (namespace under Docker Hub) ---
|
|
docker.io/bitnami:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/bitnami-mirror
|
|
|
|
# --- Red Hat registry ---
|
|
registry.redhat.io:
|
|
endpoint:
|
|
- https://oci.limbosolutions.com/v2/readheat-mirror
|
|
|
|
|
|
configs:
|
|
oci.limbosolutions.com:
|
|
auth:
|
|
username: <username>
|
|
password: <password>
|
|
tls:
|
|
insecure_skip_verify: false
|
|
```
|
|
|
|
## Master Node
|
|
|
|
| Name | Value |
|
|
| --- | --- |
|
|
| **NAME** | casa.dev.lan |
|
|
| **VLAN** | casa |
|
|
| **IP** | 192.168.14.9 |
|
|
| **OS** | Debian GNU/Linux 12 (bookworm) |
|
|
|
|
### Master Node - proxmox vm
|
|
|
|
*hosted on surfacepro.*
|
|
|
|
``` yaml
|
|
agent: 1
|
|
balloon: 0
|
|
boot: order=scsi0;ide2;net0
|
|
cores: 2
|
|
cpu: host
|
|
ide2: none,media=cdrom
|
|
memory: 2355
|
|
meta: creation-qemu=10.1.2,ctime=1762626497
|
|
name: casa
|
|
net0: virtio=BXX:XX:XX:XX:XX:XX,bridge=vmbr0,tag=xx
|
|
numa: 0
|
|
onboot: 1
|
|
ostype: l26
|
|
scsi0: local-lvm:vm-XXX-disk-0,iothread=1,size=24G,ssd=1
|
|
scsihw: virtio-scsi-single
|
|
smbios1: uuid=cxxxx-xxxx-xxxx-xxxx-xxxx
|
|
sockets: 1
|
|
usb0: host=1-1.1
|
|
```
|
|
|
|
### Master Node - network configuration
|
|
|
|
``` bash
|
|
ip a # check ethernet name
|
|
|
|
# removes automatic configuration as dhcp client
|
|
sed -i '/ens18/d' /etc/network/interfaces
|
|
|
|
cat <<EOF > /etc/network/interfaces.d/ens18
|
|
# my network configuration
|
|
auto ens18
|
|
iface ens18 inet static
|
|
address 192.168.14.9/24
|
|
gateway 192.168.0.1
|
|
EOF
|
|
|
|
```
|
|
|
|
### Master Node - k3s setup
|
|
|
|
``` bash
|
|
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.2+k3s1 sh -
|
|
```
|
|
|
|
**Taint NoSchedule on master node:**
|
|
|
|
kubectl taint nodes <master-node-name> node-role.kubernetes.io/control-plane=:NoSchedule
|
|
|
|
``` bash
|
|
kubectl taint nodes casa node-role.kubernetes.io/control-plane=:NoSchedule
|
|
```
|
|
|
|
## minion01 - worker node
|
|
|
|
### Minion01 - proxmox vm
|
|
|
|
*hosted on gaia.*
|
|
|
|
```yaml
|
|
agent: 1
|
|
balloon: 0
|
|
boot: order=scsi0;ide2;net0
|
|
cores: 4
|
|
cpu: host
|
|
ide2: none,media=cdrom
|
|
memory: 4096
|
|
meta: creation-qemu=10.1.2,ctime=1763219351
|
|
name: casa-minion-01
|
|
net0: virtio=BXX:XX:XX:XX:XX:XX,bridge=vmbr0,tag=xx
|
|
numa: 0
|
|
onboot: 1
|
|
ostype: l26
|
|
scsi0: fastcore:vm-XXX-disk-0,iothread=1,size=8G,ssd=1
|
|
scsi1: fastcore:vm-XXX-disk-1,iothread=1,size=16G,ssd=1
|
|
scsihw: virtio-scsi-single
|
|
smbios1: xxxx-xxxx-xxxx-xxxx-xxxx
|
|
sockets: 1
|
|
```
|
|
|
|
| Name | Value |
|
|
| --- | --- |
|
|
| **NAME** | minion01 |
|
|
| **VLAN** | casa |
|
|
| **IP** | 192.168.14.10 |
|
|
| **OS** | Debian GNU/Linux 12 (bookworm) |
|
|
|
|
### casa-minion-01 - k3s - setup
|
|
|
|
``` bash
|
|
# install k3s as agent / worker node
|
|
# execute on server to get token
|
|
# cat /var/lib/rancher/k3s/server/node-token
|
|
|
|
K3S_TOKEN="???"
|
|
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.35.2+k3s1 INSTALL_K3S_EXEC="agent --data-dir /dataDisk/k3s --server https://casa.dev.lan:6443 --token ${K3S_TOKEN}" sh -s -
|
|
```
|
|
|
|
Change kubectl -n kube-system edit configmap local-path-config on kube-system to set path to provisioner.
|
|
|
|
``` yaml
|
|
config.json: |-
|
|
{
|
|
"nodePathMap":[
|
|
{
|
|
"node":"DEFAULT_PATH_FOR_NON_LISTED_NODES",
|
|
"paths":["/var/lib/rancher/k3s/storage"]
|
|
},
|
|
{
|
|
"node":"casa-minion-01",
|
|
"paths":["/dataDisk/k3s/storage"]
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
**Set node labels:**
|
|
|
|
``` bash
|
|
kubectl label node casa-minion-01 role=worker-node
|
|
```
|