Skip to content

Building a Homelab with k3s

  • The goal is simple
    • we get two machine
    • we ensure they can talk to each other
    • we install k3s on one and k3s agent on the other,
    • we run our cluster resources there
    • we expose the portal through metalLB service
    • and then expose to the world using cloudflare tunnel or ssh remote port forwarding.

Here are my machines:

yours could differ [not a problem, commands will largly be the same except the firewall ones]

  • Fedora → Control Plane (server)
  • Manjaro → Worker (agent)

Initial Setup

0. Prerequisites (VERY IMPORTANT)

Decide IPs (static)

  • You must use static IPs for this machine.
  • The simplest way to do this is from the network settings on desktop. Else you can use nmcli to do the same.
  • Ensure that the ip is not in use by any other machine. I purposely selected a ip further up in my subnet range 192.168.1.0/24 to avoid any conflicts.
  • Check that the ip remains the same after restart.

Example:

  • Fedora (control plane): 192.168.1.199
  • Manjaro (worker): 192.168.1.200

Confirm:

ip a
ip route

From each machine, ensure they can ping each other:

ping 192.168.1.199
ping 192.168.1.200

Disable the firwalls firewalld / ufw

(for now) You can re-enable later once cluster works.

Fedora

sudo systemctl disable --now firewalld

Manjaro

sudo systemctl disable --now ufw

Disable swap (mandatory)

Why is this necessary? read here

Fedora + Manjaro

sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab

Verify:

free -h

Enable required kernel modules

Fedora + Manjaro

sudo modprobe br_netfilter
sudo modprobe overlay

Persist:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
overlay
EOF

Sysctl:

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables=1
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-ip6tables=1
EOF

sudo sysctl --system

1. Install K3s on Fedora (Control Plane)

On Fedora:

curl -sfL https://get.k3s.io | \
INSTALL_K3S_EXEC="server \
  --node-ip=192.168.1.199 \
  --tls-san=192.168.1.199 \
  --write-kubeconfig-mode=644" \
sh -

Verify service

sudo systemctl status k3s

Verify cluster

kubectl get nodes

Expected:

NAME        STATUS   ROLES                  AGE   VERSION
fedora      Ready    control-plane,master   XXs   v1.xx.x+k3s

2. Get the Join Token (VERY IMPORTANT)

On Fedora:

sudo cat /var/lib/rancher/k3s/server/node-token

Copy it somewhere safe. You’ll need it on Manjaro.

3. Install K3s on Manjaro (Worker Node)

On Manjaro:

curl -sfL https://get.k3s.io | \
K3S_URL=https://192.168.1.199:6443 \
K3S_TOKEN=<PASTE_TOKEN_HERE> \
INSTALL_K3S_EXEC="agent --node-ip=192.168.1.200" \
sh -

Verify agent service

sudo systemctl status k3s-agent

4. Verify the Cluster (From Fedora)

kubectl get nodes -o wide

Expected:

NAME       STATUS   ROLES                  INTERNAL-IP
fedora     Ready    control-plane,master   192.168.1.199
manjaro    Ready    <none>                 192.168.1.200

🎉 Your 2-node K3s cluster is LIVE

5. Test with a Real Workload

Deploy NGINX

kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort

Check:

kubectl get svc nginx

Example output:

NodePort: 80:31234/TCP

Access from browser:

http://192.168.1.199:31234
http://192.168.1.200:31234

K3s already installs kubectl, but to persist config:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $USER:$USER ~/.kube/config

7. Common Pitfalls (Read This Once)

Problem Cause
Worker not joining Wrong token or wrong server IP
Node shows NotReady Firewall / CNI blocked
Can’t access NodePort Firewall still enabled
Pods stuck in ContainerCreating br_netfilter not enabled

8. Monitoring the servers

1️⃣ Install Node Exporter (Both Machines)

Create a user

sudo useradd -rs /bin/false node_exporter

Download & install

curl -LO https://github.com/prometheus/node_exporter/releases/latest/download/node_exporter-*.linux-amd64.tar.gz
tar xvf node_exporter-*.tar.gz
sudo mv node_exporter-*/node_exporter /usr/local/bin/

Systemd service

sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Verify

curl http://localhost:9100/metrics

✔️ Repeat on both Fedora & Manjaro

  • Create a docker compose
version: "3.8"

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=15d"
    ports:
      - "9090:9090"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - ./grafana/data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - prometheus
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

Metallb setup