A Recovery-First Kubernetes Homelab

This site is served by the project it describes: a single-machine Kubernetes homelab built the way I’d build production. The goal was never to run the most services — it was to run a small number of services on a foundation that is declarative, recoverable, and secure by default. Everything below is the design; nothing here is a secret, a credential, an address, or anything an attacker could use.

Principles

• Declarative over imperative. Every change is a file describing desired state, applied idempotently. Running it twice equals running it once.
• Git is the source of truth; the runtime is a cache. If a change isn’t in Git, it didn’t happen — and reconciliation will undo it.
• Recovery-first. Anything required to rebuild the cluster lives outside the cluster and outside the machine. I designed the restore path before deploying workloads.
• Immutable artifacts. Container images are pinned by digest; chart and package versions are pinned. No floating latest.
• Least privilege & secure by default. Workloads run non-root under a restricted Pod Security baseline; the network defaults to deny; only one service is exposed to the public internet (this blog).

The stack

How it fits together

Substrate. A single ARM64 Linux machine runs k3s with flannel and kube-proxy disabled, secrets encryption on, and an audit policy that never logs secret bodies. Keeping the substrate portable (it can be re-created on another ARM64 hypervisor) is part of the recovery story.

Networking. Cilium provides the dataplane and the Gateway API implementation, with Hubble for flow visibility. Public traffic arrives exclusively through a Cloudflare Tunnel — there are no inbound ports open on the host. Everything that isn’t meant for the public reaches me over Tailscale instead.

GitOps. Argo CD continuously reconciles the cluster against Git. The primary source is an in-cluster Git forge (Gitea), which push-mirrors every repository to an off-host mirror - the recovery root. That split matters: the in-cluster forge is a convenience, never the root of trust, and a recovery variant of the root application re-points everything at the off-host mirror for a rebuild. The repo is structured as a base plus environment overlays; the live cluster tracks exactly one overlay, while the others exist only as render-and-validate targets in CI. Self-healing is on, so manual drift is reverted automatically.

Secrets. Bootstrap secrets are encrypted at rest with SOPS + age; the private key is kept offline. The design deliberately avoids making any in-cluster service the root of trust for recovery.

Identity. Kanidm is the OIDC provider; internal tools (Argo CD, Grafana, and others) authenticate against it, and it’s reachable only over the tailnet.

Observability. A Prometheus stack with Alertmanager, Grafana dashboards, and Hubble flows. Alerts fan out to push notifications, and an external dead-man’s-switch watches the watcher — if the cluster goes dark, something outside it notices.

Storage & backup — four independent layers:

1. Git holds the declared state of everything, mirrored off-host.
2. Persistent volumes use a retain reclaim policy so data survives a careless delete.
3. Velero backs up Kubernetes resources to off-host, S3-compatible object storage.
4. Application data is backed up by a host-driven job that archives, checksums, and age-encrypts it before uploading off-host — and the restore path is cluster-independent, so it works even if the cluster is gone.

Supply chain. Images are built by CI on rootless BuildKit (no Docker socket, no privileged containers), pushed to a local registry, scanned, given a signed SBOM attestation, and signed by digest before anything is promoted. This page is the proof: the blog lives in its own Git repo, CI builds and signs the image, then opens a digest-bump pull request against the platform repo. Merging that PR is the deploy gate - Argo CD rolls it out, and admission control verifies the signature before the pod is allowed to run.

Security posture

• Restricted Pod Security by default; root-requiring images are quarantined to explicit, documented exception namespaces.
• Policy as code (Kyverno): image-signature verification, no privileged containers, no latest tags, and digest pinning — enforced at admission, with a policy regression test suite in CI.
• Default-deny networking, live and enforcing in every application namespace: each one starts from deny-all and gets an explicit L3-L7 allow-list. Even the tunnel connector’s egress is restricted.
• Tailnet-only by default. Exactly one hostname resolves on the public internet — this blog. Everything else is private.

Recovery model

The test I hold the design to: if the machine is destroyed, can I rebuild from nothing but Git and off-host backups? The answer is yes — the cluster state, the encrypted secrets material, the resource backups, and the application data all live off-host, and the rebuild is a documented, rehearsed procedure rather than tribal knowledge.

Status

The platform is built in phases, each with its own exit criteria, rather than all at once. Public blog, private developer services, and a media platform run on top of it today; a local-AI phase is scaffolded but deliberately not deployed until its own recovery gates pass - incomplete things stay off by default. The part I’m proudest of is the foundation underneath: boring, declarative, and recoverable.