Homelab-Ops is a production-grade, self-healing Sovereign Cloud Platform engineered on physical bare-metal hardware in Mumbai, India, integrated with supporting cloud infrastructure in Oracle Cloud Infrastructure (OCI) and Google Cloud Platform (GCP).
This platform simulates enterprise-scale infrastructure engineering while operating under strict real-world constraints:
- Carrier-Grade NAT (CGNAT) Traversal: Securely accepting external webhooks and public user traffic without exposing home router ports or relying on static IPv4 leasing.
- Deterministic GitOps Continuous Delivery: Managing platform operators and applications declaratively with Flux CD v2 and Mozilla SOPS, eliminating configuration drift and keeping secrets versioned safely in Git.
- Dual-Tier Hardware Economics: Overcoming disk I/O bottlenecks on a single Mini PC by partitioning high-IOPS NVMe flash for database engines and durable SATA mechanical disks for multi-terabyte media and document archives.
- Hybrid Multi-Cloud Resilience: Supplementing the on-premise sovereign cluster with cloud infrastructure across Oracle Cloud Infrastructure (OCI) and Google Cloud Platform (GCP) for out-of-band availability monitoring, remote state locking, and secondary cloud compute.
To provide complete architectural clarity without visual clutter, the platform is organized into three distinct structural diagrams:
How external traffic, edge security, multi-cloud support, and the secure administrative mesh are organized:
How physical hardware, hypervisor resource fencing, and tiered storage are partitioned:
How code changes flow automatically from Git into production without configuration drift:
Every robust platform begins with physical constraints. The on-premise foundation is an ultra-efficient Intel Core i5 Mini PC with 16GB DDR4 RAM, a 256GB NVMe SSD, and a 1TB SATA mechanical drive.
To convert this single machine into an enterprise platform, Proxmox VE 8 was chosen as the Type-1 hypervisor. By budgeting host resources strictly—reserving 3.5GB of RAM for the Debian kernel, memory caching, and backup snapshot compression—a dedicated virtual machine (k3s-prod) was allocated 12GB RAM and 4 vCPUs, ensuring high memory headroom for Kubernetes workloads while safeguarding the physical host against out-of-memory (OOM) kernel panics.
Residential internet connections rarely provide static public IP addresses and are almost universally bound behind Carrier-Grade NAT (CGNAT), making traditional port-forwarding impossible or insecure.
- The Evolution: Early iterations explored cloud gateway relays. While functional, routing cross-continental traffic introduced latency hops and incurred ongoing NAT maintenance.
- The Production Standard: The platform upgraded to Cloudflare Zero Trust Anycast Tunnels (
cloudflared). Operating as an in-cluster deployment,cloudflaredinitiates outbound-only QUIC connections to Cloudflare's nearest edge data centers (Mumbai, Delhi, Chennai). Public webhooks and traffic terminate at the edge in <15ms, protected by Cloudflare WAF and DDoS mitigation, with zero open inbound ports on the home firewall.
Operating infrastructure through manual kubectl apply commands or ad-hoc scripts leads to configuration drift and operational opacity.
The modern platform runs on pure GitOps Continuous Delivery powered by Flux CD v2:
- The cluster continuously synchronizes its desired state from this Git repository.
- Workload reconciliation is deterministically ordered: platform foundations (storage classes, operators, ingress daemons) must report healthy before application workloads are scheduled (
appsdepends onplatform). - Sensitive tokens and database passwords are encrypted declaratively using Mozilla SOPS + Age. Because secrets remain encrypted in Git, they can be safely reviewed in pull requests, while the in-cluster Flux decryptor hydrates them into memory without leaving plaintext traces on physical disks.
- Zero-Trust Ephemeral CI/CD Automation: Continuous integration workflows (such as automated documentation deployments) leverage Zero-Trust Network Access (ZTNA) via Tailscale. Instead of exposing Kubernetes API server ports to the public internet or maintaining static VPN credentials, cloud GitHub Actions runners authenticate dynamically using scoped OAuth clients (
tag:ci), join an ephemeral WireGuard mesh, execute rolling updates, and self-terminate immediately upon job completion—ensuring zero permanent attack surfaces.
Single-node virtualization often stumbles when high-IOPS database queries compete with heavy background disk writes:
- Storage Tiering: Fast NVMe storage is dedicated strictly to the OS, K3s state, and PostgreSQL data files. Multi-terabyte document indexing (Paperless-ngx) and audio/book streaming libraries are routed directly to the high-capacity 1TB SATA mechanical disk.
- Database Modernization: Rather than deploying fragile, static database pods, relational data is managed by the CloudNativePG Operator. The operator provides automated PostgreSQL lifecycle management, health monitoring, self-healing failover, and Write-Ahead Log (WAL) archiving.
A monitoring system running inside the cluster it is supposed to monitor cannot alert you when the cluster itself dies. Furthermore, local backups are vulnerable if the physical host suffers hardware failure.
To achieve enterprise-grade resilience, the architecture integrates Oracle Cloud Infrastructure (OCI) and Google Cloud Platform (GCP) support instances:
- Out-of-Band Health Probes: An independent cloud VM in OCI Mumbai runs Uptime Kuma, continuously polling public endpoints (
https://docs.vijaysingh.cloud,https://hooks.vijaysingh.cloud) over the public internet and dispatching alerts to Slack/Discord if homelab broadband or power fails. - Remote State & Offsite Backup: Terraform state is stored securely with remote locking in an OCI Object Storage S3-compatible backend, and nightly encrypted Restic backups are pushed offsite, fulfilling the 3-2-1 backup rule (3 copies, 2 media types, 1 offsite).
- Multi-Cloud Compute Support: Supporting cloud virtual machines across OCI and GCP provide compute targets for auxiliary services and remote operations.
| Architectural Pillar | Core Technologies | How It Solves the Problem |
|---|---|---|
| Zero-Trust Edge & Ingress | Cloudflare Zero Trust, cloudflared, Traefik |
Traverses CGNAT with outbound-only QUIC tunnels; provides edge WAF and reduces latency to <15ms with zero open router ports. |
| GitOps Continuous Delivery | Flux CD v2, Kustomize, GitHub | Pull-based reconciliation loop eliminates configuration drift; deterministic ordering ensures zero failed boots. |
| In-Git Secret Decryption | Mozilla SOPS, Age Cryptography | Secrets versioned declaratively in Git with asymmetric encryption; decrypted exclusively in-memory by Flux. |
| Dual-Tier Storage Strategy | NVMe Flash (256GB), SATA Mechanical (1TB) | Isolates high-IOPS database operations from bulk document indexing and media streaming workloads. |
| Stateful Resilience & HA | CloudNativePG (PostgreSQL Operator) | Self-healing relational database clustering with automated failover and continuous WAL archiving. |
| Zero-Trust CI/CD Automation | Tailscale Ephemeral Mesh, GitHub Actions | Connects cloud runners to private Kubernetes clusters via scoped OAuth and WireGuard; eliminates public API exposure and credential expiration. |
| Multi-Cloud Support Plane | OCI Mumbai, GCP Support VM, Uptime Kuma | Independent external heartbeat monitoring and remote state locking; operates even during power or ISP outages. |
All services are containerized, declared in Git, and routed through Cloudflare Zero Trust:
Platform & Security
├── cloudflared # High-availability Anycast edge ingress tunnel
├── postgres-operator # CloudNativePG enterprise PostgreSQL lifecycle operator
├── homepage # Homelab Command Center (Live CPU/RAM & status portal)
├── monitoring # Prometheus metrics scraping & Grafana telemetry
├── kwatch # Instant Discord/Slack notifier for crashed pods
└── docs # MkDocs Material engineering handbook (docs.vijaysingh.cloud)
Operations & Workflow Automation
├── n8n # Hardened event-driven workflow automation engine
└── pdf-generator # Automated document compilation microservice
Sovereign Documents & Media (1TB HDD Tier)
├── paperless-ngx # OCR document ingestion, tagging, and search archive
└── bookorbit # Multi-user digital book library & sync manager
Productivity & Personal Health
├── miniflux # Ultra-fast, lightweight Go RSS reader
├── linkding # Bookmarking service with tag search
└── ryot / wger # Fitness analytics and workout tracking platform
pie title Mini PC 16GB RAM Allocation
"k3s-prod Kubernetes Cluster" : 12
"Proxmox VE Base OS & ZFS/Caches" : 3.5
"Emergency Host Safety Buffer" : 0.5
- Physical Node: Intel Core i5 Mini PC (4 Cores / 8 Threads)
- Total Host RAM: 16,384 MB (16 GB DDR4)
- Production Cluster Allocation (
k3s-prodVM 500): - Memory: 12,288 MB (12 GB RAM) — gives >60% memory headroom for peak OCR ingestion and database queries.
- Compute: 4 vCPUs dedicated to Kubernetes scheduler.
- Storage: 50GB NVMe root disk + 1TB SATA HDD attached virtual disk.
- Proxmox VE Host OS Allocation:
- Memory: 3,500 MB (3.5 GB RAM) reserved for Debian kernel, KVM hypervisor daemons, and
vzdumpbackup compression. - Storage: ~20GB NVMe root partition for hypervisor binaries and ISOs.
In alignment with production platform engineering standards and peer review, the following improvements are prioritized on the active roadmap:
Following CNCF and enterprise platform standards, all architecture specifications, flagship project case studies, and incident post-mortems are documented in the docs/ hierarchy:
- 01. System Architecture Overview — High-level multi-cloud hybrid topology, design goals, and component boundaries.
- 02. Hardware & Virtualization — Bare-metal Mini PC specifications and Proxmox VE 8 memory fencing.
- 03. Kubernetes & K3s Cluster — Single-node production K3s cluster architecture and namespaces.
- 04. Zero-Trust Networking — Cloudflare Zero Trust Anycast Tunnels and Tailscale administrative mesh.
- 05. Dual-Tier Storage Strategy — Partitioning high-IOPS NVMe flash from bulk mechanical SATA storage.
- Case Study: Zero-Trust Hybrid Ingress Engine — CGNAT traversal, Anycast edge routing, WAF, and <15ms latency.
- Case Study: Declarative GitOps & In-Git Secrets — Continuous delivery via Flux CD v2 and Mozilla SOPS + Age encryption.
- Case Study: Stateful PostgreSQL Operator on Bare-Metal — CloudNativePG high-availability operator, automated failover, and WAL archiving.
- Case Study: Multi-Cloud Resilience & Out-of-Band DR — OCI Mumbai & GCP support compute, Uptime Kuma external probes, and 3-2-1 backup replication.
- Case Study: Autonomous Sysadmin & Patch Automation — Host OS patching engine, K3s cordon/drain coordination, and Renovate GitOps guardrails.
- Comprehensive ADR Register — 16 formal Architecture Decision Records documenting every pivotal architectural decision (ADR-001 through ADR-016), following the Michael Nygard standard.
- Backup & Disaster Recovery Runbook — Procedures for 3-2-1 backup verification and bare-metal disaster recovery.
- Day-2 Cluster Operations Runbook — SOPS secret rotation, manual GitOps reconciliation, and host maintenance.
- Incident Response & Triage Playbook — Emergency diagnostic workflows, P0–P3 severity triage, and automated failover playbooks.
- Execution Master Index — Comprehensive 7-phase execution records detailing the migration from legacy multi-bastion lab to single-node sovereign cloud.
- 01. Architecture Baseline & 02. Codebase Pruning
- 03. Remote State Migration & 04. Hypervisor Consolidation
- 05. Edge Ingress & Registry & 06. GitOps & SOPS Secrets
- 07. Fleet Deployment & ChatOps
- Engineering Archive Overview — Historical milestones, early technical challenges, and iterative migrations leading to v3.0.
- Bare-Metal Boot Failure Post-Mortem & WAN Connectivity Post-Mortem
- Ansible Automation Journey & Terraform Modularization
- Hybrid Cloud Automation Journal (n8n) & Self-Healing Watchdog
# Encrypt an updated Kubernetes Secret manifest before committing to Git
sops --encrypt --in-place kubernetes/apps/n8n/secret.enc.yaml
# Directly edit an encrypted secret file using your default editor
sops kubernetes/apps/n8n/secret.enc.yaml# Force immediate reconciliation of the platform layer
flux reconcile kustomization platform --with-source
# Force immediate reconciliation of the application layer
flux reconcile kustomization apps --with-source# Direct SSH access to the Proxmox VE hypervisor (Tailscale mesh)
ssh root@<tailscale-mesh-host>
# Direct SSH access to the Production K3s node
ssh devops@192.168.1.x
# Inspect active Cloudflare tunnel connections
kubectl logs -n platform -l app.kubernetes.io/name=cloudflared --tail=50Vijay Singh — DevOps & Platform Engineer
- GitHub: @vsingh55
- Documentation Portal: https://docs.vijaysingh.cloud
- Homelab Command Center: https://hub.vijaysingh.cloud
- System Status & Uptime: https://status.vijaysingh.cloud
Engineered with precision, operational discipline, and pride in sovereign platform engineering.