Home-Plumbing my personal data platform
This last weekend I achieved the milestone I had my eyes on since May 2025! I finally deployed Airflow 3 (a workflow orchestration tool) in my own Kubernetes cluster running on K3s (lightweight Kubernetes) and Flux (GitOps automation). The Airflow deployment runs my custom Airflow image and my DAGs orchestrate the code from my own home-plumbing python package. This deployment marks the “go live” of my personal data platform which I jokingly named home plumbing - a nod toward the messy plumbing involved in the day-to-day work I face as a data engineer. ...