The GitOps Philosophy

GitOps is a paradigm shift in how we think about infrastructure and application deployment. The core principle is simple: Git is the single source of truth.

Every aspect of your infrastructure and applications is defined in Git:

  • Kubernetes manifests
  • Helm charts
  • Kustomize overlays
  • Application configurations
  • Infrastructure as Code

When you commit to Git, automated processes reconcile the actual state of your cluster with the desired state in Git.

Why FluxCD?

We evaluated several GitOps tools before settling on FluxCD:

FluxCD v2 advantages:

  • Native Kubernetes CRDs
  • Modular architecture (source controller, kustomize controller, helm controller)
  • Multi-tenancy support
  • Progressive delivery with Flagger
  • Strong community and CNCF backing

Our Implementation

Repository Structure

infrastructure/
├── clusters/
│   ├── production/
│   ├── staging/
│   └── development/
├── base/
│   ├── networking/
│   ├── monitoring/
│   └── security/
└── apps/
    ├── backend-services/
    ├── frontend-apps/
    └── data-pipelines/

Cluster Bootstrapping

Bootstrapping a new cluster is as simple as:

flux bootstrap github \
  --owner=myorg \
  --repository=infrastructure \
  --branch=main \
  --path=clusters/production

FluxCD installs itself and begins reconciling the desired state from Git.

The Benefits We Realized

1. Auditability

Every change to production is a Git commit. We can see:

  • Who made the change
  • When it was made
  • Why it was made (commit message)
  • What was changed (diff)

2. Disaster Recovery

Lost a cluster? Recreate it from Git:

# New cluster
flux bootstrap github ...
# Wait 5 minutes
# Cluster is fully reconciled

3. Environment Parity

Staging and production environments are managed from the same base configurations with environment-specific overlays. No more “it works in staging but not production.”

4. Automation

No more kubectl apply in CI/CD pipelines. FluxCD watches for changes and applies them automatically.

Challenges and Solutions

Challenge: Secret Management

Secrets can’t (shouldn’t) be stored in plain text in Git.

Solution: Mozilla SOPS + Age encryption

apiVersion: v1
kind: Secret
metadata:
    name: database-credentials
data:
    password: ENC[AES256_GCM,...]

Challenge: Image Updates

How do we deploy new application versions?

Solution: Image automation controllers FluxCD can watch container registries and automatically update manifests when new images are pushed.

Challenge: Progressive Rollouts

How do we safely deploy to production?

Solution: Flagger for canary deployments

apiVersion: flagger.app/v1beta1
kind: Canary
spec:
  analysis:
    threshold: 10
    maxWeight: 50
    stepWeight: 5

Key Learnings

  1. Start small: Begin with non-critical environments
  2. Structure matters: Invest time in your repository structure upfront
  3. Use Kustomize: Base + overlays pattern works beautifully
  4. Monitor reconciliation: Set up alerts for reconciliation failures
  5. Document patterns: GitOps requires discipline - document your patterns

The Impact

Before GitOps:

  • Manual kubectl commands
  • Configuration drift between environments
  • No audit trail
  • Difficult rollbacks
  • “Works on my machine” issues with kubectl versions

After GitOps with FluxCD:

  • Declarative, versioned infrastructure
  • Git-based audit trail
  • Automated reconciliation
  • Easy rollbacks (git revert)
  • Consistent tooling (flux CLI)

Final Thoughts

GitOps with FluxCD has fundamentally changed how we manage Kubernetes infrastructure. It’s not just about automation - it’s about bringing software engineering best practices (version control, code review, CI/CD) to infrastructure management.

If you’re managing multiple Kubernetes clusters or struggling with configuration drift, GitOps is worth serious consideration.

The learning curve exists, but the operational benefits are substantial.