ADR 0016 — GitOps Deployment Model

Status: Accepted (Phase 10) Context date: 2026-07

Context

The EthioConnect platform has Helm charts for all 11 services (ADR 0013), Kustomize base manifests for cluster policy (ADR 0012), and Terraform modules for cloud infrastructure (ADR 0015). The platform now requires a continuous delivery model that connects source code changes to running workloads in staging and production Kubernetes clusters. The delivery model must satisfy three requirements:

• Declarative single source of truth. The desired state of every cluster must be fully described in Git. Manual kubectl apply or helm upgrade commands are prohibited in production.
• Environment promotion gates. Changes must flow automatically to staging for validation, but require explicit approval before reaching production.
• Secret management without Git-committed plaintext. The Helm charts reference Kubernetes Secrets by name (ADR 0013, section 5), but the Secret objects themselves must be synced from AWS Secrets Manager without storing sensitive values in the repository.

Decision

1. ArgoCD as the GitOps controller

ArgoCD is the GitOps controller for all Kubernetes environments. It watches the Git repository for changes to Helm charts, values files, and Kustomize manifests, and reconciles the cluster state to match.

2. Why ArgoCD over Flux

Both ArgoCD and Flux are CNCF-graduated GitOps controllers. ArgoCD was chosen for three reasons:

• Sync waves. ArgoCD supports argocd.argoproj.io/sync-wave annotations that control the order in which resources are applied within a single Application. The platform requires ordered deployment: cluster policies (namespaces, NetworkPolicies) must exist before External Secrets Operator can create Secrets, and Secrets must exist before application pods can start. Flux achieves ordering through Kustomization dependencies, which requires splitting the deployment into multiple Kustomization objects and managing their dependency graph explicitly. ArgoCD sync waves express the same ordering within a single Application, keeping the deployment topology in one place.
• Web UI. ArgoCD provides a built-in web dashboard that visualizes application sync status, resource health, and diff views. This is valuable for incident response (quickly identifying which services are out of sync or degraded) and for onboarding operators who are not Helm or Kubernetes experts. Flux has no built-in UI; third-party dashboards (Weave GitOps) exist but add another component to maintain.
• Native Helm support. ArgoCD renders Helm charts server-side using helm template with the specified values files, then applies the rendered manifests. It understands Helm chart dependencies, hooks, and CRD lifecycle. Flux's HelmRelease CRD also supports Helm, but ArgoCD's approach of treating the rendered manifests as the source of truth (rather than wrapping Helm in a CRD) produces more predictable diff and rollback behavior.

3. External Secrets Operator over Sealed Secrets or Vault Agent

External Secrets Operator (ESO) syncs secrets from AWS Secrets Manager into Kubernetes Secret objects. ESO was chosen over two alternatives:

Sealed Secrets. Sealed Secrets encrypts secrets with a cluster-specific key and commits the ciphertext to Git. This is GitOps-pure (secrets are in the repository), but has three drawbacks: (a) secret rotation requires re-encrypting and committing a new SealedSecret manifest, (b) the cluster's sealing key is a single point of failure (losing it makes all committed secrets unrecoverable), and (c) the approach does not integrate with AWS Secrets Manager, which the Terraform modules already use for RDS master passwords, MSK SASL credentials, and TURN shared secrets. Rejected.

Vault Agent sidecar. HashiCorp Vault with agent sidecars provides dynamic secret injection. This is powerful but introduces significant operational complexity: Vault itself must be deployed, unsealed, backed up, and upgraded. For a platform that already stores secrets in AWS Secrets Manager (provisioned by Terraform), adding Vault creates a redundant secret store. Rejected.

ESO is the right fit because:

• The IAM module (ADR 0015) already creates an IRSA role for the external-secrets service account in comms-control-plane with secretsmanager:GetSecretValue and secretsmanager:ListSecrets permissions scoped to comms/* secrets.
• ESO creates standard Kubernetes Secret objects, which the Helm charts already reference via valueFrom.secretKeyRef. No application or chart changes are needed.
• Secret rotation in AWS Secrets Manager is automatically reflected in the cluster. ESO polls at a configurable interval (default 1 hour, tunable to 5 minutes for sensitive credentials) and updates the Kubernetes Secret in place.

4. Sync wave ordering

ArgoCD sync waves control the order in which platform components are deployed. The ordering ensures that dependencies exist before dependents attempt to start:

Wave	Category	Resources	Rationale
0	Cluster policy	Namespaces, NetworkPolicies (deploy/k8s/base/)	Namespaces must exist before any namespaced resource can be created.
1	External Secrets	ESO controller Deployment, ClusterSecretStore, ExternalSecret manifests	ESO must be running and configured before it can create Secret objects.
2	Platform services	All 11 application Helm charts (umbrella chart)	Services depend on Secrets created in wave 1.
3	Observability	Prometheus, Grafana, OTel Collector, ServiceMonitors	Observability depends on service endpoints existing for scrape targets.

Each wave completes (all resources healthy) before the next wave begins. If a wave fails, ArgoCD halts the sync and reports the failure. This prevents cascading failures where application pods crash-loop because Secrets do not yet exist.

5. Progressive delivery: auto-sync staging, manual gate for production

The deployment model uses different sync policies per environment:

Staging:

• ArgoCD auto-sync is enabled. Any commit to the main branch that changes Helm charts, values files, or Kustomize manifests triggers an automatic sync to the staging cluster.
• Auto-prune is enabled: resources removed from Git are deleted from the cluster.
• Self-heal is enabled: manual changes made via kubectl are reverted to match Git within the polling interval (3 minutes).
• This provides continuous deployment to staging, enabling rapid validation of changes.

Production:

• ArgoCD auto-sync is disabled. Changes appear as "OutOfSync" in the ArgoCD dashboard but are not applied automatically.
• A human operator reviews the diff in the ArgoCD UI (or CLI) and clicks "Sync" to apply. This is the manual gate.
• Auto-prune is disabled in production to prevent accidental resource deletion from a mistaken Git commit.
• Self-heal is enabled to prevent configuration drift from manual kubectl interventions, but only after a sync has been explicitly approved.

This model ensures that staging receives every change immediately (catching integration issues early), while production changes require deliberate human approval.

6. IRSA for pod-level IAM isolation

IAM Roles for Service Accounts (IRSA) provides pod-level AWS IAM isolation. The IAM module (deploy/terraform/modules/iam/) creates one IAM role per service, each with a trust policy that constrains sts:AssumeRoleWithWebIdentity to a specific namespace:service-account pair via the EKS OIDC provider.

The IRSA configuration for all services:

Service	Namespace	Permissions
auth	comms-control-plane	secretsmanager:GetSecretValue on comms/*/auth/*
room	comms-control-plane	secretsmanager:GetSecretValue on comms/*/room/*, S3 PutObject/GetObject/ListBucket on recordings bucket
signaling	comms-control-plane	secretsmanager:GetSecretValue on comms/*/signaling/*
turn-mgmt	comms-control-plane	secretsmanager:GetSecretValue on comms/*/turn/*
notification	comms-control-plane	secretsmanager:GetSecretValue on comms/*/notification/*, ses:SendEmail
analytics	comms-control-plane	secretsmanager:GetSecretValue on comms/*/analytics/*
billing	comms-control-plane	secretsmanager:GetSecretValue on comms/*/billing/*
presence	comms-control-plane	secretsmanager:GetSecretValue on comms/*/presence/*
gateway	comms-gateway	secretsmanager:GetSecretValue on comms/*/gateway/*
external-secrets	comms-control-plane	secretsmanager:GetSecretValue + ListSecrets on comms/*
cluster-autoscaler	kube-system	autoscaling:SetDesiredCapacity, TerminateInstanceInAutoScalingGroup (tag-scoped), ec2:Describe*

Each trust policy uses two StringEquals conditions: the OIDC audience must be sts.amazonaws.com, and the OIDC subject must match system:serviceaccount:<namespace>:<service-account>. This prevents a compromised pod in one namespace from assuming another service's role, even if it obtains the other service's ServiceAccount token (which Kubernetes RBAC should separately prevent).

IRSA eliminates the need for long-lived AWS access keys in Kubernetes Secrets. Pods receive temporary STS credentials (default 1-hour lifetime, automatically refreshed) via the projected service account token volume. The EKS node role has only the minimum permissions needed for node operation (ECR pull, CNI, SSM); all application-level AWS access flows through IRSA.

7. CI/CD pipeline design: OIDC federation, no long-lived AWS keys

The CI/CD pipeline (GitHub Actions) authenticates to AWS using OIDC federation rather than long-lived IAM access keys. GitHub Actions' OIDC provider is registered as an identity provider in AWS IAM, and a dedicated CI role's trust policy constrains the OIDC subject to the repository and branch:

Condition:
  StringEquals:
    token.actions.githubusercontent.com:aud: "sts.amazonaws.com"
  StringLike:
    token.actions.githubusercontent.com:sub: "repo:<org>/communication_baas:ref:refs/heads/main"

This design has three security benefits:

• No stored secrets. No AWS access keys exist in GitHub Secrets. The OIDC token is generated per workflow run, is short-lived (valid for the duration of the job), and cannot be reused.
• Branch scoping. The trust policy's subject condition restricts which branches can assume the CI role. Only main branch workflows can push to ECR and update ArgoCD Application manifests. Feature branch workflows can run tests but cannot deploy.
• Auditability. Every AssumeRoleWithWebIdentity call appears in CloudTrail with the GitHub repository, branch, and workflow run ID in the session tags. This provides a complete audit trail from Git commit to AWS API call.

The pipeline flow:

1. Build: GitHub Actions builds container images, runs tests, and pushes to GHCR (tagged with Git SHA).
2. Mirror: The pipeline assumes the CI IAM role via OIDC, authenticates to ECR, and pushes the image to the regional ECR repository.
3. Update: The pipeline updates the imageTag in the environment-specific Helm values file (deploy/helm/environments/staging-values.yaml) and commits the change.
4. Sync: ArgoCD detects the values file change and syncs the staging cluster (auto-sync). For production, the updated prod-values.yaml is committed but ArgoCD waits for manual approval.

Consequences

• All cluster state is observable in Git. A git log on the Helm values files and Kustomize manifests produces a complete deployment history with author, timestamp, and diff for each change. This eliminates the "who ran kubectl" problem.
• ArgoCD adds a stateful component to the cluster (ArgoCD's own Redis and repo-server pods). This must be backed up and upgraded independently of the application services. ArgoCD is deployed in a dedicated argocd namespace with its own PDB and resource budgets.
• The manual production gate introduces deployment latency. A change that passes staging validation still requires a human to click "Sync" in ArgoCD. This is an intentional trade-off: deployment speed is less important than deployment safety for a real-time communication platform.
• OIDC federation for CI means that if GitHub's OIDC provider experiences an outage, the pipeline cannot push to ECR or update the cluster. This is an acceptable dependency because GitHub Actions is already a hard dependency for the CI pipeline.
• ESO adds a polling-based secret sync model. There is a window (up to the poll interval) where a rotated secret in AWS Secrets Manager has not yet propagated to the Kubernetes Secret. For most secrets (database URLs, API keys), this window is acceptable. For the TURN shared secret (which clients may cache), the poll interval should be set to 5 minutes to minimize the window.
• Sync waves add ordering constraints that increase the total sync time for a full platform deployment. A clean install with all four waves takes longer than a parallel deploy would. This is acceptable because full installs are rare; incremental syncs (deploying one service) touch only wave 2 and complete in seconds.

Verification

• ArgoCD Application health is verified by confirming that all 11 services show "Healthy" and "Synced" status in the ArgoCD dashboard after a fresh deployment.
• Sync wave ordering is verified by deploying to a clean cluster and confirming that namespaces exist before ESO starts, Secrets exist before application pods start, and ServiceMonitors exist before Prometheus begins scraping.
• IRSA is verified by exec-ing into a pod and running aws sts get-caller-identity, confirming the assumed role matches the expected per-service role ARN.
• OIDC CI authentication is verified by running a GitHub Actions workflow on a feature branch and confirming that the AssumeRoleWithWebIdentity call is rejected (because the trust policy only allows main).
• ESO secret sync is verified by creating a test secret in AWS Secrets Manager, deploying an ExternalSecret referencing it, and confirming the corresponding Kubernetes Secret appears within the poll interval.