ADR 0012 — Kubernetes Deployment Strategy

Status: Accepted (Phase 7) Context date: 2026-07

Context

The EthioConnect platform comprises 11 independently deployable services (auth, room, signaling, turn-mgmt, notification, analytics, billing, presence, gateway, admin-gateway, admin-dashboard), plus stateful data stores (PostgreSQL, Redis, Kafka, ClickHouse, MinIO) and an observability stack (Prometheus, Grafana, OTel Collector). These services need a packaging and deployment strategy that supports environment promotion (dev, staging, prod), zero-trust network isolation between architectural planes, and GitOps-driven continuous delivery.

The key constraints are:

• The deployment architecture must mirror the logical separation defined in the system's deployment diagram (docs/08): gateway, control-plane, media-plane, data, and observability are distinct failure and security domains.
• Services share an identical structural pattern (Deployment, Service, ConfigMap, HPA, PDB, ServiceMonitor) but differ in resource budgets, scaling policies, and secret references.
• The platform must be runnable on a single developer workstation (Docker Compose today) and on multi-node production clusters without forking the deployment artifacts.

Decision

1. Helm as primary packaging, Kustomize base manifests as the GitOps seam

Helm is the primary packaging tool. Every service is a Helm chart, and an umbrella chart (communication-baas) composes all 11 service charts into a single installable unit. Environment-specific configuration is expressed as values files (one per environment), not as separate charts.

Kustomize base manifests under deploy/k8s/base/ define cluster-level resources that are not service-specific and have no templating needs: namespaces and NetworkPolicies. These are applied directly by a GitOps controller (ArgoCD, Flux) or kubectl apply -k and do not flow through Helm. This split keeps Helm focused on application lifecycle and Kustomize focused on cluster policy.

2. Alternatives considered

Raw manifests only. Writing plain YAML for 11 services with identical structure would produce roughly 66 manifest files (Deployment, Service, ConfigMap, HPA, PDB, ServiceMonitor per service) with near-identical content. Maintenance cost scales linearly with service count, and environment differentiation requires either envsubst or manual duplication. Rejected.

Kustomize-only. Kustomize handles overlays well but lacks the templating power needed for conditional resources (HPA only when autoscaling.enabled, lifecycle hooks only when defined, custom metrics only for signaling). Strategic merge patches become unwieldy when the base resource must vary structurally across services. Rejected as the primary tool, retained for cluster-level resources where overlays are sufficient.

Helm-only. Helm could manage namespaces and NetworkPolicies, but these are cluster-scoped concerns that benefit from being visible, auditable, and diffable without requiring helm template. GitOps controllers handle raw manifests natively. Putting everything in Helm would couple cluster policy changes to application release cycles. Rejected as the sole tool.

3. Namespace topology

Five namespaces partition the cluster by architectural plane:

Namespace	Plane	Contents
comms-gateway	gateway	API gateway, admin gateway, admin dashboard
comms-control-plane	control-plane	auth, room, signaling, turn-mgmt, notification, analytics, billing, presence
comms-media-plane	media-plane	LiveKit SFU pods, recording egress jobs (dedicated node pool)
comms-data	data	PostgreSQL, Redis, Kafka, ClickHouse, MinIO
comms-observability	observability	Prometheus, Grafana, OTel Collector, OpenSearch

Every namespace carries two labels used by NetworkPolicy selectors: app.kubernetes.io/part-of: communication-baas (platform membership) and comms.io/plane: <plane> (architectural plane). These labels decouple policy rules from namespace names, making them resilient to namespace renaming.

Rationale for five rather than per-service namespaces: per-service namespaces (11+) create operational overhead in RBAC, resource quotas, and NetworkPolicy management without a proportional isolation benefit. The five-namespace model maps directly to the deployment diagram's failure domains: a compromised gateway pod cannot reach data stores; a compromised data pod cannot initiate connections to the control plane. Within the control plane, pod-level NetworkPolicies (using app.kubernetes.io/name selectors) enforce service-to-service boundaries.

4. NetworkPolicy approach: default-deny with explicit allow rules

Every namespace has a default-deny-all NetworkPolicy that blocks both ingress and egress for all pods. Traffic is then opened incrementally through seven allow-policy files:

Policy file	What it opens
allow-dns.yaml	Egress to kube-dns (port 53) in kube-system, all namespaces
allow-gateway-ingress.yaml	External ingress to gateway on port 8090 (the single entry point)
allow-gateway-to-control-plane.yaml	Gateway egress to control-plane on service ports 8081-8088
allow-control-plane-to-data.yaml	Control-plane egress to data on store ports (5432, 6379, 9092, 8123, 9000); gateway egress to data on 6379 (rate limiting)
allow-control-plane-internal.yaml	Pod-to-pod traffic within control-plane, scoped by app.kubernetes.io/name labels to documented call paths only
allow-observability.yaml	Observability egress to all namespaces for metrics scraping; all namespaces egress to observability for OTLP telemetry
allow-gateway-ingress.yaml	External ingress to the gateway namespace on port 8090

This is a zero-trust posture: any new service or port is denied by default until an explicit policy is added and reviewed. The policies reference namespace labels (comms.io/plane) and pod labels (app.kubernetes.io/name), not IP addresses, so they survive pod rescheduling and node changes.

5. Environment promotion

Environment promotion follows a values-file hierarchy:

deploy/helm/charts/communication-baas/values.yaml        # defaults (dev-safe)
deploy/helm/environments/staging-values.yaml              # staging overrides
deploy/helm/environments/prod-values.yaml                 # production overrides

The base values.yaml embedded in the umbrella chart defines dev-safe defaults: replicaCount: 2, autoscaling disabled for most services, and imageTag: latest. Staging and production values files override replica counts, enable autoscaling for latency-sensitive services, set real image tags, and adjust resource requests/limits. A GitOps controller (ArgoCD or Flux) points at the appropriate values file per cluster.

Helm install invocations per environment:

helm install communication-baas ./deploy/helm/charts/communication-baas \
  -f deploy/helm/environments/staging-values.yaml

6. What is implemented vs. deferred

Implemented:

• All 11 per-service Helm charts with the shared library chart dependency.
• Umbrella chart (communication-baas) composing all services.
• Namespace manifests for all five namespaces.
• Full NetworkPolicy set: default-deny plus seven allow-policy files covering all documented traffic flows.
• Common library chart templates: Deployment, Service, ConfigMap, HPA, PDB, ServiceMonitor, secret-placeholder.
• Per-service values with resource budgets, probe paths, secret references, and lifecycle hooks.
• ServiceMonitor resources for Prometheus scraping.

Deferred:

• Terraform for cloud infrastructure (VPC, node pools, managed databases, IAM). The Helm charts assume a running cluster; Terraform provisions what they run on. Deferred until the first cloud deployment target is selected.
• ArgoCD/Flux installation and ApplicationSet definitions. The charts and manifests are GitOps-ready (declarative, values-file-driven), but the controller itself is not installed or configured.
• Service mesh (Istio/Linkerd). NetworkPolicies provide L3/L4 isolation. mTLS between services and L7 traffic policies are a service mesh concern deferred until the threat model requires them. The namespace and label structure is designed to be compatible with mesh injection.
• Ingress controller and TLS certificates. The allow-gateway-ingress NetworkPolicy opens port 8090 for external traffic, but no Ingress resource or cert-manager configuration is defined. These are cluster-specific and deferred.
• Kustomize overlays for environment-specific base resources. The deploy/k8s/overlays/ directory is reserved but empty. When environment-specific namespace annotations or NetworkPolicy tweaks are needed, overlays will be added.

Consequences

• Deploying the entire platform is a single helm install of the umbrella chart, plus kubectl apply -k deploy/k8s/base/ for cluster policy. This keeps the operational surface small.
• Adding a new service requires creating a thin chart (Chart.yaml + values.yaml + one-line template files), adding it as a dependency in the umbrella chart, and adding its NetworkPolicy entries. The library chart eliminates boilerplate.
• The five-namespace model means RBAC can be scoped per plane. An SRE responsible for the data plane does not need access to control-plane pods, and vice versa.
• Default-deny NetworkPolicies mean that a misconfigured or compromised pod is contained to its namespace by default. This is a meaningful defense-in-depth layer even without a service mesh.
• The Helm + Kustomize split introduces two tools that operators must understand. This is mitigated by keeping the boundary clean: Helm for applications, Kustomize for cluster policy.

Verification

• helm template on the umbrella chart renders all 11 services with correct resource values, proving the library chart and values hierarchy work.
• kubectl apply --dry-run=server -k deploy/k8s/base/ validates the namespace and NetworkPolicy manifests against a live cluster's API.
• NetworkPolicy correctness is verified by deploying to a Kind cluster with a CNI that enforces NetworkPolicies (Calico) and confirming that cross-namespace traffic is blocked unless explicitly allowed.