ADR 0013 — Helm Chart Structure

Status: Accepted (Phase 7) Context date: 2026-07

Context

The platform has 11 deployable services. Each requires the same set of Kubernetes resources: a Deployment, a Service, a ConfigMap, an optional HPA, a PDB, and a ServiceMonitor. The services differ only in their image, port, resource budget, scaling policy, secret references, and (in one case) lifecycle hooks. Maintaining 11 independent charts with duplicated templates would create a maintenance burden where a single fix (e.g., adding a security context field) requires editing 11 files identically.

Decision

1. Three-tier chart architecture: library chart, per-service charts, umbrella chart

Library chart (common, type: library). A Helm library chart at deploy/helm/charts/common/ contains parameterized Go templates for every resource type: _deployment.tpl, _service.tpl, _configmap.tpl, _hpa.tpl, _pdb.tpl, _servicemonitor.tpl, _secret-placeholder.tpl, and _helpers.tpl. Library charts cannot be installed directly; they exist only to be included by other charts.

Per-service charts (type: application). Each service has a thin chart at deploy/helm/charts/<service>/ containing:

• Chart.yaml -- declares the chart name and a dependency on the common library chart (repository: "file://../common").
• values.yaml -- service-specific configuration: image, port, resource requests/limits, autoscaling settings, probe paths, environment variables (including valueFrom.secretKeyRef entries), and optional lifecycle hooks.
• Template files -- one-line wrappers that invoke the library templates: {{ include "common.deployment" . }}, {{ include "common.service" . }}, etc.

A per-service chart's template directory contains no structural logic. All structural logic lives in the library chart. This means a structural change (e.g., adding a topologySpreadConstraints field) is made once in common and inherited by all 11 services.

Umbrella chart (communication-baas, type: application). A parent chart at deploy/helm/charts/communication-baas/ declares all 11 service charts as dependencies in its Chart.yaml. Its values.yaml provides platform-wide defaults, keyed by sub-chart name. A single helm install of the umbrella chart deploys the entire platform.

2. Why a library chart rather than copy-paste or Kustomize bases

The 11 services produce 8 resource types each, totaling 88 rendered manifests. Without a library chart, maintaining consistency across 88 templates requires discipline that does not scale. A library chart provides:

• Single source of truth for security context (runAsNonRoot: true, readOnlyRootFilesystem: true, drop: ALL), probe structure, label conventions, and annotation patterns.
• Conditional rendering driven by values: HPA is only rendered when autoscaling.enabled is true; lifecycle hooks are only rendered when .Values.lifecycle is non-empty; custom HPA metrics are only appended when .Values.autoscaling.customMetrics is defined. This conditional logic would require strategic merge patches in Kustomize, which are harder to reason about.
• Config checksum annotation on the Deployment pod template (checksum/config: {{ sha256sum }}), ensuring pods are rolled when their ConfigMap changes. This is a Helm-specific pattern that Kustomize cannot replicate without a generator plugin.

3. Signaling graceful drain design

The signaling service maintains long-lived WebSocket connections. A naive pod termination would sever active connections, causing clients to experience a disconnect and reconnect cycle. The graceful drain mechanism works as follows:

1. Kubernetes sends SIGTERM and executes the preStop hook.
2. The preStop hook runs: touch /tmp/drain && sleep 60.
3. The sentinel file /tmp/drain is detected by the signaling process's readiness probe logic. The readiness endpoint (/readyz) begins returning HTTP 503, which causes the Service to remove the pod from its endpoints. No new WebSocket connections are routed to this pod.
4. The 60-second sleep gives existing WebSocket clients time to receive a server-initiated close frame and reconnect to a healthy pod.
5. After the sleep completes, the container receives SIGTERM (if it has not already exited) and shuts down.

terminationGracePeriodSeconds is set to 75 (60s drain + 15s buffer for shutdown logic). This exceeds the 60s sleep to ensure Kubernetes does not SIGKILL the pod before the drain completes.

terminationGracePeriodSeconds: 75
lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "touch /tmp/drain && sleep 60"]

This pattern is only applied to the signaling chart. All other services handle stateless HTTP requests and drain naturally via the default Kubernetes SIGTERM flow (readiness probe fails, in-flight requests complete within the default 30s grace period).

4. Resource budget rationale

All control-plane services use Burstable QoS class, meaning resource requests are set lower than limits. This is a deliberate choice for the control plane:

• Requests represent the steady-state baseline. A control-plane pod handling typical API traffic uses far less CPU and memory than its peak (e.g., a burst of room joins or a large analytics query).
• Limits cap the burst ceiling to prevent a single runaway pod from starving its neighbors on the same node.
• Guaranteed QoS (requests == limits) is reserved for media-plane pods (LiveKit SFU) where latency predictability justifies dedicating resources, and for data-plane stores where memory pressure causes OOM kills.

Specific budget decisions:

Service	CPU req/limit	Mem req/limit	Rationale
gateway	100m/500m	128Mi/256Mi	Stateless proxy; CPU scales with request rate, not per-request cost
auth	100m/500m	128Mi/256Mi	JWT issuance and JWKS serving are lightweight
room	200m/1000m	256Mi/512Mi	Heavier: DB queries, LiveKit API calls, S3 uploads for recordings
signaling	200m/1000m	256Mi/512Mi	WebSocket fan-out is CPU-intensive under load
turn-mgmt	50m/200m	64Mi/128Mi	Stateless HMAC credential issuance; minimal compute
notification	100m/500m	128Mi/256Mi	Webhook dispatch with retry backoff
analytics	100m/500m	128Mi/256Mi	Kafka consumer + ClickHouse writer; batched, not latency-critical
billing	100m/500m	128Mi/256Mi	Kafka consumer + periodic aggregation
presence	100m/500m	128Mi/256Mi	Redis reads/writes for online status
admin-gw	100m/500m	128Mi/256Mi	Low-traffic admin API
admin-dash	100m/300m	128Mi/256Mi	Next.js SSR; lower CPU ceiling since it serves an internal UI

5. Secret handling: valueFrom.secretKeyRef with External Secrets Operator seam

Secrets (database URLs, API keys, JWT seeds, internal tokens) are never stored in Helm values or ConfigMaps. Each service's values.yaml declares secret references as valueFrom.secretKeyRef entries in the env array:

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: auth-secrets
        key: DATABASE_URL

The referenced Secret objects (auth-secrets, room-secrets, signaling-secrets, etc.) are not created by the Helm charts. This is intentional. The charts define the contract (which Secret name and key each service expects), and the secret lifecycle is owned by one of:

• External Secrets Operator (ESO) -- in production, ESO syncs secrets from a cloud provider's secret store (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault) into Kubernetes Secret objects. The Helm charts are ESO-compatible out of the box because they reference Secrets by name without creating them.
• Manual kubectl create secret -- for dev and staging environments where a cloud secret store is not available.
• Sealed Secrets or SOPS -- for GitOps workflows where encrypted secrets are committed to the repository.

The library chart includes a _secret-placeholder.tpl template that can optionally render a placeholder Secret (with empty values) to prevent pod startup failures in development. This is opt-in and disabled by default.

6. HPA strategy

Horizontal Pod Autoscaler is enabled selectively based on the service's traffic pattern:

Service	HPA enabled	Metric	Min	Max	Rationale
gateway	yes	CPU utilization (70%)	2	10	Entry point; scales with external request volume
signaling	yes	CPU (70%) + ws_connections_active (avg 500)	2	10	WebSocket count is a better load signal than CPU for connection-holding services
All others	no	--	2	varies	Steady-state traffic is predictable; manual scaling or enabling HPA in prod values is preferred over premature autoscaling

The signaling chart is the only service with a custom HPA metric (ws_connections_active). This metric is exposed on the service's Prometheus metrics endpoint and scraped by the Prometheus Adapter, which makes it available to the HPA controller via the custom.metrics.k8s.io API. The target of 500 connections per pod means the HPA scales out before connection density causes latency degradation in the WebSocket fan-out path.

HPA for other services (room, auth, notification, etc.) is pre-configured but disabled (autoscaling.enabled: false) in the default values. Production values files enable it with tuned thresholds.

7. PDB strategy

Every service has a PodDisruptionBudget with minAvailable: 1. This guarantees that at least one pod remains available during voluntary disruptions (node drains, cluster upgrades, spot instance preemptions).

Design rationale:

• minAvailable: 1 rather than maxUnavailable: 1 -- with a default replica count of 2, both formulations are equivalent (1 of 2 must be available == at most 1 of 2 can be unavailable). minAvailable: 1 is preferred because its semantics remain correct regardless of replica count: even if someone scales a service to 1 replica, the PDB prevents voluntary eviction of the last pod. maxUnavailable: 1 with 1 replica would allow eviction.
• minAvailable: 1 rather than minAvailable: 50% -- percentages introduce rounding ambiguity at low replica counts and are harder to reason about during incident response.
• All services, including non-critical ones (admin-dashboard) -- the cost of a PDB is negligible, and the protection against accidental disruption during upgrades applies to every workload. It is easier to have a uniform policy than to decide per-service whether disruption is acceptable.

PDBs interact with the signaling graceful drain: when a node drain triggers pod eviction, the PDB ensures at least one signaling pod remains in-service while the draining pod runs its 60-second preStop hook. This prevents a scenario where both signaling pods are evicted simultaneously during a rolling node upgrade.

Consequences

• Adding a new service to the platform requires four files: Chart.yaml (7 lines), values.yaml (with service-specific overrides), and one-line template wrappers. No Go template logic is duplicated. The umbrella chart's Chart.yaml gains one dependency entry.
• A structural change to all Deployments (e.g., adding topologySpreadConstraints or a sidecar) is a single edit to _deployment.tpl in the library chart, followed by a version bump. All 11 services inherit the change.
• The signaling drain design adds 60 seconds to pod termination time, which slows rolling deployments of the signaling service. This is acceptable because signaling deployments are infrequent relative to the user experience cost of dropped WebSocket connections.
• Burstable QoS for control-plane services means pod performance varies with node contention. If this causes latency spikes under load, the fix is to raise requests toward limits (moving toward Guaranteed QoS) rather than redesigning the chart structure.
• The secret-by-reference pattern means helm install alone does not produce running pods; the referenced Secrets must exist first. This is documented in the umbrella chart's README and enforced by the startup probe (pods fail startup if required env vars are missing).

Verification

• helm template on each per-service chart confirms it renders valid Kubernetes manifests with the expected resource values.
• helm dependency build on the umbrella chart confirms all 11 sub-chart dependencies resolve via file:// references.
• The signaling drain is verified by deploying to a test cluster, establishing a WebSocket connection, initiating a pod delete, and confirming the client receives a close frame and reconnects to the surviving pod within the 60-second window.
• PDB behavior is verified by running kubectl drain on a node hosting both signaling pods and confirming that only one pod is evicted at a time.