ADR 0015 — Cloud Infrastructure Strategy

Status: Accepted (Phase 10) Context date: 2026-07

Context

The EthioConnect platform has been developed through Phases 1-9 with Docker Compose for local development, Helm charts for Kubernetes packaging (ADR 0012, 0013), and a five-namespace deployment topology. The platform now requires a production cloud target. The infrastructure must support 11 application services, stateful data stores (Aurora PostgreSQL, Redis, Kafka, ClickHouse, MinIO), a Coturn TURN fleet, and an observability stack, all deployed across multiple availability zones with security isolation matching the namespace topology defined in ADR 0012.

The key constraints are:

• Every service owns its own database (DDD isolation, per project conventions).
• Coturn runs on bare EC2, not Kubernetes, because TURN servers require stable public IPs and a relay port range (UDP 49152-65535) that conflicts with Kubernetes networking (ADR 0008).
• The infrastructure must be reproducible across staging and production environments with cost-appropriate sizing.
• Cloud portability is a future concern but must not be sacrificed by baking provider-specific assumptions into application code.

Decision

1. AWS as the initial cloud target

AWS is the first cloud target. The platform's data plane requirements (managed Kafka, Aurora PostgreSQL, container registry with VPC endpoints, Elastic IPs for TURN) map directly to mature AWS services. GCP and Azure are viable alternatives, but selecting one provider and building production-grade Terraform modules for it is more valuable than building provider-agnostic abstractions prematurely.

Cloud portability is preserved at two levels: (a) application services connect to infrastructure through environment variables and DNS names, not AWS SDK calls, and (b) Terraform modules expose a stable interface (outputs like cluster_endpoint, writer_endpoint, bootstrap_brokers) that a GCP or Azure module could implement without changing the root composition.

2. Aurora PostgreSQL over plain RDS

The RDS module (deploy/terraform/modules/rds/) provisions Aurora PostgreSQL 16 clusters rather than standard RDS PostgreSQL instances. Aurora was chosen for three reasons:

• Automated storage scaling. Aurora storage grows automatically in 10 GB increments up to 128 TiB. Standard RDS requires pre-provisioned EBS volumes and manual resizing, which creates operational toil for a multi-database platform.
• Faster failover. Aurora failover completes in under 30 seconds (typically 10-15s) compared to 60-120 seconds for standard RDS Multi-AZ. For a real-time communication platform where room state, auth tokens, and billing records are in PostgreSQL, shorter failover windows directly reduce user-facing impact.
• Reader endpoint. Aurora provides a cluster reader endpoint that load-balances read traffic across reader instances. Services like analytics and billing that perform heavy read queries can target the reader endpoint, offloading the writer. Standard RDS Multi-AZ standbys do not serve read traffic.

The module supports configurable instance counts (instance_count variable, default 2: 1 writer + 1 reader), Performance Insights for query analysis, and pg_stat_statements via a cluster parameter group. Instance sizing defaults to db.r7g.large for production and db.t4g.medium for staging.

3. Separate Aurora cluster per service (production) with shared-mode for staging

In production, each service that requires PostgreSQL (auth, room, signaling, notification, analytics, billing) gets its own Aurora cluster. This enforces DDD database isolation: a schema migration in the billing service cannot lock tables in the auth service, connection pool exhaustion in analytics cannot starve room, and each cluster can be sized independently.

In staging, a single Aurora cluster with separate databases (one per service) is acceptable to reduce cost. The RDS module is invoked once per service in the production root module and once with a shared database in the staging root module. The application services are unaware of this distinction because they connect via DATABASE_URL environment variables that resolve to different endpoints per environment.

4. Amazon MSK over self-managed Kafka

The MSK module (deploy/terraform/modules/msk/) provisions Amazon Managed Streaming for Apache Kafka rather than deploying Kafka on Kubernetes. Self-managed Kafka on EKS would require managing ZooKeeper (or KRaft), broker storage lifecycle, rolling upgrades, and monitoring -- operational overhead that is not justified when Kafka is used as an event bus rather than a core product differentiator.

MSK provides:

• Managed broker patching and upgrades with zero-downtime rolling updates.
• SASL/SCRAM authentication with credentials stored in Secrets Manager, eliminating plaintext credentials in Kubernetes Secrets.
• KMS encryption at rest and TLS encryption in transit (enforced: client_broker = "TLS", in_cluster = true).
• Storage auto-scaling from the provisioned volume size up to 2 TB per broker, triggered at 70% utilization via Application Auto Scaling.
• Per-topic-per-broker monitoring (enhanced_monitoring = "PER_TOPIC_PER_BROKER") with CloudWatch metrics, providing the granularity needed to detect consumer lag and partition skew.

The cluster is configured with default.replication.factor = 3 and min.insync.replicas = 2, ensuring that no single broker failure causes data loss for the platform's event streams (room events, billing events, analytics events).

5. ECR in addition to GHCR

The ECR module (deploy/terraform/modules/ecr/) creates one repository per service with immutable tags, scan-on-push, and KMS encryption. ECR is used alongside GitHub Container Registry (GHCR) for three reasons:

• VPC endpoints. ECR supports AWS PrivateLink, allowing EKS nodes in private subnets to pull images without traversing NAT Gateways. This eliminates NAT Gateway data processing charges for image pulls (which are frequent during rolling deployments of 11 services) and removes an internet dependency from the critical deployment path.
• IAM-native authentication. EKS nodes authenticate to ECR via their instance role (AmazonEC2ContainerRegistryReadOnly policy attached in the EKS module). No imagePullSecrets configuration is needed in Helm charts, reducing secret management complexity.
• Regional locality. Images are stored in the same region as the EKS cluster, providing lower pull latency compared to GHCR (hosted in US regions regardless of deployment region).

GHCR remains the CI build target. The CI pipeline pushes to GHCR first (for PR image availability and open-source visibility), then mirrors to ECR for production deployment. The ECR lifecycle policy retains the last 30 tagged images and expires untagged images after 7 days.

6. EC2 Auto Scaling Group for Coturn

Coturn TURN servers run on EC2 instances in an Auto Scaling Group (deploy/terraform/modules/coturn-fleet/) rather than as Kubernetes pods. This decision follows from ADR 0008 (TURN architecture) and is driven by three requirements that Kubernetes cannot satisfy cleanly:

• Stable public IPs. TURN clients discover relay addresses via the TURN protocol's Allocate response, which contains the server's public IP. Kubernetes pod IPs are private and ephemeral. Elastic IPs pre-allocated by Terraform (one per minimum instance) provide stable, predictable addresses for DNS records and firewall allowlisting.
• UDP relay port range. Coturn allocates relay ports from UDP 49152-65535 (16,383 ports). Exposing this range through a Kubernetes Service would require 16,383 NodePort or LoadBalancer entries, which is operationally impractical. On EC2, the security group opens this range directly.
• Compute-optimized instances. Coturn is CPU-bound (TURN relay is essentially packet forwarding). The fleet uses c7i.xlarge instances, which provide high single-thread performance at a lower cost-per-packet than general-purpose instances used in EKS node groups.

The fleet is deployed across public subnets (one per AZ) with an NLB for UDP health checking and initial TURN signaling on port 3478. Instances run Amazon Linux 2023, enforce IMDSv2, and retrieve the TURN shared secret from Secrets Manager at boot via cloud-init. SSH access is restricted to the bastion host's security group.

7. VPC design: 3 AZs, three subnet tiers, per-AZ NAT for production

The VPC module (deploy/terraform/modules/vpc/) implements a three-tier subnet architecture across three availability zones:

Tier	CIDR size	Hosts per AZ	Contents	Internet access
Public	/20	4,094	NAT Gateways, ALBs, Coturn TURN servers	Direct (Internet Gateway)
Private	/20	4,094	EKS worker nodes, application pods	Outbound via NAT Gateway
Isolated	/24	254	Aurora PostgreSQL clusters	None

Design rationale for each tier:

• Public subnets host resources that must be internet-reachable: the NLB for Coturn, ALBs for the API gateway, and NAT Gateways. Coturn instances are placed here because they require direct client connectivity on UDP 3478 and the relay port range.
• Private subnets host EKS worker nodes. Pods can reach the internet (for image pulls, webhook delivery, external API calls) via NAT Gateways, but are not directly addressable from the internet.
• Isolated subnets host Aurora PostgreSQL. These subnets have no route to the internet (no NAT Gateway route), which means a compromised database instance cannot exfiltrate data to external endpoints. The only ingress is from the EKS node security group on port 5432.

NAT Gateway topology varies by environment:

• Production: One NAT Gateway per AZ (single_nat_gateway = false). This eliminates cross-AZ data transfer charges and provides AZ-level fault isolation. If one NAT Gateway fails, only pods in that AZ lose outbound internet access; pods in other AZs are unaffected.
• Staging: Single NAT Gateway (single_nat_gateway = true) to reduce cost. The staging environment does not require AZ-level fault isolation.

VPC Flow Logs are conditionally enabled (when flow_log_bucket_arn is set), writing to S3 with a 60-second aggregation interval for network forensics and compliance.

8. Cloud portability via module interfaces

Every Terraform module exposes its integration surface through outputs, not through provider-specific resource references. The root module (deploy/terraform/environments/staging/) consumes these outputs to wire modules together:

• VPC module outputs: vpc_id, public_subnet_ids, private_subnet_ids, isolated_subnet_ids
• EKS module outputs: cluster_endpoint, cluster_ca_certificate, oidc_provider_arn, node_security_group_id
• RDS module outputs: cluster_endpoint, reader_endpoint, master_password_secret_arn
• MSK module outputs: bootstrap_brokers_sasl_scram, sasl_secret_arn

A GCP or Azure implementation would create equivalent modules with the same output names, and the root composition would not change structurally. This is not full cloud abstraction -- the module internals are provider-specific -- but it prevents provider lock-in from leaking into inter-module wiring.

9. What is implemented vs. deferred

Implemented:

• VPC module with three-tier subnets, per-AZ NAT toggle, VPC Flow Logs, and EKS subnet tagging.
• EKS module with three managed node groups (control-plane, media-plane, observability), KMS envelope encryption, OIDC provider for IRSA, encrypted EBS volumes, and IMDSv2 enforcement.
• RDS module with Aurora PostgreSQL 16, configurable instance count, isolated subnet placement, Performance Insights, and pg_stat_statements.
• MSK module with SASL/SCRAM authentication, KMS encryption, TLS enforcement, storage auto-scaling, and per-topic-per-broker monitoring.
• ECR module with per-service repositories, immutable tags, scan-on-push, KMS encryption, and lifecycle policies.
• Coturn fleet module with EC2 ASG, NLB, Elastic IPs, cloud-init provisioning, IMDSv2 enforcement, and bastion-only SSH.
• IAM module with IRSA roles for all 11 services plus External Secrets Operator and Cluster Autoscaler.
• Staging environment root module wiring all modules together.

Deferred:

• Multi-region deployment. The current design is single-region (us-east-1). Multi-region requires Aurora Global Database, MSK replication, cross-region EKS federation, and global DNS routing. These are deferred until latency requirements or regulatory compliance demand them.
• Service mesh (Istio/Linkerd). NetworkPolicies (ADR 0012) provide L3/L4 isolation. mTLS between pods and L7 traffic policies are deferred until the threat model requires them. The namespace and label structure is mesh-injection-compatible.
• VPC endpoints for AWS services. ECR, Secrets Manager, and S3 VPC endpoints would eliminate NAT Gateway dependencies for AWS API calls. Deferred to reduce initial Terraform complexity; to be added when NAT Gateway costs become material.
• Production environment root module. The staging root module exists and validates module composition. The production root module (with per-service Aurora clusters, per-AZ NAT, larger instance types, and deletion protection enabled) is deferred until the first production deployment.

Consequences

• All cloud infrastructure is provisioned declaratively via Terraform. Manual AWS Console changes are prohibited by convention and detectable via terraform plan drift.
• The three-tier subnet design means database instances are unreachable from the internet by network topology, not just by security group rules. This is a defense-in-depth layer that survives security group misconfiguration.
• Per-AZ NAT Gateways in production cost approximately $100/month (3 x NAT Gateway hourly charge + data processing). This is justified by the AZ fault isolation benefit for a real-time communication platform.
• Aurora PostgreSQL costs more than standard RDS (approximately 20% premium on instance pricing). This is justified by the automated storage scaling, faster failover, and reader endpoint benefits described above.
• MSK costs more than self-managed Kafka on EKS (approximately 30-40% premium). This is justified by eliminating the operational burden of Kafka lifecycle management, which would otherwise require dedicated SRE capacity.
• The Coturn EC2 fleet operates outside Kubernetes, creating a split operational model. Coturn instances are managed via ASG lifecycle, cloud-init, and SSM Session Manager rather than kubectl. This is an accepted trade-off for the networking requirements described above.

Verification

• terraform validate on every module confirms HCL syntax and provider schema compliance.
• terraform plan on the staging environment confirms that all modules compose correctly and no circular dependencies exist.
• VPC subnet CIDR allocation is verified by inspection: public /20s do not overlap private /20s, and isolated /24s fall outside both ranges within the /16 VPC CIDR.
• Security group rules are verified by confirming that Aurora ingress is restricted to the EKS node security group, MSK ingress is restricted to SASL/TLS on port 9096, and Coturn SSH ingress is restricted to the bastion security group.
• IRSA trust policies are verified by confirming each role's trust condition constrains the OIDC subject to a specific namespace:service-account pair, preventing cross-service impersonation.