High availability is not a feature you switch on. It is a property that only shows itself when something goes wrong.
On one project we ran a Go service that attributed traffic for a global AdTech platform in real time. Every request on the platform passes through this service, which assigns attribution values to it in under 10ms so campaigns can be tracked and billed correctly. If the service goes down, tracking collapses, with direct revenue consequences for a platform in the nine-figure annual revenue range. The service was deployed in EU and US and processed several million requests a day.
Some time ago an AWS region went down.
The service survived. Revenue was not affected. But we found two hidden dependencies that could have become expensive at a less convenient moment. This article describes what happened, why it happened and how we solved it.
The architecture: why we built it this way
The service attributes traffic for an AdTech platform. In concrete terms that means: every request coming into the platform, whether organic traffic or paid campaign, has to be assigned to a channel in real time so media buyers can evaluate and optimize their campaigns. If that assignment is missing, even for a fraction of requests, campaigns are judged wrongly, budgets are allocated wrongly and revenue is not recorded correctly. The latency requirement of under 10ms is not negotiable here: the service sits in the critical path of every single request.
That shaped our architecture decisions.
We deliberately chose a primary-secondary multi-region deployment instead of a classic active-active setup. The reason was data consistency. The service hands out values from a shared pool every day. If both regions wrote at the same time, race conditions would appear that could lead to values being handed out twice. That would be worse than a short outage.
The primary region is the single source of truth and the only one that writes into the DynamoDB Global Table. At the start of the day the secondary region asks the primary region via gRPC whenever new values have to be handed out. Values that have already been assigned are held locally in both regions across several cache layers: in-memory cache for maximum performance, Valkey for distributed regional caching.
Failover control via Route 53 ARC
The biggest risk in a primary-secondary deployment with shared write access is a split-brain scenario: both regions believe they are the primary at the same time and write into the DynamoDB Global Table. That would lead to inconsistent data, in our case to attribution values handed out twice.
To prevent that, we control write authority via AWS Route 53 Application Recovery Controller. ARC is a highly available control plane that runs outside the regions it manages. That is the decisive part: in the outage scenario of a region, ARC is not affected and can reliably signal when the secondary region can safely switch from read-only to read-write mode.
That makes the failover sequence clearly defined. As long as the primary region is available, the secondary region stays in read-only mode and asks the primary via gRPC whenever new values have to be handed out. If the primary region goes down, ARC detects the outage and gives write authority to the secondary region. At no point are there two regions writing at the same time.
AWS Global Accelerator sits in front of both Application Load Balancers and reroutes traffic automatically to the available region as soon as health checks detect an outage. At least two ECS tasks run behind the ALB in each region. Individual task failures within a region cause no downtime.
What happened
AWS Global Accelerator did its job. Health checks detected the outage, traffic was rerouted automatically from the failed US region to eu-central-1. ARC handed write authority to the secondary region. No manual intervention, no downtime for end users. The automation worked.
Then we wanted to deploy.
Dependency 1: the ECR image
Our ECS task definition referenced a Datadog Agent image as a sidecar container for monitoring and observability. That image was hosted in an ECR repository in us-east-1. The ECS task definition in eu-central-1 pulled this image cross-region from the failed US region.
# Problematic configuration
container_definitions = jsonencode([
{
name = "app"
image = "${var.aws_account_id}.dkr.ecr.eu-central-1.amazonaws.com/my-service:latest"
},
{
name = "datadog-agent"
# Image from the US region - hidden cross-region dependency
image = "${var.aws_account_id}.dkr.ecr.us-east-1.amazonaws.com/datadog-agent:latest"
}
])ECS could not pull the image. New task instances could not start. Deployments failed.
The running tasks were not affected, ECS does not replace running containers automatically. But every new deployment, every scaling event, every container restart would have made the problem visible.
Dependency 2: Terraform
The obvious fix would have been to update the task definition via Terraform and change the image to a regional path. But that was not possible either.
Our Terraform state sat safely in eu-west-1 and was not affected. But during terraform apply, Terraform tried to query the current state of the resources in us-east-1 in order to plan changes. Since the region was unreachable, the apply failed.
Error: reading ECS Service: operation error ECS: DescribeServices
RequestError: send request failed
dial tcp: lookup ecs.us-east-1.amazonaws.com: no such hostWe could neither deploy nor change the configuration via Terraform. Both of the tools we normally use depended on the failed region.
That is an important point that often gets overlooked: Terraform plans changes based on the current state of the entire infrastructure. If even one region is unreachable, Terraform cannot make safe assumptions about the overall state and refuses the apply. The state itself plays no role in this.
We were stuck in a double dependency: the ECR image was unreachable and the tool to change the configuration was too. Both problems had clear solutions, but they had to be recognized as systemic weak points first.
The solutions
ECR cross-region replication
AWS ECR supports automatic cross-region replication. Images are replicated to configured target regions automatically after a push.
resource "aws_ecr_replication_configuration" "main" {
replication_configuration {
rule {
destination {
region = "us-east-1"
registry_id = data.aws_caller_identity.current.account_id
}
destination {
region = "eu-central-1"
registry_id = data.aws_caller_identity.current.account_id
}
repository_filter {
filter = ".*"
filter_type = "PREFIX_MATCH"
}
}
}
}After replication was in place, all ECS task definitions were updated to regional image paths:
container_definitions = jsonencode([
{
name = "app"
image = "${var.aws_account_id}.dkr.ecr.eu-central-1.amazonaws.com/my-service:latest"
},
{
name = "datadog-agent"
# Regional now - no cross-region dependency any more
image = "${var.aws_account_id}.dkr.ecr.eu-central-1.amazonaws.com/datadog-agent:latest"
}
])The IAM role of the ECS task execution needs the matching permissions for regional ECR access:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
],
"Resource": "arn:aws:ecr:eu-central-1:ACCOUNT_ID:repository/*"
},
{
"Effect": "Allow",
"Action": "ecr:GetAuthorizationToken",
"Resource": "*"
}
]
}Terraform strategy for multi-region infrastructure
There is no perfect solution for the Terraform dependency, but there are strategies that reduce the risk.
The most effective one is separating Terraform workspaces by region. Instead of one shared configuration for all regions, each region gets its own workspace with its own state. A terraform apply for eu-central-1 then only touches resources in eu-central-1 and is completely independent of us-east-1.
The directory structure for that is straightforward:
infrastructure/
├── modules/
│ ├── ecs-service/
│ ├── alb/
│ └── ecr/
├── regions/
│ ├── eu-central-1/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ └── backend.tf
│ └── us-east-1/
│ ├── main.tf
│ ├── variables.tf
│ └── backend.tf
└── global/
├── ecr-replication/
├── global-accelerator/
└── route53-arc/Every region uses the same modules but has its own state and can be deployed independently. Global resources such as Global Accelerator, ECR replication and Route 53 ARC live in a separate global workspace that is only touched when the global infrastructure changes.
On top of that it is worth documenting AWS CLI commands for critical emergency scenarios that can also be run without Terraform. In an outage there is no time to look up commands. What is not prepared and tested does not exist in an emergency.
Global Accelerator: what worked
It would be unfair to write only about the gaps. In this scenario AWS Global Accelerator did exactly what it was built for.
The configuration was straightforward: two endpoint groups, one per region, with health checks on the ALB health check endpoint. Traffic dial at 100% for both regions in normal operation.
resource "aws_globalaccelerator_accelerator" "main" {
name = "my-service-accelerator"
ip_address_type = "IPV4"
enabled = true
}
resource "aws_globalaccelerator_listener" "main" {
accelerator_arn = aws_globalaccelerator_accelerator.main.id
protocol = "TCP"
port_range {
from_port = 443
to_port = 443
}
}
resource "aws_globalaccelerator_endpoint_group" "eu" {
listener_arn = aws_globalaccelerator_listener.main.id
endpoint_group_region = "eu-central-1"
traffic_dial_percentage = 100
health_check_path = "/health"
health_check_protocol = "HTTPS"
health_check_interval_seconds = 10
threshold_count = 3
endpoint_configuration {
endpoint_id = aws_lb.eu.arn
weight = 100
}
}
resource "aws_globalaccelerator_endpoint_group" "us" {
listener_arn = aws_globalaccelerator_listener.main.id
endpoint_group_region = "us-east-1"
traffic_dial_percentage = 100
health_check_path = "/health"
health_check_protocol = "HTTPS"
health_check_interval_seconds = 10
threshold_count = 3
endpoint_configuration {
endpoint_id = aws_lb.us.arn
weight = 100
}
}A health check interval of 10 seconds and a threshold of 3 means Global Accelerator reacts to an outage within 30 seconds. For a service that directly affects revenue that is acceptable. If you need faster reaction times you can reduce the interval to 30 seconds with a threshold of 1, but you have to expect more false positives.
Checklist: regional isolation
After this incident we developed a systematic checklist for multi-region deployments that I recommend to anyone running active-active or primary-secondary across several regions.
Failover control
Is write authority controlled via an external control plane such as Route 53 ARC? Is the failover sequence documented and tested? Is there a split-brain scenario that could arise from simultaneous writes?
Container images
All ECR repositories configured with cross-region replication? All task definitions updated to regional image paths? Third-party images such as monitoring agents, service mesh proxies and security sidecars checked as well?
Terraform and IaC
Are Terraform workspaces separated by region? Are there documented AWS CLI commands for critical emergency scenarios? Has an apply been tested during a simulated region outage?
Secrets and configuration
All secrets present in AWS Secrets Manager in every region? Parameter Store values replicated regionally? No hardcoded regional endpoints in the configuration?
Deployment pipeline
Does the CI/CD system have access to both regions independently of each other? Build artefacts in regional S3 buckets? Can a deployment be run into each region independently?
Monitoring and observability
Can monitoring agents start regionally without external dependencies? Are CloudWatch dashboards configured separately for each region? Does alerting still work when a region is down?
Failover test
Has a regional outage been actively simulated? Not only testing traffic failover but also deployment, scaling and container restart in the remaining region?
The real lesson
The architecture held. Route 53 ARC handed over write authority cleanly, Global Accelerator rerouted the traffic, the cache layers kept the secondary region alive. The system did exactly what it was built for.
But a region outage always reveals more than you expect. In our case two dependencies that were invisible under normal operations: a monitoring image in the wrong region and a Terraform configuration that fails on partial outages.
Both problems were solvable. Both would have been avoidable. And both would have had real consequences at a less convenient moment, during a hotfix that had to go out in the middle of the outage.
Multi-region deployments are not a one-off architecture project. They are an ongoing process that has to be questioned regularly. Not the application itself, but everything needed to run it: container registries, deployment pipelines, IaC tooling, monitoring. Every new dependency introduced without regional isolation is a hidden gap waiting for its moment.
A simulated region outage in a non-production environment is the only reliable way to find these gaps before AWS finds them for you.
