The story about infrastructure as code is tidy: everything is in the repository, the environment appears at the push of a button, and nobody clicks around in a console any more. In environments that existed before the code, reality looks different.
There you find resources nobody has codified, resources that are in the code and were changed in the console, and resources that exist in the code and no longer in reality. The difference between described and actual state is called drift, and in a grown environment it is not an exception.
This article is about making drift visible, the three possible answers to it, and where codification deliberately stops.
Drift is normal, not a failure of the team
The first reaction to drift is usually a question of blame: who changed that by hand? That is the wrong question, because it explains the behaviour without changing it.
Drift comes from four sources, and only one of them is carelessness.
The emergency. At three in the morning a security group gets opened, an instance resized, a limit raised. That is right, and it would be wrong at that moment to open a branch and wait for a review. What is missing is the step afterwards.
The provider itself. A managed service sets values it considers correct: a default tag, an adjusted configuration after a version bump, an automatically created rule. That is not a person and it is still drift.
Another tool. Kubernetes creates a load balancer, an autoscaler changes an instance count, an application creates a queue. All legitimate, none of it in the code.
The shortcut. Somebody clicks something because it is faster. That is the source everybody pictures, and in my experience the smallest of the four.
From that list follows the stance that holds: drift gets measured and answered, not prevented. An environment with no drift at all only exists where nobody has access any more.
Making drift visible
The technical part is pleasingly small. terraform plan already compares described with actual state; all that is missing is somebody looking regularly.
# -detailed-exitcode: 0 = same, 1 = error, 2 = drift
terraform plan -detailed-exitcode -lock=false -out=/dev/null
case $? in
0) echo "no drift" ;;
2) echo "DRIFT" ;;
*) echo "plan failed"; exit 1 ;;
esac-lock=false matters: a run that only looks should not block somebody who is actually changing something.
That run belongs on a schedule, once a day, with a message to chat. Two things decide whether anything useful comes of it.
The message names the resources, not just the count. "Drift in 14 resources" nobody reads twice. "Security group web-sg: one rule more" gets read.
The message goes to somebody by name. A notice in a channel with no addressee gets dismissed after two weeks. That is not a technical detail but an organizational one, and it decides whether the whole setup is worth anything.
The three answers to drift
Every finding gets one of three answers, and all three are right. The only wrong move is giving none.
Adopt it. The change was good, so it goes into the code. That is the normal case after an emergency: the security group stays open, but it is now in the repository, with a comment saying why.
Revert it. The change was not wanted, terraform apply restores the described state. That is the second most common case and the one where the plan has to be read first, because reverting can cost data too.
Ignore it, explicitly. Some fields do not belong in the code because they are set from outside. For those there is a declaration in the code rather than silent tolerance:
resource "aws_ecs_service" "api" {
name = "api"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.api.arn
desired_count = 2
lifecycle {
# The autoscaler sets desired_count at runtime. Without this line every
# apply would reset the count to 2, including in the middle of a load
# spike. The 2 above is the starting value, not the target.
ignore_changes = [desired_count]
}
}The comment in the example is the actual content. An ignore_changes without a reason is indistinguishable from carelessness in two years, and then nobody dares remove it.
How much belongs in code at all depends on what the migration produced: Lift and shift or replatforming.
What deliberately stays out of the code
Completeness is not the goal. Three kinds of resource I regularly leave out, and in all three cases that is a decision that gets written down.
Whatever another tool owns. A load balancer created and managed by Kubernetes belongs to Kubernetes. Codifying it as well creates two owners for one resource, and the argument between them ends in a loop of changes.
Whatever is one-off and dangerous. The root account, the organizational structure, an encryption key. Those change rarely, and destroying one by accident outweighs the gain in reproducibility. Where they do go into the code, they get a guard.
resource "aws_kms_key" "data" {
description = "Encryption of application data"
deletion_window_in_days = 30
lifecycle {
# Destroying this by accident makes every piece of data encrypted with
# it unreadable. That outweighs any convenience.
prevent_destroy = true
}
}Whatever is going away anyway. An environment being switched off in three months does not need describing. That is the same logic as freezing a legacy system.
Adopting what exists, and where that stops
For environments that predate the code, import is the first step. Since import blocks exist, it is traceable and visible in the plan, rather than a command somebody once typed.
import {
to = aws_security_group.web
id = "sg-0a1b2c3d4e5f"
}
resource "aws_security_group" "web" {
name = "web-sg"
vpc_id = aws_vpc.main.id
# ... rules, copied from what exists
}The sequence that has proved itself is always the same: import, run plan, adjust the code until the plan is empty. An empty plan is the acceptance test. While changes are still listed, the code describes something other than reality, and that is worse than no code, because it pretends to be safe.
Where to stop: I import from the outside in. Network, security groups, permissions, databases, compute. What is left after that is usually leftovers from experiments, and those do not get a description, they get a decision: needed or gone.
State is the one thing that can really break
Code can be reverted, resources can be recreated. The state file is the only piece whose loss does real damage: after that Terraform no longer knows which resource belongs to which block, and an apply would create everything a second time.
Three settings prevent the worst cases, and all three cost nothing.
terraform {
backend "s3" {
bucket = "company-terraform-state"
key = "production/terraform.tfstate"
region = "eu-central-1"
# Lock against concurrent runs. Without it two people write at the same
# time, and the resulting state describes no reality at all.
use_lockfile = true
encrypt = true
}
}On top of that, versioning on the storage location so an earlier state can be restored. That is the backup you need once a year and then urgently.
And the rule that follows: nobody edits the state file by hand. Where something is wrong there is terraform state mv and terraform state rm, both logged and both reversible. A text editor on a state file is the route to an afternoon nobody forgets.
The most common place for manual work in the console is permissions, and that is where cleaning up pays first: Cleaning up IAM in a grown AWS account.
The rule for emergencies
Finally the part that decides whether the whole setup is lived or merely exists.
At three in the morning nobody changes things through a branch. Demanding that produces either a longer outage or secret changes, and both are worse than an open rule.
The rule I recommend has three parts. In an emergency anybody with access may change things directly, without asking. The change gets reported in the same shift, one line in the incident channel, not documentation. On the next working day somebody decides whether it is adopted or reverted, and the daily run above makes sure it is not forgotten.
With that, drift stops being a violation and becomes a process with a route. That is the difference between an environment where infrastructure as code actually applies and one where the repository describes a wish.
How I introduce Terraform into an existing environment is on its own page; the import above is the first block of work there.
This article belongs to a series about systems that already exist. The retrospective orders every article in it by situation.
