An AWS account that has been worked in for five years has a permissions situation nobody holds in their head any more. Eighteen users, six of whom have left the company. Forty roles, a dozen of them belonging to tools that were retired. And somewhere an access key from 2019 that still works.
The reason is never sloppiness. It is always the same sentence, spoken at some point under time pressure: "give them admin for now, we will tighten it later." The "for now" then held for five years.
This article describes the clean-up in the order that actually works, and with the precaution without which it breaks something at night.
Where not to start
The obvious first move is to take permissions away. It is also the most reliable way to end the effort after two weeks.
Because what happens is predictable: some nightly job nobody remembered stops writing to its bucket, fails silently for three days, and on the fourth the question arises whether this clean-up was really necessary. After that nobody touches the topic for a year.
The order that holds turns it around: see first, then the keys, then the permissions. The first two steps carry little risk and deliver the larger part of the improvement.
Seeing what is there
AWS answers two questions by itself, and together they are the entire basis for the work. Who can do what? And who has used any of it?
The first question is answered by the credential report, and it is a single line:
# One row per user: password, keys, last use, age, MFA.
# The best entry point AWS ships with.
aws iam generate-credential-report
aws iam get-credential-report --query Content --output text \
| base64 -d > credential-report.csv
# And per role: which services the role actually touched
# over 400 days.
aws iam generate-service-last-accessed-details \
--arn arn:aws:iam::123456789012:role/deployThe second question is answered by the last-accessed data, and that is the real treasure. A role that touched three services in four hundred days while carrying permissions on thirty is no longer a matter of opinion, it is arithmetic.
On top of that comes the access analyzer, which answers the question that is most uncomfortable in a grown account: what is reachable from outside it? Buckets opened years ago for an exchange with a partner show up there reliably.
The result of this step is a table, not a rebuild. It costs a day and turns a matter of opinion into a set of facts.
The step with the largest effect
Of everything that can be done, one step has by far the best ratio: get rid of long-lived access keys.
A key pair in a developer's file does not expire, is not logged when it gets copied, and survives a resignation effortlessly. The vast majority of published AWS incidents do not start with an over-broad permission, they start with a key like that in a repository or on a laptop.
For humans the replacement is Identity Center: sign-in through the existing directory, roles instead of users, sessions that expire. For machines inside AWS it is roles anyway. Which leaves the case that produces most of the keys: build runs on the outside.
For those there has been a route without any secret at all for years, and it takes twenty lines to set up:
// GitHub gets no keys, it gets to prove who it is. The anchor of
// trust is the token of the run, not a value in a vault.
data "aws_iam_policy_document" "trust" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
principals {
type = "Federated"
identifiers = [aws_iam_openid_connect_provider.github.arn]
}
// Without this condition, ANY repository on GitHub may
// assume this role. It is the most common mistake in this
// migration, and it goes unnoticed because everything works.
condition {
test = "StringEquals"
variable = "token.actions.githubusercontent.com:sub"
values = ["repo:company/shop:ref:refs/heads/main"]
}
}
}The condition on repository and branch is the point at which this migration is either an improvement or a regression. Without it you have replaced a key with a role that half the world may assume.
Where a key is used from, before switching it off
At this point there is regularly one key in the way that nobody can account for. It belongs to a user called deploy, it is four years old, and the answer to the ownership question is "we probably still need that one."
You do not have to guess. The audit trail knows, and the query takes a minute:
# Who worked with exactly this key, and from where? The source
# address in the result usually answers the ownership question
# faster than any company-wide email.
aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA... --start-time "$(date -d '90 days ago' -Iseconds)" --query 'Events[].{time:EventTime,what:EventName,who:Username}'Three outcomes are possible and all three are good. Nothing comes back: deactivate the key, do not delete it. Deactivating is reversible in a second, deleting is not. A known address comes back: the key has an owner. An unknown one comes back: the clean-up has just found its reason.
That middle state is the actual tool: a deactivated key that nobody misses for four weeks gets deleted without discussion. Same idea as switching off a cache, with less drama.
Taking permissions back without knocking something over at night
Only now the permissions, and here too there is an order that makes the difference.
First what was never used. The last-accessed data hands it to you: services a role has not touched in four hundred days can be taken away from it. That is the large, boring part, and it carries no risk a number has not already answered.
Then the exceptions, with an expiry date. Every role that may do more for a specific reason gets the reason as a tag and a date. Without a date, every exception turns back into the state this clean-up came out of.
And never without a way back. Every revocation happens in a window where somebody is watching, and as a change that can be reversed in a minute. A runbook of three lines is enough: what was changed, how to undo it, who to call.
What does not come out of this is a perfectly tailored set of permissions. Two hundred individually tailored policies are just as incomprehensible after a year as what was there before, only with more lines. The common reading of least privilege as fine-grained work on every single action is why efforts like this get stuck.
So that the result does not fall apart at the next piece of manual work in the console, it belongs in code: Terraform drift.
The boundary above beats the rule below
What holds over time does not sit on the individual roles, it sits above them.
A service control policy on the organization does not say who may do something, it says what nobody in this account may do, the administrator included. Three or four such sentences replace dozens of individual rules and stay understandable for years: no region outside Europe, no disabling the audit trail, no deleting its log files, no opening buckets to the public.
The difference is not theoretical. A permission on a role gets widened under pressure during the next incident, and nobody remembers. A boundary on the organization has to be changed in a place where it gets noticed.
The same ordering applies to the accounts themselves: separate accounts for production and development are the most effective permission boundary AWS offers, because they do not rest on rules but on separation. Anyone cleaning up here anyway should pull that along; that is what building an AWS landing zone means.
The same exercise one level up, in the application itself, is covered in Ten holes that sit in every old PHP system.
One week, and what remains
In practice all of the above can be brought to a good state in a week, and the split is regularly the same.
One day of inventory, two days for moving the build runs across and switching off the long-lived keys, one day for removing what demonstrably was never used, one day for the three or four boundaries above. After that the account is not perfectly tailored, but it has no unattended keys, no permissions for people who left, and no way to switch off the audit trail.
What remains is a single habit, and it decides whether the state holds: every exception gets a date. Not a justification, not a request form, a date. Everything else grows back. How I order accounts and access is described on its own page.
This article belongs to a series about systems that already exist. The retrospective orders every article in it by situation.
