In almost every grown system there is a place nobody wants to touch, and often it is a cache. It was added years ago because a page was too slow. It worked. It has been sitting there ever since, and nobody knows exactly what it holds, when it gets discarded, or what happens if you turn it off.
A cache is not a mistake. It is a promise: I know when this data is wrong. As long as somebody keeps that promise, it is a tool. The moment nobody keeps it any more, it is a debt paying interest in a currency of its own: wrong data that nobody can reproduce.
How a cache turns into a debt
The road there is always the same, and it is not the result of sloppiness but of pressure.
A page is slow, the deadline is fixed, and the cause sits deep inside a query nobody untangles in two days. A cache in front of it solves the problem in two hours. In that moment it is the right call.
It only turns wrong through what does not happen afterwards: nobody writes down why it is there. No note about the condition under which it could go away again, no measurement of what it actually buys, and no name of anyone responsible. Two years later it is part of the architecture without ever having been designed.
The cost does not arise when something is stored. It arises with every change after that: every new feature has to know which caches it invalidates, and every investigation starts with the question of whether the problem is real at all or merely old.
Three kinds of cache, and only one of them is healthy
Any teardown starts with an inventory, and the caches you find almost always fall into three groups.
The justified one. It sits in front of an expensive calculation, it has a deliberate lifetime, and when it is empty the application gets slower but not wrong. That is the healthy case, and you leave it alone.
The load-bearing one. Without it the system stops. The database cannot take the load that arrives when it is empty. That is no longer a cache, it is a part of the architecture chosen by accident, and it has an unpleasant property: it is empty at exactly the worst moment, namely after a restart under load.
The ghost. Nobody knows what it holds. It was built for a use case that has been solved differently for years, and it is never discarded because the operation that would discard it has been removed. It is the source of the bug reports that "cannot be reproduced here".
The inventory is not an analysis phase, it is an afternoon: find every place that writes, and record three things for each. What is in it, for how long, and who notices when it is wrong.
Discarding: the expensive answer is rarely the right one
The common assumption is that precise invalidation beats an expiry time. That is true in theory and costs the most in practice.
Event-based invalidation means every write in the system has to know which keys it makes stale. That is coupling between places that otherwise have nothing to do with each other, and it gets forgotten again with every new feature. The bugs that come out of it only show up in a particular order of events, which is why they never show up in a test.
The honest question is not "how do we invalidate precisely" but: how old may this data be before the business side objects? For category trees, configuration, overview pages and search results the answer is more often "two minutes" than "immediately", and at that point an expiry time is the complete solution.
Where the answer really is "immediately", the question is usually a different one: then the value does not belong in a cache, the query underneath belongs made fast enough.
For the load-bearing case there is one measure that is often missing and that counts at every restart under load. When a hot key expires, a thousand concurrent requests run the same expensive query, and the database goes down at exactly the moment it is needed. It is enough for one of them to do the work:
// singleflight collapses concurrent requests for the same key:
// one does the work, the others wait for its result.
var group singleflight.Group
func (s *Catalog) Tree(ctx context.Context, id string) (*Tree, error) {
if t, ok := s.cache.Get(id); ok {
return t, nil
}
// Without this line, an expired hot key sends every concurrent
// request down to the database at once.
value, err, _ := group.Do(id, func() (any, error) {
t, err := s.db.LoadTree(ctx, id)
if err != nil {
return nil, err
}
s.cache.Set(id, t, 2*time.Minute)
return t, nil
})
if err != nil {
return nil, err
}
return value.(*Tree), nil
}That is a few lines, and in most cases it replaces the entire discussion about warming the cache.
Where stale data is not a cosmetic flaw
For most values a stale state is annoying. For three kinds of value it is an incident, and those three get looked at first in any inventory: prices, stock levels and permissions.
The most dangerous one is the third, because it disguises itself as an optimization. A permission check is expensive, so the result gets cached. From then on, revoking a right only takes effect once the entry expires:
// This sits in more systems than anyone would like.
// Whoever gets let go still has full access for another hour.
$allowed = $this->cache->get(
"rights_{$userId}_{$action}",
fn () => $this->rights->check($userId, $action),
3600,
);The fix is not to shorten the lifetime. The fix is to cache the right thing: not the decision, but the data the decision is made from, and to make the decision itself on every request.
// The user's roles are cached and discarded on revocation.
// The decision is made fresh every time.
$roles = $this->cache->get(
"roles_{$userId}",
fn () => $this->rights->roles($userId),
300,
);
return $this->rules->allows($roles, $action, $object);The difference costs almost nothing in compute and removes an entire class of incidents. The rule behind it holds generally: decisions do not get cached, data does.
As soon as several customers sit on the same system, a key without a tenant id becomes a data leak: Retrofitting multi-tenancy.
How the debt gets paid back
Paying it back does not mean switching things off. It means making one of three decisions for every entry in the inventory, and the order matters.
The ghosts first. For every cache nobody can explain, measure the hit rate. A cache with a two percent hit rate costs complexity and buys nothing; it gets removed, not documented. That is the cheapest part of the work and often the largest.
Then the load-bearing ones. They do not get removed, they get made visible: a measurement of how the system responds with an empty cache, and a collapse of concurrent requests as shown above. Whoever does not want to take that measurement will get it anyway, just not at a moment of their choosing.
The justified ones last. They get a comment of three sentences: why they exist, how long they are valid, and under which condition they may go away. That last one is the one that is otherwise missing.
The experiment that ends the discussion fastest is switching off one single cache for a slice of the traffic, for an hour, with a measurement before and after. In half the cases the difference is not measurable. That is not an argument for a bold rebuild, it is permission to remove one place where every change used to cost more than it should.
The same exercise for features rather than caches is covered in Removing features nobody uses any more.
The rule that keeps it from growing back
The teardown does not hold if the next tight week produces the next unexplained cache. One rule is enough, and it can be introduced in an afternoon: a new cache needs four statements or it does not get merged. What is in it, for how long, who discards it, and how you would recognize that it has become unnecessary.
Those four statements are not a process, they are four lines in the change request. And the fourth is the important one: it is the difference between a decision somebody can review later and one that turns into architecture because nobody understands it any more.
Anyone who wants to know where the caches in their system sit and what they cost starts with the technical debt they already know about: the place nobody wants to touch is rarely a different one. How I develop systems like these is described on its own page.
This article belongs to a series about systems that already exist. The retrospective orders every article in it by situation.
