Blue/green and canary are treated as topics for systems with container orchestration, and in almost every text about them Kubernetes appears in the first paragraph. That is a misunderstanding with consequences: teams who could use both well consider them out of reach, because they would have to introduce a platform first.
Both techniques are older than containers, and they need exactly two things. Two versions have to be able to run at the same time, and there has to be a switch that distributes traffic between them. A PHP monolith on two virtual machines behind nginx satisfies both.
This article shows what that looks like concretely, and covers the precondition it actually fails on: not the tooling, the database.
The condition nobody states
Before any technique comes one question, and skipping it makes the whole setup a reassurance without effect: can the old and the new version work against the same database at the same time?
With blue/green both run in parallel for minutes. With canary for hours. And when switching back, which is the entire point of the exercise, the old version has to be able to work with the data the new one wrote.
Rename a column and move the code onto it in the same release and there is no way back. The new version writes the new column, the old one reads the old one, and after switching back twenty minutes of data are missing. That is not a flaw in the technique, that is a schema change that does not fit it.
So the condition is: every schema change is backwards compatible, following expand and contract. Add the new column alongside first, write both, then move the readers, and remove the old one only in a later release. That is more work on three days a year and the precondition for a way back on all the others.
How a backwards compatible migration of the data runs is covered in Dual write and backfill.
Blue/green with two targets and one switch
The simplest implementation needs no new infrastructure: two identical environments, a load balancer in front, and a file saying which one is live.
# /etc/nginx/conf.d/active.conf - generated, not maintained.
upstream application {
server 10.0.1.10:8080; # blue
# server 10.0.1.11:8080; # green
}
# The other version stays reachable, but only deliberately:
# acceptance testing happens over this before the switch.
server {
listen 8443 ssl;
server_name new.internal.example.com;
location / { proxy_pass http://10.0.1.11:8080; }
}Switching is then a script, and its most important part is not the two lines that change the file but the check before it and the way back:
#!/usr/bin/env bash
set -euo pipefail
new=$1 # 10.0.1.11
# Ask first whether the new side answers at all. Switching to an
# instance that is still starting up is a self-inflicted outage.
curl -fsS --max-time 3 "http://$new:8080/ready" > /dev/null
cp /etc/nginx/conf.d/active.conf /etc/nginx/conf.d/active.previous
printf 'upstream application { server %s:8080; }\n' "$new" \
> /etc/nginx/conf.d/active.conf
# reload, not restart: existing connections drain, new ones go
# to the new side. No dropped connections.
nginx -t && nginx -s reload
echo "switched to $new. back: mv active.previous active.conf && nginx -s reload"That last line is not a convenience. The way back has to be known and written out at the moment of switching, because it gets needed at the moment when nobody is thinking calmly.
What the new side has to say about itself
One point appears in both techniques and therefore deserves its own section: switching depends on the new version's statement about itself, and in most applications that statement is too optimistic.
An endpoint that simply answers "ok" as soon as the web server is up answers the wrong question. It says the process started, not that the application can work. You then switch to an instance that cannot reach the database yet.
What works is two separate answers. One answers "is the process alive" and must check nothing external, or a brief database hiccup turns into a restart of every instance. The other answers "can this instance take traffic" and checks exactly what it cannot run without:
// GET /ready - only what is needed without exception. Every
// further check here makes switching unreliable without
// lowering the risk.
$checks = [
'database' => fn () => $this->db->fetchOne('SELECT 1') === 1,
'schema' => fn () => $this->migrations->pending() === 0,
'cache' => fn () => $this->cache->reachable(),
];
foreach ($checks as $name => $check) {
if (!$check()) {
return new Response("not ready: $name", 503);
}
}
return new Response('ready', 200);The second line of that list is the one most often missing and the one that prevents the most: an instance with pending schema changes is not ready, even when it answers.
Canary with a share instead of a switch
Blue/green answers "does it work" with yes or no for everybody at once. Canary answers it for one percent first, and the same load balancer is enough for that.
# A stable share: the same session always lands on the same
# side. Without that binding a user bounces between two
# versions.
split_clients "${identity}" $target {
5% new;
* old;
}
map $target $backend {
new 10.0.1.11:8080;
old 10.0.1.10:8080;
}Binding to an identity rather than to the address is where many first attempts go wrong. A user switching between versions sees one interface and then the other, and in a multi-step form they lose what they typed. The session works as the identity, or failing that an attribute of your own set on the first request.
Without an abort rule, canary is just slower
Here is the difference between a technique and a gesture. A share of five percent buys nothing if nobody is watching and nobody decided in advance when to abort.
Three things belong in place before the start, in the same document as the way back. Which metric: the error rate of the new side compared to the old one, not its absolute value. A comparison is robust against the fluctuations of the day, a threshold is not.
Which window: long enough for enough requests to accumulate, short enough that a fault does not act for an hour. At ordinary traffic, ten minutes per step works.
Who aborts: ideally nobody. The abort belongs in the same automation that counts the steps up, because a human looking at a chart at ten in the evening hesitates too long.
The sequence of steps is then a question of magnitude, not of discussion: five percent, twenty-five, fifty, one hundred. And rolling back is a value in the configuration, not a new release.
The forgotten half: everything without a request
Both techniques distribute incoming traffic. Part of a system sees none of it, and that is exactly where the surprises come from.
Background jobs and queues. During a canary the workers process jobs without passing through the switch. The usable rule is to leave them entirely on the old version at first and move them after the switch: a job created by the new version and processed by the old one is the case nobody tested.
Scheduled runs. When both versions are live, they run twice. They belong on exactly one side, and the simplest choice is the old one until the canary is over.
Schema changes. They run once, before the release, and in a way that lets the old version keep working. That is the same condition as above, appearing here a second time, because in practice it gets overlooked twice.
Anyone still deploying over FTP starts one step earlier: When deployment still runs over FTP.
When the platform is worth it after all
Finally the honest boundary, so that this article does not produce the mirror image of the misunderstanding it is meant to dissolve.
Two machines and a switch carry surprisingly far: for one or two applications, a few releases a week and a team that understands the load balancer. From around ten services with rhythms of their own, managing the switches becomes work in itself, and then a platform that brings it along is the cheaper choice.
The order stays the same regardless, and that is the actual point: first the backwards compatibility of the database, then the switch, then the abort rule. Introduce a platform without satisfying the first condition and you have a blue/green that loses data on rollback, just with more tooling around it. How I build delivery pipelines is described on its own page.
This article belongs to a series about systems that already exist. The retrospective orders every article in it by situation.
