Every grown system has a file nobody likes opening: the list of scheduled jobs. Thirty lines, accumulated over ten years, with comments like "do not delete, invoices" and at least one entry whose time was chosen so that it runs after another one.
The model behind it is simple and has carried for a long time: every five minutes, check whether there is anything to do. It stops carrying at a particular point, and that point is easy to recognize.
This article describes when that point is reached, which four problems the cron model has, and how the move to jobs in a queue works without losing anything on the way.
When a cron job is no longer enough
First: for many tasks a schedule is exactly right and stays right. A nightly report, a clean-up of old records, a reconciliation meant to run once a day. Nobody needs a queue for that.
The point is reached when at least two of the following four sentences are true.
"The job runs every five minutes and usually does nothing." That is a queue, just badly implemented: the database is used as a task list, and polled while empty.
"When the job runs longer than its interval, things get unpleasant." The classic overlapping run. Usually patched over with a file lock that stops working on two machines.
"The user waits until the job runs next." An operation that could have started on submit starts up to five minutes later. That is not a technical problem, it is a business one.
"If it fails once, the record is gone." The most expensive of the four. A cron job has no retry; whatever fails during a run has to be picked up by chance on the next one, and often is not.
The four problems in detail
All four share one root: a schedule describes when something should run, not what is to be done. The job does not exist anywhere as an object of its own.
No state per task. There is no place recording that this order still has to be handed to the ERP, that three attempts have failed, and that the next one is at 14:20. Instead there is an exported column, and it knows two states where five are needed.
No retry with backoff. When a third-party system briefly stops answering, the right answer is a second attempt in a minute. A cron job cannot express that; it can only try everything again on the next run, or nothing.
No concurrency you can steer. Either one run, which is slow. Or several, which may work on the same thing. The middle ground, eight at once and never the same one twice, is not provided for in the cron model.
No visibility. How many tasks are waiting right now? How long does one take on average? Which ones keep failing? Those questions cannot be answered in the cron model, because the task does not exist as an object.
First step: turn the job into a task
The move does not start with a queue, it starts with a table. That matters, because the table prepares everything else and needs no new infrastructure.
CREATE TABLE job (
id BIGINT AUTO_INCREMENT PRIMARY KEY,
kind VARCHAR(64) NOT NULL, -- 'order.export'
ref VARCHAR(128) NOT NULL, -- business key, e.g. order number
payload JSON NOT NULL,
state ENUM('open','running','done','failed') NOT NULL DEFAULT 'open',
attempts SMALLINT NOT NULL DEFAULT 0,
run_after DATETIME NOT NULL,
last_error TEXT NULL,
created_at DATETIME NOT NULL,
-- Prevents the same job twice. The most important line in the table.
UNIQUE KEY unique_job (kind, ref),
KEY pick (state, run_after)
) ENGINE=InnoDB;The unique key is the core. It makes enqueueing repeatable: enqueue the same job twice and you get one, not two. That rules out the most dangerous failure in this whole topic before it can occur.
From here the role of the cron job changes: it no longer determines what is to be done, it works through what is in the table. That is already half the move, and it needs not a single new component.
Repeatability is the entry ticket
As soon as retries exist, one condition applies without exception: every job has to be executable twice without causing damage. That is idempotency, and it is not a nicety, it is the precondition.
The reason is uncomfortable: there is no guarantee that a job runs exactly once. A worker can die mid-processing, after sending the invoice and before saving the state. The next attempt starts from the beginning.
In practice one ordering carries most jobs.
- First check whether the effect already happened. Not whether the job ran, but whether the result exists. If the invoice is already in the target system, the job is done.
- Give third-party systems an idempotency key. Almost all payment services and APIs support a key for retries. That makes the question from the first point their problem, and it is better handled there.
- Side effects last. If a job calculates and then sends an email, the send goes at the end. Then an abort in the middle costs at most a recalculated invoice, not a second email to the customer.
Picking jobs without two workers taking the same one
When several workers run, they must not grab the same job. In MySQL that has been one query since 8.0:
-- SKIP LOCKED: locked rows are skipped rather than waited for.
-- Without it, eight workers queue up behind each other.
START TRANSACTION;
SELECT id, kind, payload
FROM job
WHERE state = 'open' AND run_after <= NOW()
ORDER BY run_after
LIMIT 10
FOR UPDATE SKIP LOCKED;
UPDATE job SET state = 'running', attempts = attempts + 1
WHERE id IN (...);
COMMIT;For a whole range of systems that is the final destination, and that is fine. A job table with this query carries tens of thousands of jobs a day without operating an additional service. A dedicated message queue pays off when the load is well above that, when several systems produce the same jobs, or when the database is the bottleneck anyway.
Failures, backoff, and the jobs that never succeed
Now comes the part the cron model was missing. A failed job does not get forgotten, it gets an appointment.
private const MAX_ATTEMPTS = 8;
public function failed(Job $job, Throwable $e): void
{
if ($job->attempts() >= self::MAX_ATTEMPTS) {
// Do not delete, do not retry forever: move it out of the way and
// make it visible. This is the shelf for the undeliverable.
$this->db->update($job->id(), [
'state' => 'failed',
'last_error' => $e->getMessage(),
]);
$this->alerts->jobAbandoned($job, $e);
return;
}
// Growing backoff with jitter. Without the jitter, every retry comes
// back at the same moment after a third-party outage and takes it down
// a second time.
$seconds = min(2 ** $job->attempts(), 3600);
$seconds += random_int(0, (int) ($seconds * 0.2));
$this->db->update($job->id(), [
'state' => 'open',
'run_after' => new DateTimeImmutable("+${seconds} seconds"),
'last_error' => $e->getMessage(),
]);
}Two things in there come from experience. The jitter stops a recovered third-party system from being hit by every retry at once. And the terminal state failed is not a deletion: those jobs stay, they are queryable, and after a fix they can be set back to open with one command.
That list of permanently failed jobs is incidentally the most useful view of the whole system. It replaces the feeling that "sometimes something does not arrive" with a list of identifiers.
A queue is also the tool that makes a synchronous dependency between two services disappear: The distributed monolith.
What the schedule keeps
A queue does not replace every cron job, and trying ends in a complicated imitation. Two things stay with the schedule.
Creating jobs at fixed times. The monthly invoice run still starts on a schedule. It just does something different: instead of producing three thousand invoices it enqueues three thousand jobs and is finished in two seconds. The difference is considerable, because that one run can no longer fail.
Clean-up. Delete finished jobs, report failed ones, reset stuck ones. The last point matters: a worker that dies leaves jobs in state running, and somebody has to release those after a while.
-- Release stuck jobs, once a minute
UPDATE job
SET state = 'open', run_after = NOW()
WHERE state = 'running' AND created_at < NOW() - INTERVAL 15 MINUTE;The order of the migration
Thirty cron jobs do not get moved in one step. The order that has proved itself goes by pain rather than by effort.
- Create the job table and move one single job. The one where lost work hurts most, usually a handover to a third-party system. After that one job, retries and visibility exist, and the rest becomes a repetition of the same work.
- Move the jobs that poll every five minutes. The return is largest here, because a query on an empty table becomes a job created at the moment the work appears.
- Break up the long runs. A two-hour run becomes a producer plus many small jobs. That ends the overlapping-run problem by itself.
- Leave the rest alone. A nightly report that has run for six years stays a cron job. Moving it costs time and delivers nothing.
The fourth point belongs there explicitly. The goal is not to empty the list of scheduled jobs, it is to make the operations reliable that are not.
For a job to sit in the same trace as the request that created it, the context has to travel in the payload: Introducing OpenTelemetry in PHP and Go.
What you have to watch from now on
The move creates a new operational task, and it needs three measurements.
The age of the oldest open job. The most informative single number. If it rises, the workers are not keeping up, regardless of how many jobs there are at that moment.
The number of permanently failed jobs. A value above zero needs a person. Not immediately, but that day.
Duration per job kind. Split by kind, because an average across kinds says nothing when one takes forty seconds and another forty milliseconds.
Those three belong wherever the team already looks. What stands out there is usually not a fault in the job system but one in a third-party system, and that is the gain: "sometimes something does not arrive" becomes a number with a name next to it.
How I set up background processing in existing systems is on its own page. In more projects than not, the job table above has been the right answer rather than a dedicated queue.
This article belongs to a series about systems that already exist. The retrospective orders every article in it by situation.
