Composer as archaeology

Taking over an old PHP project has a moment that decides the next two weeks. It happens when somebody types composer install and finds there is no composer.lock. Or that there is one, but it does not match what is in vendor/. Or that vendor/ is committed to the repository and nobody can say when it was last updated.

That is not an edge case. In projects started before 2016 it is closer to the norm, and it has a simple reason: Composer and the lock file only became standard once those projects were already running.

This article describes how to reconstruct the dependency tree of such a project before raising it. The order matters: an upgrade on a foundation nobody knows is not a migration, it is an attempt.

Why this comes before the upgrade

The temptation to skip the foundation is strong. The destination is known: current PHP version, current packages. So why do archaeology first?

Because without this step, two questions stay unanswered that poison every later investigation.

Is production running what is in the repository? Until that is settled, every test on a development machine is a statement about a different system. I have seen projects where a package in production was three minor versions ahead of the repository, because somebody had updated it directly on the server years earlier.

Has anything been changed by hand? A patched package in vendor/ survives no composer update. If the patch fixes a bug nobody else noticed, that bug comes back with the upgrade, and nobody connects it to the upgrade.

Both questions can be answered in one or two days. Skipping both costs a multiple of that later.

Step 1: what is actually installed?

The most reliable source is not composer.json but what sits in production. Since Composer 2 there is a file there that describes the state completely:

# On the production server, not locally
php -r 'foreach (require "vendor/composer/installed.php" as $k => $v) {
    if ($k !== "versions") continue;
    foreach ($v as $package => $info)
        echo $package, " ", $info["pretty_version"] ?? "?", PHP_EOL;
}' | sort

If that file does not exist because it is still Composer 1, the older installed.json gives the same:

php -r '$d = json_decode(file_get_contents("vendor/composer/installed.json"), true);
foreach ($d["packages"] ?? $d as $p) echo $p["name"], " ", $p["version"], PHP_EOL;' | sort

That list is the truth. Everything else is checked against it, and the first comparison is with the repository:

diff <(ssh server 'cd /var/www/app && php -r "..."') \
     <(php -r "..." )

Every differing line is a finding of its own and belongs written down before anything is changed.

Step 2: which packages were changed by hand?

This is the finding that saves the most time, because otherwise it only surfaces after the upgrade, as an inexplicable bug.

Composer records the source and version of every installed package. That makes it possible to compare each one against its original state. It is an afternoon of manual work and it can be pushed together:

mkdir -p /tmp/original
while read -r package version; do
  target="/tmp/original/${package//\//_}"
  composer create-project --no-install --no-scripts --quiet \
    "$package:$version" "$target" 2>/dev/null || continue
  if ! diff -rq "$target" "vendor/$package" >/dev/null 2>&1; then
    echo "CHANGED: $package $version"
  fi
done < installed-packages.txt

In practice this does not work for every package, because some exclude files when publishing. For the packages where it does work, it reliably finds every change, and those are usually the interesting ones: the small libraries where somebody once had to adjust something quickly.

Every finding then needs a decision, and there are only three possible ones.

• The patch is already in the current package. More common than expected: somebody fixed a bug that the maintainer later fixed as well. Then the patch simply falls away with the upgrade.
• The patch is needed and never reached the maintainer. Then it belongs in a patch file applied at install time, not in a directory that every update overwrites.
• Nobody knows what the patch was for. The most common case. Then it gets removed and the place gets a test. If the patch was necessary, that shows now, under controlled conditions, rather than in three months in production.

Step 3: producing the lock file

With the list from step 1 you can generate a lock file that pins exactly today's state. The reflex to take current versions here is wrong: that would be the upgrade, and the upgrade is the next step, not this one.

# Pin every version exactly as it runs today
while read -r package version; do
  composer require --no-update "$package:$version"
done < installed-packages.txt

composer update --lock   # write the lock file only, raise nothing

Then comes the check without which the whole step is worthless: a fresh install from the new lock file, compared against what sits in production.

rm -rf vendor && composer install
diff -rq vendor/ /tmp/production-vendor/ | grep -v "\.git"

What still differs here is either the patched packages from step 2 or packages that no longer exist in that version. The second case is the awkward one, and it leads to the next section.

From this point the project is reproducible, and that is a milestone in itself. It belongs in a commit of its own with a message saying that nothing was raised here.

The cases that cannot be resolved

Three situations regularly turn up where a package is no longer installable. All three have an answer, and all three need a decision rather than a trick.

The version has been withdrawn. Rare, but it happens. If the nearest remaining version works, take it and review the difference. If not, the next case applies.

The package no longer exists. The maintainer removed it, the repository is gone. The only thing that helps is to take what is in production and put it into a private repository as a package of its own. It is not a pretty solution. It is more honest than a directory that only exists on one server.

It is an internal package with no repository. The code sits in vendor/, was delivered by a supplier and has never been in version control. Then it becomes your code: into a repository of its own, with a version, and with a note about where it came from.

In all three cases the outcome is the same: what used to sit silently on a server now sits in a named place with an owner. That turns the rest of the upgrade into an ordinary project.

An abandoned package is rarely only a maintenance problem: Ten holes that sit in every old PHP system.

Step 4: abandoned packages, before they hurt

Now, and only now, it is worth looking forward. With a working lock file, Composer itself answers the question of what will be expensive in the upgrade:

composer audit                 # known vulnerabilities
composer outdated --direct     # distance to the maintained version
composer why-not php 8.4       # who blocks the target PHP version?

The third command is the most useful and the least known. It answers in one line which packages prevent the jump, and turns a feeling into a list.

That list sorts into the same three categories as in every upgrade: outdated but maintained, abandoned with a successor, abandoned without one. The third category decides how long the whole project runs, and it is now known before anybody has given an estimate.

So that the question about the state of the packages does not get asked again every two years, it belongs in a calendar: An end-of-life calendar for the whole stack.

So that it does not happen again

Finally, the three things that stop somebody doing this same work again in five years. All three are small, and all three were missing in every project where I had to do this archaeology.

The lock file belongs in the repository, the vendor/ directory does not. The opposite was once common advice, and its cost shows up here.

Installation runs in the build, not on the target server. As long as somebody can install on the server, somebody eventually will, and the states drift apart again.

A job regularly checks whether production and repository agree. A checksum comparison, once a week, with an alert on any difference. That is half an hour of work and it ends a whole class of surprises.

How the jump itself works, which breaking changes really hurt and which behaviour changes happen quietly, is covered in its own article: PHP 7.4 to PHP 8.4. And how I set up a PHP upgrade overall is on its own page.

This article belongs to a series about systems that already exist. The retrospective orders every article in it by situation.