A grown system with PHP at its core and a few Go services alongside usually has both: too many signals and too little answer. There are log files, there is a vendor console, there are metrics from three sources, and yet answering "why was this slow at two o'clock yesterday" takes half a day.
That is rarely about the volume of data. It is because the data does not connect: the log line from PHP and the metric from the Go service describe the same request and know nothing about each other.
OpenTelemetry solves exactly that, and it solves it in the right place. This article describes an introduction that pays off in weeks rather than quarters, including the places where a PHP and Go stack actually gets awkward.
Why OpenTelemetry and not the vendor
Every vendor ships its own libraries, and they are more convenient than the standard. The price does not appear on the invoice, it appears in the code: instrumentation is the expensive and least reversible half of observability, and it does not belong to the vendor.
With OpenTelemetry it sits once in your own code, and the decision about the destination becomes a matter of configuration. Moving from one vendor to another, or to a stack of your own, is then not a rebuild but an endpoint and a restart of the collector.
That is not a theoretical argument. The most common reason to reconsider observability is an invoice that has tripled over a year. Finding out at that point that moving means six months of instrumentation means you do not move, you negotiate from a weak position.
When nothing at all is measured yet, three numbers are enough to start: Three measurements before you refactor a single line.
Traces first, and one path only
The common order is metrics, then logs, then traces at some point. It is backwards when the open question is "where does the time go".
Metrics tell you that something is slow. You already know that, or you would not be looking. A trace tells you where, for a single request, across every service involved. That is the answer that otherwise costs half a day.
Just as important is the shape of the first step: one path, not the whole system. Checkout, sign-in, search. One path runs from the entry point into the database and through every service involved, and it is done in a week. Blanket instrumentation is half done after a quarter, and the missing half is always the one you need right now.
PHP: the quirk nobody mentions
PHP has automatic instrumentation through the extension, and it covers Symfony, Laravel, Doctrine, PDO and the usual clients. So getting started is short:
pecl install opentelemetry
composer require \
open-telemetry/sdk \
open-telemetry/exporter-otlp \
open-telemetry/opentelemetry-auto-symfonyThe quirk is not in the setup, it is in the export. PHP ends the process when the request ends; there is no background process shipping spans in batches. Exporting straight to the vendor therefore attaches every request to a network connection you do not control.
The answer is a collector on the same host, addressed over a Unix socket or over localhost. Sending then takes microseconds, and batching, retries and sampling happen outside the request:
# Not to the vendor, to the collector next door.
# Everything else is its job, not the PHP process's.
OTEL_SERVICE_NAME=shop-web
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
OTEL_TRACES_SAMPLER=parentbased_always_on
OTEL_PHP_AUTOLOAD_ENABLED=trueThe second point is about cost: in a grown system the automatic instrumentation quickly produces thirty spans per request, and most of them are database calls that all look alike. That is not only expensive, it is unreadable. The queries inside a loop belong collapsed into one span, and that is manual work in exactly one place.
Go: the context is all of the work
Go has no automatic instrumentation, and that matters less than it sounds. The middleware for HTTP and the wrappers for the database and clients are there; what is left is passing the context along.
That is exactly where most first attempts fall over. A trace always tears in the same place: where somebody did not pass the context on.
func (s *Service) PlaceOrder(ctx context.Context, o Order) error {
ctx, span := s.tracer.Start(ctx, "order.create")
defer span.End()
// Business attributes yes, identifiers of individual users no:
// every distinct value costs money in the backend.
span.SetAttributes(
attribute.String("order.channel", o.Channel),
attribute.Int("order.items", len(o.Items)),
)
if err := s.repo.Save(ctx, o); err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, "save failed")
return err
}
// This is where the trace tears in almost every first attempt:
// a goroutine without context is a second, rootless tree.
go s.notify(context.WithoutCancel(ctx), o)
return nil
}context.WithoutCancel is where two correct intentions meet: the goroutine should keep running when the request ends, and it should stay in the same trace. Without that distinction you get either cancelled work or a torn trace.
The seam between PHP and Go
The real payoff only arrives once both sides sit in the same tree. Over HTTP that happens by itself, because both libraries set and read the same header. Over a queue it does not, and that is where most introductions stop halfway.
A job that PHP enqueues and a Go service processes does not carry the context with it. It has to be handed the context as a field:
// On enqueue: write the current context into the payload.
// The format is the W3C standard, both sides understand it.
$carrier = [];
TraceContextPropagator::getInstance()->inject($carrier);
$this->queue->push('invoice.create', [
'order_id' => $order->id,
'trace' => $carrier,
]);On the Go side the same carrier is extracted again before the span starts. After that, invoice creation hangs in the order's trace, and the question "why did the customer get their invoice two hours later" is a view rather than an investigation.
One hint that saves a week: the trace of a queue deliberately looks different from the trace of a request. Processing starts later and takes longer than the request that triggered it. Read that as a bug and you will build the wrong relationship.
How jobs get into a queue in the first place, and what to watch for, is covered in From cron jobs to queues.
The log files that are already there
The part many introductions skip because it looks like tidying up: a legacy system already has log files, and they often contain exactly the line you need. It is just not findable, because nothing connects it to the request.
The log pipeline does not have to be replaced for that. It is enough to write the trace id into every line, and that is one processor in the logger, not a change at the call sites:
// A Monolog processor, registered once. After that every line
// written anywhere in the system carries its trace id.
$logger->pushProcessor(function (LogRecord $record): LogRecord {
$span = Span::getCurrent()->getContext();
if ($span->isValid()) {
$record->extra['trace_id'] = $span->getTraceId();
$record->extra['span_id'] = $span->getSpanId();
}
return $record;
});The payoff is out of proportion to the effort. "Go search yesterday's logs" turns into a filter on an id that sits right there in the trace, across every service, without a single log line being rewritten.
Cardinality, sampling, invoice
Observability does not get expensive through the number of requests. It gets expensive through two decisions taken early and noticed late.
Attributes with many distinct values. A user id, an order number or a session id as an attribute on a metric creates one time series per value. That is the usual way to turn a two hundred euro invoice into a four thousand euro one. On a span such values are fine, on a metric never.
Keeping everything. Sampling at one percent is cheap and useless when the failure case sits at one in a thousand. The answer to that is not a higher percentage but a decision taken after the trace has finished, in the collector:
# Decided once the trace is complete: everything notable in full,
# one tenth of the rest.
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: slow
type: latency
latency: { threshold_ms: 1000 }
- name: rest
type: probabilistic
probabilistic: { sampling_percentage: 10 }Those fifteen lines are regularly the difference between observability that gets cut in the next cost review and observability that stays.
How you know it is enough
An introduction like this is not finished when everything is instrumented. It is finished when one particular question loses its answering time.
The test is concrete: somebody names a time and an order number, and it takes less than five minutes to say where the time went and which service took it. Once that is true, the next path gets instrumented, and it is the one with the next most frequent questions, not the technically most interesting one.
What does not come out of this is a project with an end date. Observability grows with the system and shrinks with it, and the only rule that keeps it alive is the same one as with caches: what nobody looks at gets switched off, not maintained. The term itself is in the glossary, the approach as a service on its own page.
This article belongs to a series about systems that already exist. The retrospective orders every article in it by situation.
