Glossary
Observability
The property of a system to reveal from the outside what is happening inside, including for questions nobody asked in advance.
The difference to classic monitoring is the kind of question. Monitoring answers prepared questions, such as whether response time exceeds a threshold. Observability allows new questions afterwards, such as why customers from one region have been waiting longer since yesterday.
It rests on three kinds of data: metrics, logs and tracing across service boundaries. What matters is that they can be joined through a shared identifier.
That identifier is the actual core. A transaction is given a number when it enters the system, which is then passed through every call, every queue and every background process and written into every log line. Without it you have three collections of data sitting side by side, and during an outage three people searching in three different tools.
With metrics, the average is the most common trap. A mean response time of 200 milliseconds can mean that everyone waits 200 milliseconds, or that ninety per cent are served in 50 milliseconds and ten per cent in two seconds. Only the second case produces complaints, and only percentiles show it. Measure on the 95th and 99th percentile, not on the mean.
The second widespread mistake is measuring only technology. Processor load and memory usage say nothing about whether orders are getting through. One metric per business transaction, meaning completed orders per minute or successful payments, often detects an outage minutes before any technical threshold does, because it measures the thing that matters.
Logs belong in structured form, as a record with fields rather than as running text. The difference becomes visible during the first incident: filtering for a particular customer in structured fields takes seconds, in free text you search with patterns and find half of it.
The cost is real and gets underestimated. Complete tracing at high traffic produces data volumes that quickly become more expensive than the systems being observed. The usual way out is sampling: keep every error and every slow transaction, and a small share of the rest. That is enough for diagnosis, because the interesting cases are precisely the outliers.
How you notice it
- During the last outage it was unclear which service caused it.
- Logs live in several places and cannot be joined.
- There are technical metrics but none for the business transaction.
- The on-call rota dismisses alerts because most of them mean nothing.
- Response times are reported as an average.
Not to be confused with
- Monitoring
- Answers prepared questions with thresholds and alerts. Necessary and not sufficient: it tells you that something is wrong, not why.
- APM
- A product category, usually tracing plus metrics from the application point of view. A tool that supports observability, but no substitute for the question of which questions should be answerable.
- SLO
- A reliability target. Observability supplies the data with which you measure it and explain the breaches.
When it fits
- Several services are involved in one transaction.
- During the last outage it was unclear where the cause sat.
- There are reliability commitments that have to be evidenced.
- Errors only occur for certain customers or data constellations.
When it does not
- For a single small application with little traffic: logs and simple metrics are enough there.
- As a tool purchase without asking which questions should be answered.
- Complete tracing at high traffic without sampling: that gets more expensive than the system being observed.
How to approach it
- Start with the questions from the last outageWhich three questions did you want to answer and could not. That determines what has to be collected, and nothing beyond it.
- Introduce an end to end correlation IDAssigned on entry, passed through calls, queues and background processes, written into every log line. Everything else hangs off it.
- Structure the logsFields instead of running text, with customer, transaction, duration and outcome. The difference shows on the first filter under time pressure.
- Percentiles rather than averagesThe 95th and 99th percentile show the cases somebody complains about. The average hides them reliably.
- One business metric per processOrders per minute, successful payments, documents processed. It detects outages earlier than any technical threshold.
- Only alert on what somebody would touch at nightAlerts without an action train people to dismiss them. Everything else belongs in a view that gets looked at in the morning.
- Decide on sampling and retentionErrors and slow transactions in full, a share of the rest. Two weeks in fast access, anything older in cheap storage.
Frequently asked
What do I need first, metrics or tracing?
A shared identifier across all services. Without it, logs and metrics are two collections nobody can join. With it a single transaction can be followed through the system, and that answers most questions during an outage.
How do you keep the cost under control?
Through three levers: sampling on the tracing, short retention in fast access with offloading to cheap storage, and restraint with cardinality. A metric that carries the customer ID as a label produces ten thousand time series for ten thousand customers, and that is exactly what drives the bill.
Are logs not enough?
For a single application, often yes. As soon as several services take part in one transaction, logs no longer answer the question of where the time went, because each service only knows its own slice. That is what tracing is for.
How do you tell that it works?
By the time to cause during the next incident. If after the build-out three people are still searching in three tools and diagnosis takes an hour, data was collected but not joined. That is the usual state after a tool purchase made without asking the question first.