Observability
Two starting points lead to this page. In one there is nothing to see: an outage is reported by a customer, not by the system. In the other there is too much, with dashboards nobody opens, alerts nobody reads any more, and a bill for watching that nobody can explain. Both share the same origin: measuring started before anyone decided which question had to be answered.
Inventory and target picture as a bounded two to four week engagement, implementation afterwards by scope. Operations stay with your team, which is why the handover is part of the work.
Remote from Germany. Straight with me, no agency in between.
The starting point
Logs, metrics and traces are means, not ends. Before any of them is set up, four questions deserve an answer: is it broken, for whom, since when, and what is causing it. A system that answers those four within minutes needs surprisingly few signals. A system that does not answer them does not improve with the two hundredth dashboard. It only gets more expensive.
The second mistake is a pricing question dressed as a technical one: cardinality. One metric per host is cheap. The same metric per host, container, customer and endpoint is the same tool and a multiple of the bill, because every combination is a time series of its own. Log volume and retention come on top. Vendors of alternatives quote growth rates of thirty to fifty percent a year, and that number has a sender, but the mechanism is real and can be recalculated on any invoice. In one of my AWS projects it was exactly one line of the savings: unused, high-cardinality CloudWatch metrics removed and retention brought down to what was actually needed.
The third mistake happens after go-live. An alert without an owner is not an alert but a notification, and a team learns to click notifications away. After four weeks of false alerts, being on call is a habit of ignoring, and the first real incident goes down in the same motion. That is why every alert needs a threshold that comes from an SLO, a person who owns it, and a runbook that says what to do.
Does any of this sound familiar?What I do
Which business paths must not fail: ordering, payment, sign-in, approval. Those get SLIs that measure from the customer side and SLOs that name a number instead of a feeling. Everything else is judged by whether it answers one of those questions. What answers none does not get built.
Instrumentation runs through OpenTelemetry, not through a vendor library. That is not ideology but a negotiating position: anyone producing signals in a proprietary format cannot switch vendors without touching half the application. With OTel the boundary sits in the collector, and behind it Datadog, Grafana or CloudWatch is a configuration.
This is where the bill is created, so this is where it gets decided rather than disputed later. Per signal: which dimensions are really needed, at what resolution, for how long. High resolution for a few days, coarse aggregates for long ranges, and sampled logs rather than complete ones wherever completeness answers nothing.
An alert comes out of an SLO, not out of a threshold that looked plausible. Per alert: who receives it, what the first action is, when it escalates. Alerts without a runbook get switched off. They cost the attention that will be missing during the next real incident.
An observability setup proves itself in an incident, not in a dashboard. So the incident gets rehearsed: slow down a dependency, take an instance away, let a queue fill up, then check whether the four questions get answered and whether the system degrades predictably instead of falling over.
To close, the things that make the result durable: dashboards and alerts as code in the repository, a runbook per alert, a note on who reviews which SLO and when, and a handover session with your team. Operations stay with you. That is not a cost saving but the goal.
Do you know right now whether it works for your customers?
Write to me about what you can see today and what you missed during the last incident. You get an assessment before you commission anything.
How it runs
Four steps, and after the second one the target picture and its cost range are on the table. No step assumes that you commission the next one.
What is in use, what it costs, what was missing during the last incident, plus the business paths that must not fail. No access is needed for this yet: a conversation and a look at the invoice will do.
Which signals, which SLOs, which alerts, and what that costs per month, broken down by cardinality, volume and retention. Enough for you to decide, including against me.
One business path after another: instrument, set the SLO, add the alert with its runbook, then the next one. Each cut is useful on its own and reversible on its own.
A rehearsed incident that shows whether the four questions get answered. Then the handover: configuration in the repository, runbooks, ownership.
The outcome
Not the customer, not social media, not the sales team. That is the whole difference between an incident and an incident with an audience.
When every alert has an owner and a first action, the number drops and the on-call rotation starts taking them seriously again.
Cardinality, volume and retention are decided rather than grown. That makes the vendor’s next price increase an arithmetic exercise instead of a surprise.
With OpenTelemetry as the boundary, a switch is a configuration rather than a project. That pays off even if you never switch.
Technologies I use
From practice
Code grown over years without tests, without documented infrastructure, with regular incidents. A test net, the migration and visibility came together; afterwards incidents were 90 percent lower. Visibility was not the only cause, but it was the precondition: nobody can fix what nobody can see.
Read the case studyAlso from real projects
Grafana, Prometheus, CloudWatch and Sentry set up so that problems become visible before they turn into outages. With payments that is not a convenience: every outage is revenue, immediately.
Read the case studyUnused, high-cardinality CloudWatch metrics removed, Athena data partitioned, retention brought down to what was actually needed. One item among several that cut the AWS bill by 83 percent.
Read the case study
Who you are talking to
I am Tim Rutte. More than 20 years in software development, and most of the outages I have seen were visible beforehand, just not to the person who needed to see them. You talk to the person who touches your code, from the first call to the handover.
Common questions
Possibly, and the answer does not belong at the start. Datadog is strong when many signal types should come together in one place and the team does not want to run a platform. It is expensive when cardinality and retention are undecided, because then you pay for data that answers no question. CloudWatch is enough more often than vendors recommend, and a Grafana stack is the right call if you have operational capability and want to keep it. I instrument through OpenTelemetry so that this decision stays reversible.
Yes, and it is usually the same work. A bill that is too high is almost always a cardinality problem or a retention problem, and both get solved by deciding which question each signal answers. The AWS cost page is the neighbour to this one: there the AWS invoice is the subject, here it is the invoice for watching.
No, and that is a decision rather than a capacity question. I build it, rehearse it with you and hand it over. An observability setup operated by an outsider puts exactly the knowledge outside that will be needed inside during an incident. If you need permanent on-call coverage, you need a team or a provider with shift operations, and I am neither.
Billed by time spent, with the inventory and target picture as a bounded two to four week engagement. I name the order of magnitude in the first conversation, as soon as it is clear how many business paths there are and what is in use today. Unlike the upgrade pages there is deliberately no fixed-price package here: the starting points differ too much to be pressed into one delivery, and there is different work between "we see nothing" and "we see too much and pay for it".
No. Existing dashboards are a treasure trove: they show what somebody once considered important, and the open questions show up in them. What holds, holds. Only what answers none of the four questions and still costs money or attention gets switched off.
Fewer than you think. I would start with two to four: the paths where money is immediately at stake. Setting one SLO per service produces a table nobody reads and dilutes the message, because if everything has a target, nothing has priority.
Yes, but with a different scope. Without on-call an alert at three in the morning is worthless, so instead the work goes into making the system degrade predictably rather than fall over, and into making it recognizable within minutes in the morning what happened during the night. Graceful degradation and a good incident log do not replace on-call, but they capture the larger part of the benefit.
There especially. A language model rarely fails outright, it answers worse, and classic signals cannot see that. On top of it comes cost per request as a metric of its own. The route is the same: decide which question has to be answered, then instrument. The cost side of it has a page of its own.
Other services
Backends for SaaS and platforms that hold under real load: Go and PHP 8, event-driven, with tenant isolation and recovery designed in.
Learn moreSo that a single fault does not take the whole application with it: dependencies and single points of failure identified, recovery rehearsed rather than documented, changes delivered as Terraform.
Learn moreFrom a pipeline nobody wants to touch to deployments on a Tuesday morning: one artefact through every stage, access without long-lived keys, a rehearsed way back.
Learn more