Service · Software development

Setting up observability and monitoring.

Software made up of a handful of services can no longer be debugged by looking through the logs. We set up metrics, log lines and traces so you can follow a request from start to finish, and we base alerting on what your users notice rather than on whatever a machine happens to report.

OpenTelemetryTracingAlertingError budget

What we do and don't do here.

In most organisations the problem isn't too little measuring, but too much, and too loosely. Three dashboards that contradict each other, two alerting channels, one of which has been muted, and a log file so large that nobody searches it any more. The question that matters during an outage – where in the chain is this going wrong – still can't be answered without pulling in three people.

Good products exist for the tooling itself: Grafana with Prometheus, Loki and Tempo, Datadog, Elastic, New Relic, Honeycomb, Sentry, and the built-in services of Azure, AWS and GCP. If you already run something like that and it fits, we'll advise you to keep it. We rarely replace a platform with a standalone solution; adding a second vendor usually fragments the picture even further.

What we do is the layer in between: instrumentation in your software so there is something to measure, a trace that keeps running across system boundaries, and alerting tied to a service rather than a server. We often do this as part of a modernisation project, alongside setting up DevSecOps in the pipeline, or on its own when incidents are piling up and nobody can pinpoint why.

We do this as a standalone assignment and as part of a broader custom software development project. If it coincides with a move to separate services, also look at microservices architecture and optimising your IT operations. How we handle your data is set out in our information security policy.

Three kinds of engagements we get in this area.

Most engagements start with one of these three and grow once the first picture is in place. In the first conversation we'll tell you which option delivers the most for your landscape, and which one you're better off postponing.

First engagement · fixed sprint budget

Setting up the four signals per service

Latency, error rate, traffic and saturation – per service, with the distribution included and not just the average. An average of two hundred milliseconds can mean that everyone waits two hundred, or that one in a hundred waits eight seconds; the latter is where your complaints come from. We build that measurement into the software, bring it together on a dashboard that is about your service delivery, and remove the alerts that are about isolated technical values.

Four golden signalsPercentilesDashboard per serviceAlert clean-up
Mid-sized project · fixed sprint budget

Tracing across system boundaries

A request that passes through four services gets a trace on arrival that goes with it everywhere. In your own code, in the queues between services and in calls to external parties. We build this with OpenTelemetry, so you aren't locked in to a vendor, and make sure log lines and metrics carry the same trace. From that point on, the question "where did this request get stuck" is a click instead of an afternoon.

OpenTelemetryTraces across queuesLinked log linesExternal calls
Larger project · fixed sprint budget

Error budgets and service level agreements

A target per service – how many requests may fail or take too long before it becomes a problem – and a budget that follows from it. This turns "it's slow" into a number you can have a conversation about, and gives your team a fair boundary between building new features and fixing stability. Including reporting to your client or, if you are the one delivering, to your customers.

Targets per serviceError budgetReportingEscalation path

What you have at the end of an engagement.

A working setup that runs on your own platform, with the knowledge in your team to extend it without calling us in.

  • Instrumentation in your softwareMetrics, log lines and traces from the code itself, via OpenTelemetry, so you can switch platforms without rebuilding everything.
  • Dashboards per serviceAn overview per service with the four signals and the distribution, plus a start page that shows at a glance whether something is wrong.
  • Alerting that makes senseAlerts based on what your users notice, with a clear recipient, an escalation path and a short description of what the recipient should do first.
  • A retention policy with the costs includedFor each type of data, we document how long it is kept and why, including trace sampling with the agreement that everything that fails is retained in full.
  • Runbooks for the most common incidentsA short page for each recurring pattern: how you recognise it, where you look first, and which intervention has resolved it before.
  • Handover to your teamTwo working sessions in which your developers instrument a service and set up an alert themselves, so that extending the setup doesn't stay with us.

When this is worth it.

Four situations we keep seeing at organisations that come to us for this. If you recognise one, there is probably something to gain.

Searching takes too long

Every incident is a search

Something breaks and three people are looking in three systems, without knowing whether they're talking about the same request. Recovery time isn't determined by the fix but by the finding. That is exactly what a continuous trace removes.

Alert fatigue

Nobody looks at the notifications any more

So many alerts come in that the channel has been muted. When something real happens, it no longer stands out from the noise. The problem is almost never the volume but the nature of the alerts: they fire on machines rather than on the service you provide.

Improvement can't be demonstrated

Every discussion comes down to gut feeling

You want to show that investing in stability has paid off, but there is no measurement from before. Without numbers, every improvement is just an opinion, and an opinion doesn't hold up in a conversation about where the budget goes.

Multiple services

It's no longer an application but a landscape

What started as one application has grown into a handful of services with queues and external integrations in between. The tooling that worked for a single application now gives you a view per component rather than of the whole.

How a project works.

1

Introduction and intake

A conversation about what is running now, what is already being measured and where the last few incidents came from. We ask about the systems in between – queues, external integrations, scheduled jobs – because that is usually where the blind spots are. We also map out which integrations will become part of the trace.

2

Baseline and scope

We measure what exists today: which services already produce signals, which don't, and what the current storage costs. We also review the most recent incidents and ask, for each one, what someone would have wanted to see. At the end you have a scope that starts with the service where the most time is lost, not the one that is easiest to instrument.

3

Instrumenting in sprints

We work in sprints and deliver one fully connected service per sprint: metrics, log lines with trace context, and traces that continue through to whatever sits behind them. Your own developers work alongside us, because instrumentation added by someone else is rarely maintained.

4

Alerting and runbooks

Only once enough is being measured do we set up alerting. For each alert we record who receives it, what the first action is and when it escalates. At the same time we clean up the existing alerts; in practice that often brings more calm than adding new ones.

5

Handover and aftercare

Two working sessions with your team, a short period in which we look over your shoulder during real incidents, and then an evaluation in which we compare recovery time against the baseline. If you want to hand maintenance over to us afterwards that's possible, but it is not a requirement.

Frequently asked questions about observability and monitoring.

What development teams, IT managers and people responsible for availability usually ask us before such a project begins.

What is the difference between monitoring and observability?
Monitoring answers questions you thought of in advance: is the service up, is the disk full, how many requests came in. Observability is about the questions you only think of during an incident, and which you then need to be able to answer without writing new code. The difference lies mainly in the richness of what you record: enough context per event to filter afterwards on something you had no dashboard for beforehand. In practice you need both, and most organisations have the first but not the second.
Which four signals do you mean exactly?
Latency, error rate, traffic and saturation. Together they describe whether your service is doing its job and whether it is running at its limit. The appeal is that the first three are about your user, not your infrastructure. A disk that is eighty per cent full is not a problem; a latency that doubles is. Alerting on the first category leads to nights when nothing was wrong, and that is exactly how alert fatigue arises.
Will you replace our current platform?
Usually not, and we rarely recommend it. Grafana, Datadog, Elastic, New Relic and your cloud provider's services do what they're meant to do; the problem almost always lies in what your software feeds into them. That's the side we set up. If you do want to switch, instrumenting through OpenTelemetry is the sensible order: it makes your telemetry independent of the platform, so the move afterwards is a matter of configuration rather than a second rebuild.
What does storage cost, and how do we keep it under control?
The observability bill comes almost entirely down to how much you keep and for how long. The approach that holds up is to choose per type of data. Metrics are small and can be kept for a long time, because they show you how something develops over months. Log lines are large and mostly useful in the days that follow. Traces are the largest, and a sample is enough, with the rule that anything that fails is kept in full. Together, those three choices make a difference of many times over in practice, without you giving anything up.
Does this also work if our software doesn't consist of separate services?
Yes, and it's simpler. With a single application, the gain lies less in tracing and more in bringing together what already exists: metrics per function, log lines with enough context to filter on, and alerting tied to the service. We set that up in the same way, only the step across system boundaries falls away. If you later grow into multiple services, the foundation is already in place.
How does this fit with our security?
The same logging that helps you during an outage is what you need to reconstruct what happened after a security incident. That isn't an extra project but the same work with a second payoff, and it connects directly to vulnerability management. We do pay attention to what ends up in the log lines: personal data and secrets don't belong there, and that's a choice you make during instrumentation rather than fixing it afterwards.
Who needs to pick this up on our side?
Someone who is on call when something breaks. Observability set up by someone other than the people who use it is rarely used: the dashboards then answer the questions the builder had, not the questions that come up at three in the morning. That's why we work together with your own developers and operations staff, and the handover is part of the engagement, not an appendix at the end.
How quickly will we notice a difference?
The first gains usually come from bringing together what already exists and cleaning up the alerting; you'll notice that within the first sprints. Tracing across system boundaries requires changes to the software and takes longer, service by service. We deliberately start with the service where the most time is lost, so the first delivery is also the most noticeable one.
Can you also do this for off-the-shelf software?
For the part we can reach, yes: traffic coming in and going out, availability, call response times, and the queues in between. What happens inside such a package depends on what the vendor makes available. In practice that's enough to determine whether a problem lies with you or with them, which is usually exactly the question you want to be able to answer.
What is an error budget and do we need one?
An error budget is the margin between your target and one hundred per cent. If you agree that 99.9 per cent of requests succeed, that last tenth of a per cent is your budget. As long as it isn't used up, new functionality can ship; once it runs out, attention goes to stability first. That's useful as soon as there's debate between pushing ahead with new features and fixing things, and unnecessary as long as there's no such debate. We recommend it for the third variant, not the first.
Do you work with OpenTelemetry or with something from a vendor?
OpenTelemetry by default. It's the open standard for metrics, logs and traces, and almost every platform can read it. The advantage is that the instrumentation in your code isn't tied to a vendor: if you switch platforms, you simply point the stream somewhere else. Only if you're already deep into a vendor-specific approach and have no plans to switch might it make more sense to keep building on that.

Talk to us about your monitoring.

A no-obligation half-hour introduction. Tell us what went wrong last time and how long it took before anyone knew where the problem was, and we'll tell you where we would start, even if we don't end up working together.

Edit content