Most production systems have logs.

But as discussed in The Difference Between “We Have Logs” and “We Can Debug an Incident”, having logs does not necessarily mean that we can understand what happened during an incident.

It is quite common for a service to become unexpectedly slow in production. When this happens, investigating the root cause without metrics, traces, and other telemetry can become extremely time consuming, or even nearly impossible.

Without sufficient observability, developers often have to dig through the code, inspect logs manually, and try to reconstruct what happened from incomplete information. With proper telemetry, however, we can quickly identify where the latency is coming from, which requests are affected, and which downstream dependencies might be contributing to the problem.

The tricky part is that these problems often occur only under very specific conditions. For example, during traffic spikes, with certain types of requests, or when a particular dependency becomes slow.

A common reaction is to increase the available cloud resources: add more CPU, memory, instances, or scale up the affected service. This can be a quick way to reduce the symptoms, and sometimes it is exactly the right thing to do.

However, it is not necessarily an efficient solution, and it is certainly not always a cheap one. Without proper telemetry, teams may end up paying for significantly more infrastructure without actually understanding what caused the performance degradation in the first place.

Modern backend systems need more than logs.

They need a way to answer three different questions:

• Logs: What happened?
• Metrics: How often and how badly is it happening?
• Traces: Where did the request go, and where did it spend its time?

And there is one more piece that connects everything together:

Context propagation.

Logs Alone Are Not Enough

Consider a typical microservice architecture:

Client
   |
   v
API Gateway
   |
   v
Order Service
   |
   +----> Payment Service
   |
   +----> Inventory Service
              |
              v
        External Service

An order request can potentially generate dozens of log entries across several services.

During an incident, searching for:

ERROR payment failed

may return hundreds of results.

The problem is not necessarily missing logs.

The problem is that we don’t know how the individual events are connected.

This is where context propagation becomes essential.

Context Propagation

Context propagation means carrying information about the current operation across service boundaries.

For example:

Request
   |
   | correlation_id=abc123
   v
Order Service
   |
   | correlation_id=abc123
   v
Payment Service
   |
   | correlation_id=abc123
   v
Inventory Service

Now logs from different services can be associated with the same request.

A useful context can contain information such as:

• Trace ID
• Span ID
• Request ID
• Correlation ID
• Tenant ID
• Business operation

The important point is that this context must travel with the request.

Distributed Tracing

Distributed tracing takes this concept further.

Instead of looking at individual log entries, we can see the complete request flow.

For example:

HTTP Request
│
├── API Gateway             15 ms
│
├── Order Service           80 ms
│   │
│   ├── Database            20 ms
│   │
│   └── Payment Service     55 ms
│       │
│       └── Payment API     50 ms
│
└── Response

This immediately tells us where the time was spent.

Without tracing, we might only see:

Order created
Payment successful

The logs tell us what happened.

The trace tells us how the request travelled through the system.

What Is a Trace?

A trace represents a single request or operation as it travels through the system.

A trace consists of multiple spans.

For example:

Trace
│
├── HTTP POST /orders
│
├── createOrder()
│
├── SELECT order
│
├── POST /payments
│
└── UPDATE order

Each span can contain:

• start time
• duration
• service name
• operation name
• attributes
• status
• events

This makes distributed systems much easier to understand.

Metrics Answer a Different Question

Metrics are not a replacement for logs or traces.

They answer a different question:

How is the system behaving over time?

For example:

http_requests_total = 2,431,920
http_request_errors = 12,430
http_request_duration_p95 = 850ms

These numbers immediately tell us that something may be wrong.

Metrics are particularly useful for detecting:

• increasing error rates
• latency degradation
• resource exhaustion
• traffic spikes
• unusual patterns

The Three Pillars Work Together

The real value comes from combining them.

Imagine an alert:

HTTP 5xx rate increased from 0.2% to 8%.

This is a metric.

It tells us that something is wrong.

We then use a trace to find:

Order Service
    ↓
Payment Service
    ↓
Payment Provider

The trace shows that the Payment Service is spending 5 seconds waiting for the external provider.

Finally, logs provide the detailed error:

ERROR payment provider request timed out

Now we have a complete picture.

Metrics
    ↓
Something is wrong

Tracing
    ↓
Where is it happening?

Logs
    ↓
What exactly happened?

Context Propagation Connects Everything

The real power comes when all three signals share the same context.

For example:

Trace ID: 4bf92f3577b34da6

appears in:

• the distributed trace
• application logs
• related metrics or exemplars

Now an engineer can move from:

High latency

to:

Affected service

to:

Specific request

to:

Specific error

without manually reconstructing the entire request flow.

OpenTelemetry

One of the most important projects in this area is OpenTelemetry.

OpenTelemetry provides a vendor-neutral framework for collecting and exporting:

• traces
• metrics
• logs

It supports many languages, including:

• Java
• Go
• Rust
• Python
• JavaScript

This is particularly valuable for organizations that do not want their application tightly coupled to a single observability vendor.

Instead of instrumenting the application specifically for one monitoring platform, OpenTelemetry provides a common telemetry model.

OpenTelemetry Collector

Applications do not necessarily need to send telemetry directly to the final observability backend.

A common architecture is:

Application
    |
    | OpenTelemetry
    v
OpenTelemetry Collector
    |
    +----> Prometheus
    |
    +----> Jaeger
    |
    +----> Grafana
    |
    +----> Other backend

The Collector can receive, process, transform, and export telemetry.

This provides another useful separation:

Application
    ≠
Observability backend

The application produces telemetry.

The infrastructure decides where that telemetry goes.

Prometheus, Grafana, and Jaeger

Several open-source tools are commonly used together.

Prometheus

Prometheus is widely used for collecting and querying metrics.

It is particularly useful for:

• request rates
• error rates
• latency
• resource usage
• application metrics

Grafana

Grafana provides dashboards and visualization.

It can combine information from multiple data sources and provide a central place for monitoring system health.

Jaeger

Jaeger is a distributed tracing platform.

It allows engineers to inspect traces and understand how requests move through distributed systems.

These tools can be combined with OpenTelemetry rather than requiring application code to be tightly coupled to each individual backend.

Observability Is Not Just Monitoring

Monitoring typically answers:

“Is something wrong?”

Observability aims to answer:

“Why is it wrong?”

For example:

Monitoring:

Payment Service latency > 2 seconds

This tells us there is a problem.

Observability allows us to investigate:

Payment Service
    |
    └── HTTP request
          |
          └── External Payment API
                  |
                  └── 2.8 second response time

Now we have a potential explanation.

Don’t Instrument Everything Blindly

More telemetry does not automatically mean better observability.

Logging every request body and creating spans for every tiny internal operation can produce enormous amounts of data.

This creates new problems:

• increased storage costs
• increased network traffic
• more noise
• harder analysis
• potentially sensitive data exposure

Instrumentation should be intentional.

Focus on information that helps answer operational questions.

What Should You Instrument?

At a minimum, consider collecting:

HTTP

• request count
• status code
• latency
• route

Database

• query latency
• connection pool usage
• error rate

External Services

• request count
• latency
• timeout rate
• error rate

Business Operations

Technical metrics are useful, but business metrics can be even more valuable.

For example:

orders_created
payments_failed
orders_cancelled
checkout_duration

A system can be technically healthy while the business is failing.

A Practical Production Example

Imagine users report:

“Checkout is slow.”

Your metrics show:

checkout_duration_p95 = 4.8s

You open a trace and see:

Checkout
│
├── Order Service       100ms
├── Inventory Service   150ms
└── Payment Service     4.4s

You investigate the Payment Service trace:

Payment Service
│
└── POST payment-provider
        └── 4.2s

Finally, the logs show:

ERROR payment provider request timed out
trace_id=4bf92f3577b34da6
order_id=12345

In a few minutes, the investigation went from:

"Checkout is slow."

to:

"Payment provider requests are taking more than four seconds."

That is the real value of observability.

Final Thoughts

Modern production systems are too distributed and too dynamic to be understood through logs alone.

Logs, metrics, and traces provide different perspectives:

Logs
  → What happened?

Metrics
  → How often and how badly?

Traces
  → Where did it happen?

Context propagation
  → How are all these events connected?

Together, they transform production debugging from searching through thousands of log entries into a structured investigation.

Tools such as OpenTelemetry, Prometheus, Grafana, and Jaeger make this possible without tightly coupling the application to a single observability platform.

The goal is not to collect more telemetry.

The goal is to collect the right telemetry so that when production breaks, engineers can understand what happened and why.

Key Takeaways

• Logs alone are rarely enough for distributed systems.
• Metrics help detect problems and understand their impact.
• Traces show how individual requests move through the system.
• Context propagation connects telemetry across service boundaries.
• OpenTelemetry provides a vendor-neutral approach to collecting telemetry.
• Prometheus, Grafana, and Jaeger can be combined to build a powerful observability stack.
• Good observability is about answering why, not just detecting that something is wrong.
• More telemetry is not necessarily better telemetry.