The Difference Between 'We Have Logs' and 'We Can Debug an Incident'
One of the most common statements during a production incident is:
“Don’t worry, we have logs.”
And technically, that may be true.
The system may generate:
- thousands of log entries per minute
- centralized logs
- multiple log levels
- timestamps
- structured JSON output
Yet when something actually breaks, the team still asks:
“What exactly happened?”
This highlights an important distinction:
Having logs is not the same as having enough information to debug an incident.
Logs Are Not Automatically Observability
A system can produce a large amount of logs and still be almost impossible to debug.
Consider this:
ERROR Request failed
ERROR Database error
ERROR Service unavailable
ERROR Operation failed
There are logs.
But there is almost no useful information.
Which request failed?
Which user or business operation was involved?
Which service produced the error?
What was the original cause?
Which other services were involved?
The problem is not the absence of logs.
The problem is the absence of diagnostic context.
The Information You Actually Need During an Incident
A useful log entry should help answer questions such as:
- What happened?
- Where did it happen?
- Which request was involved?
- Which business operation was being executed?
- Which service produced the error?
- What was the original cause?
- What happened immediately before the failure?
In a distributed system, this information becomes even more important.
A request might travel through:
API Gateway
↓
Order Service
↓
Payment Service
↓
External Payment Provider
If the request fails, a log entry such as:
ERROR Payment failed
is almost useless.
A much more useful event would look like:
ERROR Failed to process payment
request_id=8f2a...
correlation_id=3e91...
order_id=12345
payment_provider=example
operation=AUTHORIZE_PAYMENT
error=timeout
Now the incident can actually be investigated.
The Problem with Logs Without Correlation
Imagine a system processing hundreds of requests per second.
You find this log:
ERROR Failed to create order
Which request was it?
Without a correlation ID or request ID, you may need to search through thousands of unrelated log entries.
With:
ERROR Failed to create order
request_id=abc-123
correlation_id=xyz-456
order_id=98765
you can follow the request through the entire system.
This is especially important in microservice architectures.
A single request may produce logs in multiple services:
API Gateway
correlation_id=abc123
Order Service
correlation_id=abc123
Payment Service
correlation_id=abc123
Inventory Service
correlation_id=abc123
The correlation ID connects the entire flow.
Rust: A Different Problem with Stack Traces
Rust introduces another important challenge.
In languages such as Java, an exception typically contains a stack trace automatically.
For example:
java.lang.NullPointerException
at com.example.OrderService.process(OrderService.java:42)
at com.example.OrderController.create(OrderController.java:27)
This immediately tells you where the error originated.
Rust does not automatically provide a stack trace for ordinary Result errors.
Consider:
fn process_order() -> Result<(), AppError> {
repository.save()?;
Ok(())
}
If the operation fails, the returned error may contain only:
database connection failed
The error message may be correct.
But it does not necessarily tell you:
- which function called the operation
- which business operation was in progress
- which request triggered it
- where the error propagated from
This can make production debugging significantly more difficult.
Error Propagation Is Not the Same as Context
Rust’s ? operator makes error propagation elegant:
fn process_order() -> Result<(), AppError> {
validate_order()?;
reserve_inventory()?;
charge_payment()?;
Ok(())
}
However, this can also hide important context.
Suppose:
fn process_order() -> Result<(), AppError> {
charge_payment()?;
Ok(())
}
The final error might be:
payment provider timeout
But the application may not know:
Order processing failed
└── Payment authorization failed
└── HTTP request timed out
The original error may be technically accurate, but the business context has been lost.
Add Context While Propagating Errors
A common solution in Rust is to add context while propagating errors.
For example, with the anyhow crate:
use anyhow::Context;
fn process_order() -> anyhow::Result<()> {
charge_payment()
.context("failed to charge payment")?;
Ok(())
}
Now the error contains more useful information:
failed to charge payment
Caused by:
payment provider timeout
With multiple layers:
use anyhow::Context;
fn process_order() -> anyhow::Result<()> {
charge_payment()
.context("failed to process payment")?;
Ok(())
}
fn charge_payment() -> anyhow::Result<()> {
call_payment_provider()
.context("payment provider request failed")?;
Ok(())
}
The resulting error can represent the chain:
failed to process payment
caused by: payment provider request failed
caused by: connection timeout
This is much more useful than a single generic message.
Typed Errors and Context
For application code, typed errors are often preferable.
Example:
#[derive(Debug, thiserror::Error)
pub enum OrderError {
#[error("payment failed")]
PaymentFailed {
#[source]
source: PaymentError,
},
#[error("order not found: {0}")]
OrderNotFound(OrderId),
}
This allows the application to distinguish between:
- expected business errors
- technical failures
- integration failures
However, the error type itself may still not contain enough context for debugging.
For example:
payment failed
may be technically structured but still not tell us:
order_id=12345
customer_id=456
payment_provider=example
That context usually belongs in structured logs.
Errors and Logs Serve Different Purposes
A common mistake is expecting the error object to contain everything required for operational debugging.
This often leads to enormous error types containing:
- request IDs
- user IDs
- service names
- URLs
- business context
- infrastructure metadata
This is usually not ideal.
A better separation is:
Errors
Describe what went wrong and allow the application to handle the failure.
PaymentProviderTimeout
Logs
Describe the operational context.
ERROR
operation=PROCESS_ORDER
order_id=12345
payment_provider=example
correlation_id=abc123
error=PaymentProviderTimeout
The error describes the failure.
The log describes the incident.
A Common Rust Logging Mistake
Consider:
match service.process_order().await {
Ok(_) => {}
Err(error) => {
tracing::error!("Order processing failed");
}
}
This log is almost useless.
The error itself is not included.
A better version:
match service.process_order().await {
Ok(_) => {}
Err(error) => {
tracing::error!(
error = ?error,
order_id = %order_id,
"Order processing failed"
);
}
}
Now the error chain is available.
If the error implements std::error::Error, structured logging can often capture the source chain as well.
Debug vs Display in Rust Logs
This distinction is important.
Consider:
tracing::error!(error = %error, "Request failed");
The % formatter uses the Display implementation.
This is usually intended for a user-friendly error message.
Using:
tracing::error!(error = ?error, "Request failed");
uses the Debug representation.
This often includes more technical details.
For example:
tracing::error!(
error = ?error,
"Failed to process order"
);
can provide much more information when the error type contains nested causes.
However, even a detailed Debug representation is not a replacement for a proper stack trace.
Rust Does Not Automatically Give You a Stack Trace
This is an important distinction.
A Rust error such as:
#[derive(Debug, thiserror::Error)]
#[error("database operation failed")]
pub struct DatabaseError;
does not automatically contain:
process_order
→ create_order
→ repository.save
→ database driver
The error contains the error information defined by the type.
The call stack is a separate concept.
If stack traces are required, they need to be explicitly captured or configured using appropriate tooling.
For example, applications may use:
- tracing
- tracing-subscriber
- color-eyre
- backtrace
- panic backtraces through RUST_BACKTRACE
But each solves a slightly different problem.
A panic backtrace is not the same as an error propagation trace.
Panics Are Not Normal Application Errors
This distinction is important in Rust.
A panic:
panic!("unexpected state");
can produce a stack trace when backtraces are enabled:
RUST_BACKTRACE=1
However, normal application errors represented by:
Result<T, E>
do not automatically generate a stack trace.
This means that a production Rust application can have:
ERROR request failed: database connection failed
without any information about where the error was originally created or propagated.
The application did not panic.
Therefore, no panic backtrace was generated.
The Ideal Production Log
A production log should ideally answer several questions immediately.
For example:
{
"level": "ERROR",
"service": "order-service",
"operation": "PROCESS_ORDER",
"order_id": "12345",
"correlation_id": "abc-123",
"error": "payment provider timeout",
"error_chain": [
"payment authorization failed",
"HTTP request timed out"
],
"duration_ms": 3021
}
This is much more useful than:
ERROR Something went wrong
The goal is not to log everything.
The goal is to log enough information to reconstruct what happened.
Logs Should Help Reconstruct the Timeline
During an incident, the most useful question is often:
What happened immediately before the failure?
For example:
10:42:01 INFO Order processing started
10:42:01 INFO Inventory reservation requested
10:42:02 INFO Inventory reservation completed
10:42:02 INFO Payment authorization requested
10:42:05 ERROR Payment provider timeout
10:42:05 ERROR Order processing failed
This creates a timeline.
Without this context:
10:42:05 ERROR Order processing failed
the incident becomes much harder to understand.
Logs Are Only One Part of Observability
Logs are important, but they are not enough.
A modern backend system should ideally combine:
Logs
What happened?
Metrics
How often is it happening?
payment_timeout_total = 124
Traces
Where did the request spend its time?
API Gateway
↓ 10 ms
Order Service
↓ 20 ms
Payment Service
↓ 3000 ms
Payment Provider
The best incident investigations use all three.
Final Thoughts
There is a significant difference between:
“We have logs.”
and:
“We can debug this incident.”
Having logs does not guarantee that a system is observable.
To debug production incidents effectively, logs need:
- meaningful context
- correlation IDs
- structured fields
- useful error information
- consistent log levels
- enough information to reconstruct the timeline
Rust adds another important consideration.
Unlike languages with automatic exception stack traces, normal Result-based error propagation does not automatically provide a call stack.
Therefore, Rust applications need to be deliberate about:
- error context
- error chains
- structured logging
- tracing
- stack trace collection when appropriate
The goal is not to produce more logs.
The goal is to make the logs answer the question:
“What exactly happened, and where should I start looking?”
That is the difference between having logs and being able to debug an incident.
Key Takeaways
- Having logs does not automatically mean having observability.
- Logs need context, not just messages.
- Correlation IDs are essential in distributed systems.
- Rust Result errors do not automatically contain stack traces.
- Error propagation and call-stack tracing are separate concepts.
- Use error chains to preserve technical context.
- Use structured logs to preserve operational and business context.
- Combine logs, metrics, and traces for serious production debugging.
