From "grep every pod and hope" to one trace ID across a 7-service chain

Service names are generic for client confidentiality. The architecture, mechanisms and constraints are real.

Context

A multi-tenant, self-hosted data platform processes records through a chain of seven Spring Boot services on Kubernetes. Apache NiFi ingests the records.

Three properties of the topology made it hard to trace:

1. An asynchronous hop. Service 1 hands records to Service 2 through RabbitMQ, not by a direct call.

   

2. A mid-chain hop through NiFi. After Service 3, the record leaves the service mesh and passes back through NiFi before it continues.

   

3. A callback. Service 7 calls back into Service 3, out of order.

   

The problem and its cost

When I joined, a record that came out wrong or never came out could not be followed:

• No shared identifier. Nothing linked a log line in one service to the log line it caused in the next.
• Manual, per-pod log reading. kubectl logs, service by service, pod by pod: seven times at minimum.
• Timestamps instead of evidence. Two log lines three seconds apart in two services might or might not be the same record.
• Metrics disconnected from both the logs and the requests that caused them.
• Evidence loss. Pod restarts rotated logs away before anyone got to them.

Every incident started from zero and depended on who remembered the system best.

Requirements

• Answer three questions for any record: what happened, in order, across every service (traces), what each service logged at that moment (logs), and whether the system was already unhealthy (metrics).
• Multiple destinations. Some customer deployments also wanted application logs in their own database or SIEM, in addition to our dashboards.
• Adding a destination must be a configuration change, not a code change in every service.
• Instrumenting a service must not depend on each developer doing it by hand, correctly.

Decision 1: one collector, not direct exporters

Wiring each service directly to Tempo, Loki and Prometheus was rejected. Every new destination would have meant a new exporter in seven or more services. Instead, every service emits OpenTelemetry to a single OpenTelemetry Collector, which fans out to Tempo (traces), Loki (logs), Prometheus (metrics) and any customer-specific sinks. Grafana sits in front of all three as one screen.

Services speak one vendor-neutral protocol and never know what is listening. A new destination means one pipeline entry in the collector config.

Decision 2: instrumentation lives in the shared library

All services already depended on a common internal library for cross-cutting concerns. That library was extended to own the whole OpenTelemetry setup:

• SDK initialization, configured from one standard YAML block
• W3C traceparent propagation on every outbound HTTP call
• trace and span IDs pushed into the logging MDC, so every log line carries them
• propagation through RabbitMQ message headers, on publish and on consume

Onboarding a service takes two steps: add the dependency and add the config block (collector endpoint, service name, sampling rate). No developer writes spans, wires an SDK or handles correlation IDs by hand. Trace quality no longer depends on who wrote the service.

Decision 3: carry context through NiFi without instrumenting NiFi

NiFi runs no OpenTelemetry SDK, and it appears twice in the chain. Each appearance needs a different answer:

• Entry NiFi. No upstream context exists yet, because this is where the record first appears. The trace starts at Service 1, the first component that understands it. The short NiFi → Service 1 hop stays untraced. That is a known, documented gap, not an oversight.
• Mid-chain NiFi. The record already carries a trace from Service 3, and NiFi must not drop it. A processor copies the incoming traceparent header into a FlowFile attribute. The outbound InvokeHTTP processor then sends that attribute back out as a header, through its “Attributes to Send” setting. NiFi passes the context through without ever parsing it.

The RabbitMQ hop uses the same idea: the shared library writes traceparent into message headers on publish and restores it on consume. The broker needs no tracing support.

End-to-end propagation, including both NiFi hops and the Service 7 → Service 3 callback:

Hop	Mechanism
NiFi (entry) → Service 1	Untraced (no upstream context); trace starts in Service 1
Service → service (HTTP)	traceparent header, set by the shared library
Service 1 → RabbitMQ → Service 2	traceparent in message headers, set/read by the shared library
Service 3 → NiFi → Service 4	FlowFile attribute + InvokeHTTP “Attributes to Send”
Service 7 → Service 3 (callback)	traceparent header; appears in the same trace
Every log line	trace ID in MDC → Loki, linked from Tempo spans in Grafana

Result

• Investigation starts from one trace ID. Paste it into Grafana to get the whole path as a single waterfall: every service, the RabbitMQ hop, the round trip through NiFi and the callback. Every log line is one click from the span that produced it.
• Logs, traces and metrics share one identifier and one screen.
• Onboarding a new service is a dependency and a config block, applied identically every time.
• Adding a customer-specific log destination is a collector config change, with no code change in any service.
• New engineers learn the system’s shape from its traces instead of from a guided tour.

Known limits

• The entry NiFi → Service 1 hop is not traced.
• Propagation is only as strong as the weakest hop: a service that drops the header splits the trace in two. The shared library makes this the default behaviour, and code review enforces it for anything outside the library.