OpenTelemetry in Node.js: Distributed Tracing From Zero to Production

Your API is slow. The logs say the request came in, the response went out, it took 2.3 seconds. Nothing logged an error. You have six services that might have touched that request. Where do you start?

This is exactly the problem distributed tracing solves. A trace is a causal record of one request's journey across your entire system - every service, every database call, every queue publish. A span is one unit of work within that journey. When your API is slow, you open the trace and see: "service A spent 50ms, service B spent 1800ms - specifically the getUserPreferences DB query". Case closed in 30 seconds.

OpenTelemetry is now the industry standard for generating this data. It's vendor-neutral (export to Jaeger, Grafana Tempo, Honeycomb, Datadog - your choice), has excellent Node.js support, and the auto-instrumentation means you can get traces from Express, Fastify, pg, redis, and most npm packages with zero manual code changes.

Core Concepts Before Code

A trace is a directed acyclic graph of spans with a shared traceId. Think of it as the full call tree for one request.

A span has: a name, start/end timestamps, a spanId, the parent spanId, a traceId, a status (OK/ERROR), and arbitrary key-value attributes. Attributes are where you put business context: user.id, order.id, http.route, db.statement.

Context propagation is how the traceId travels from service to service. Over HTTP it goes in headers (traceparent: 00-<traceId>-<spanId>-01). Over message queues you embed it in message metadata. Without propagation, you get separate disconnected traces instead of one unified trace.

Baggage is key-value data that propagates with the trace context - useful for carrying tenantId or userId through every service without threading it through every function signature. Use sparingly: baggage is sent on every outgoing call.

Project Setup

We'll instrument two services: an HTTP API (order-service) and a worker (notification-service). Start with the SDK setup, which MUST be the first thing that runs before any other imports.

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions \
  @opentelemetry/api

The Instrumentation File (Load This First)

// src/instrumentation.ts
// This file must be loaded via --require before anything else
// node --require ./dist/instrumentation.js ./dist/server.js

import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_SERVICE_VERSION, SEMRESATTRS_DEPLOYMENT_ENVIRONMENT } from '@opentelemetry/semantic-conventions';
import { BatchSpanProcessor, ParentBasedSampler, TraceIdRatioBased } from '@opentelemetry/sdk-trace-base';

const exporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318/v1/traces',
  headers: {
    // For Grafana Cloud, add auth header here:
    // Authorization: `Basic ${Buffer.from(`${instanceId}:${apiKey}`).toString('base64')}`,
  },
});

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: process.env.SERVICE_NAME ?? 'order-service',
    [SEMRESATTRS_SERVICE_VERSION]: process.env.npm_package_version ?? '0.0.0',
    [SEMRESATTRS_DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV ?? 'development',
  }),

  // BatchSpanProcessor buffers spans and sends in batches - much more efficient
  // than SimpleSpanProcessor which sends every span immediately
  spanProcessor: new BatchSpanProcessor(exporter, {
    maxQueueSize: 2048,
    maxExportBatchSize: 512,
    scheduledDelayMillis: 5000,
    exportTimeoutMillis: 30000,
  }),

  // Production sampling: trace 10% of requests, but always trace errors
  // ParentBasedSampler respects the sampling decision from upstream services
  sampler: new ParentBasedSampler({
    root: new TraceIdRatioBased(
      process.env.NODE_ENV === 'production' ? 0.1 : 1.0
    ),
  }),

  instrumentations: [
    getNodeAutoInstrumentations({
      // Disable noisy instrumentations you don't need
      '@opentelemetry/instrumentation-fs': { enabled: false },
      '@opentelemetry/instrumentation-http': {
        // Don't trace health check endpoints
        ignoreIncomingRequestHook: (req) => {
          return req.url === '/health' || req.url === '/metrics';
        },
      },
      '@opentelemetry/instrumentation-pg': {
        // Capture the actual SQL - useful but be careful with PII in WHERE clauses
        enhancedDatabaseReporting: true,
        addSqlCommenterCommentToQueries: true, // adds trace context to slow query logs!
      },
    }),
  ],
});

sdk.start();

// Flush spans before process exits - critical, don't skip this
process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('Tracing terminated'))
    .catch(console.error)
    .finally(() => process.exit(0));
});

export default sdk;

The addSqlCommenterCommentToQueries: true option is underrated: it appends /* traceparent='...' */ to your SQL. When that query shows up in pg_stat_statements or your slow query log, you can directly correlate it to a trace in Jaeger. We found this invaluable during an incident where a slow query was triggered by a specific code path that only activated for enterprise customers.

Manual Spans for Business Logic

Auto-instrumentation captures infrastructure calls. For your actual business logic, you need manual spans.

// src/services/orderService.ts
import { trace, SpanStatusCode, context, propagation } from '@opentelemetry/api';

const tracer = trace.getTracer('order-service', '1.0.0');

export async function processOrder(orderId: string, userId: string) {
  // Start a span for the entire business operation
  return tracer.startActiveSpan('order.process', async (span) => {
    try {
      // Attributes give context to the span - use semantic conventions where they exist
      span.setAttributes({
        'order.id': orderId,
        'user.id': userId,
        'app.component': 'order-service',
      });

      // Nested spans for sub-operations
      const inventory = await tracer.startActiveSpan('order.checkInventory', async (inventorySpan) => {
        inventorySpan.setAttribute('order.id', orderId);
        try {
          const result = await checkInventory(orderId);
          inventorySpan.setAttribute('inventory.available', result.available);
          inventorySpan.setStatus({ code: SpanStatusCode.OK });
          return result;
        } catch (err) {
          inventorySpan.setStatus({
            code: SpanStatusCode.ERROR,
            message: (err as Error).message,
          });
          inventorySpan.recordException(err as Error);
          throw err;
        } finally {
          inventorySpan.end();
        }
      });

      if (!inventory.available) {
        span.setAttribute('order.outcome', 'out_of_stock');
        span.setStatus({ code: SpanStatusCode.OK }); // Not an error, just a business outcome
        return { success: false, reason: 'out_of_stock' };
      }

      const payment = await chargePayment(orderId, userId);
      span.setAttribute('payment.method', payment.method);
      span.setAttribute('order.outcome', 'completed');
      span.setStatus({ code: SpanStatusCode.OK });

      return { success: true, paymentId: payment.id };
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: (err as Error).message,
      });
      throw err;
    } finally {
      span.end(); // Always end the span, even on error
    }
  });
}

The recordException call is important: it attaches the stack trace and error message as span events, so when you're looking at a failed span in Jaeger you can see exactly what went wrong without correlating to a separate log.

Context Propagation Across HTTP

Auto-instrumentation handles this for Express/Fastify/fetch automatically. But you need to understand it for message queues and background jobs.

// Service A: Injecting trace context into an HTTP request manually
import { context, propagation } from '@opentelemetry/api';

async function callNotificationService(payload: NotificationPayload) {
  const headers: Record<string, string> = {
    'Content-Type': 'application/json',
  };

  // Inject current trace context into outgoing headers
  propagation.inject(context.active(), headers);
  // headers now contains 'traceparent' and optionally 'tracestate'

  const response = await fetch('http://notification-service/notify', {
    method: 'POST',
    headers,
    body: JSON.stringify(payload),
  });

  return response.json();
}

// Service B: Extracting trace context from incoming message queue message
import { context, propagation, trace, SpanKind } from '@opentelemetry/api';

// For a BullMQ/RabbitMQ message processor
async function processQueueMessage(message: QueueMessage) {
  // The producer embedded trace context in message.headers
  const parentContext = propagation.extract(context.active(), message.headers ?? {});

  // Run our processing inside the extracted context
  // so spans will be children of the original trace
  return context.with(parentContext, async () => {
    const tracer = trace.getTracer('notification-service');
    return tracer.startActiveSpan('notification.process', {
      kind: SpanKind.CONSUMER, // Semantic convention for queue consumers
    }, async (span) => {
      try {
        span.setAttributes({
          'messaging.system': 'bullmq',
          'messaging.destination': 'notifications',
          'user.id': message.data.userId,
        });

        await sendNotification(message.data);
        span.setStatus({ code: SpanStatusCode.OK });
      } catch (err) {
        span.recordException(err as Error);
        span.setStatus({ code: SpanStatusCode.ERROR });
        throw err;
      } finally {
        span.end();
      }
    });
  });
}

Without the propagation.extract + context.with pattern, your notification service spans will be orphaned - separate traces with no connection to the original HTTP request that triggered the message. This is the gotcha that bites almost every team doing their first distributed tracing setup.

Correlating Logs with Traces

Your logs are useless in isolation during an incident if you can't connect them to the trace that was active when they were written. Here's how to inject trace context into your Winston logs:

// src/lib/logger.ts
import winston from 'winston';
import { trace, context } from '@opentelemetry/api';

const logger = winston.createLogger({
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format((info) => {
      // Inject trace context into every log entry
      const span = trace.getActiveSpan();
      if (span) {
        const ctx = span.spanContext();
        info['trace_id'] = ctx.traceId;
        info['span_id'] = ctx.spanId;
        info['trace_flags'] = `0${ctx.traceFlags.toString(16)}`;
      }
      return info;
    })(),
    winston.format.json()
  ),
  transports: [new winston.transports.Console()],
});

export default logger;

Now every log line contains trace_id. In Grafana, you can configure Loki to correlate with Tempo using trace_id as the link field. Click a slow trace span in Tempo → click "Logs for this span" → see every log line emitted during that span's execution. This workflow cuts incident investigation time dramatically.

Sampling in Production: Don't Trace Everything

Tracing 100% of production traffic is expensive and often unnecessary. A 10% tail sample means you see statistical patterns but miss individual slow requests. The right approach is tail-based sampling: always keep traces that had errors or exceeded a latency threshold.

// For tail-based sampling, you need a collector (OpenTelemetry Collector)
// and configure the tailsampling processor there. In your collector config:

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 500
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow-requests
        type: latency
        latency: { threshold_ms: 2000 }
      - name: probabilistic-sample
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

This configuration keeps 100% of error traces, 100% of traces slower than 2 seconds, and 5% of everything else. You'll capture all the interesting events and sample down the noise.

Running Locally with Jaeger

docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

Set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces, start your services, make a request, open http://localhost:16686. You'll see the full trace within seconds.

---

Building microservices that need to be observable in production, not just testable in dev? Aunimeda engineers have wired OpenTelemetry, Grafana Tempo, and structured logging into production stacks. Learn more about our AI and infrastructure solutions or contact us to discuss your observability setup.