Pomerium’s OpenTelemetry Tracing Support: Deeper Observability, Made Easy

Pomerium allows you to securely access Kubernetes APIs, internal apps, databases, and more—without a VPN. But even with faster, direct access, understanding performance issues or request failures in a distributed environment still requires the right observability to trace what’s happening behind the scenes. That’s why we’re excited to announce Pomerium’s newly improved OpenTelemetry (OTEL) tracing support! With detailed, contextual tracing, following request flows across Pomerium and your apps just became far easier—and a lot more delightful. What is OpenTelemetry (OTEL)? OpenTelemetry is an open-source observability framework that standardizes how applications collect, process, and export telemetry data such as metrics, traces, and logs. It’s the successor to OpenCensus and OpenTracing, and is now the de facto industry standard for modern observability. Why Move to OTEL? Previously, Pomerium used OpenCensus for tracing. However, OpenCensus has been deprecated, and upstream projects—including Envoy Proxy—have removed support for it entirely. If you haven’t run into it before: Pomerium is built on Envoy Proxy, a modern, battle-tested Layer 7 proxy that powers huge production systems at companies like Lyft, Apple, and Shopify. Key reasons for moving to OpenTelemetry: Future-proofing: OTLP (OpenTelemetry Protocol) is now the ubiquitous standard across most observability tools. Deeper tracing: OpenCensus could not trace request flows end-to-end through Pomerium and Envoy. Better integration: OpenTelemetry support ensures broad compatibility with a wide range of collectors and backends like Jaeger, Tempo, and Honeycomb. Now, Pomerium’s request tracing flows seamlessly from Envoy into Pomerium’s core services—and you can export it into any OTLP-compatible backend like Jaeger, Tempo, Honeycomb, or others. Why Are OTEL Traces Important? Traces capture the full context of a request—from the moment it enters the system to all the internal services it touches. While logs and metrics show individual points of data, traces show the journey of a request across systems. As a context- and identity-aware proxy, Pomerium often acts as the critical entrypoint into your distributed systems, especially when handling: Authentication (OAuth/OIDC flows) Authorization decisions Secure routing to protected upstream services Without proper tracing, debugging complex workflows—like an OAuth login flow that fails after five redirects—becomes guesswork. With tracing, every step becomes visible. How Pomerium Implements OpenTelemetry Tracing Pomerium now uses the OpenTelemetry SDK to instrument key parts of its architecture: Envoy: Ingress traffic and HTTP request handling Authentication: Identity provider interactions Authorization: Policy evaluation and enforcement Proxy: Secure traffic forwarding Data Broker: Internal service communication Control Plane: Configuration and service coordination Even when running Pomerium in "all-in-one" mode, traces are logically separated by component (via service names). Custom Envoy span naming ensures traces include meaningful names like host, path, and method. Full End-to-End Tracing Through Envoy Pomerium’s ability to offer full, detailed tracing starts with its foundation: Envoy Proxy. By building on Envoy, Pomerium inherits: Scalability: Envoy is designed for cloud-native, high-volume environments. Feature richness: Native support for advanced protocols, observability, retries, load balancing, and OTEL tracing. Reliability: Used in production by companies like Lyft, Apple, and Shopify. Our close integration with Envoy means your access proxy is future-proofed. Pomerium didn’t just slap tracing on top. We contributed upstream fixes—like PR #37692—to ensure accuracy and completeness. Solving Redirects and Sampling in Tracing Distributed tracing isn’t just plug-and-play—especially when redirects are involved. Pomerium often handles OAuth flows with 5+ redirects across Identity Providers. However, HTTP redirects don't carry tracing headers, so traces could become fragmented. Our Solution Propagating trace context in query parameters and OAuth state. Ensuring sampling consistency throughout redirects. Unified trace view for complex auth flows. Use Cases for Tracing Pomerium Requests Real-world examples: Understand user authentication flows Debug mysterious authentication failures Diagnose performance bottlenecks Analyze network or external system slowness Each trace provides detailed timing, metadata, and authorization decisions. Set Up Tracing Today Tracing isn’t always easy, but we’ve made it as simple as possible with Pomerium v0.29.0. We can’t wait to hear what you build and what insights you uncover! Get started today: Visit the Pomerium Tracing Documentation → Read more about what’s new in Pomerium v0.29.0 → Try Pomerium Zero

May 1, 2025 - 19:37

Pomerium’s OpenTelemetry Tracing Support: Deeper Observability, Made Easy

Pomerium allows you to securely access Kubernetes APIs, internal apps, databases, and more—without a VPN. But even with faster, direct access, understanding performance issues or request failures in a distributed environment still requires the right observability to trace what’s happening behind the scenes.

That’s why we’re excited to announce Pomerium’s newly improved OpenTelemetry (OTEL) tracing support!

With detailed, contextual tracing, following request flows across Pomerium and your apps just became far easier—and a lot more delightful.

What is OpenTelemetry (OTEL)?

OpenTelemetry is an open-source observability framework that standardizes how applications collect, process, and export telemetry data such as metrics, traces, and logs. It’s the successor to OpenCensus and OpenTracing, and is now the de facto industry standard for modern observability.

Why Move to OTEL?

Previously, Pomerium used OpenCensus for tracing. However, OpenCensus has been deprecated, and upstream projects—including Envoy Proxy—have removed support for it entirely.

If you haven’t run into it before: Pomerium is built on Envoy Proxy, a modern, battle-tested Layer 7 proxy that powers huge production systems at companies like Lyft, Apple, and Shopify.

Key reasons for moving to OpenTelemetry:

Future-proofing: OTLP (OpenTelemetry Protocol) is now the ubiquitous standard across most observability tools.
Deeper tracing: OpenCensus could not trace request flows end-to-end through Pomerium and Envoy.
Better integration: OpenTelemetry support ensures broad compatibility with a wide range of collectors and backends like Jaeger, Tempo, and Honeycomb.

Now, Pomerium’s request tracing flows seamlessly from Envoy into Pomerium’s core services—and you can export it into any OTLP-compatible backend like Jaeger, Tempo, Honeycomb, or others.

Why Are OTEL Traces Important?

Traces capture the full context of a request—from the moment it enters the system to all the internal services it touches.

While logs and metrics show individual points of data, traces show the journey of a request across systems.

As a context- and identity-aware proxy, Pomerium often acts as the critical entrypoint into your distributed systems, especially when handling:

Authentication (OAuth/OIDC flows)
Authorization decisions
Secure routing to protected upstream services

Without proper tracing, debugging complex workflows—like an OAuth login flow that fails after five redirects—becomes guesswork. With tracing, every step becomes visible.

How Pomerium Implements OpenTelemetry Tracing

Pomerium now uses the OpenTelemetry SDK to instrument key parts of its architecture:

Envoy: Ingress traffic and HTTP request handling
Authentication: Identity provider interactions
Authorization: Policy evaluation and enforcement
Proxy: Secure traffic forwarding
Data Broker: Internal service communication
Control Plane: Configuration and service coordination

Even when running Pomerium in "all-in-one" mode, traces are logically separated by component (via service names).

Custom Envoy span naming ensures traces include meaningful names like host, path, and method.

Full End-to-End Tracing Through Envoy

Pomerium’s ability to offer full, detailed tracing starts with its foundation: Envoy Proxy.

By building on Envoy, Pomerium inherits:

Scalability: Envoy is designed for cloud-native, high-volume environments.
Feature richness: Native support for advanced protocols, observability, retries, load balancing, and OTEL tracing.
Reliability: Used in production by companies like Lyft, Apple, and Shopify.

Our close integration with Envoy means your access proxy is future-proofed.

Pomerium didn’t just slap tracing on top. We contributed upstream fixes—like PR #37692—to ensure accuracy and completeness.

Solving Redirects and Sampling in Tracing

Distributed tracing isn’t just plug-and-play—especially when redirects are involved.

Pomerium often handles OAuth flows with 5+ redirects across Identity Providers.

However, HTTP redirects don't carry tracing headers, so traces could become fragmented.

Our Solution

Propagating trace context in query parameters and OAuth state.
Ensuring sampling consistency throughout redirects.
Unified trace view for complex auth flows.

Use Cases for Tracing Pomerium Requests

Real-world examples:

Understand user authentication flows
Debug mysterious authentication failures
Diagnose performance bottlenecks
Analyze network or external system slowness

Each trace provides detailed timing, metadata, and authorization decisions.

Set Up Tracing Today

Tracing isn’t always easy, but we’ve made it as simple as possible with Pomerium v0.29.0.

We can’t wait to hear what you build and what insights you uncover!

Get started today:

Places you can connect with us: