Skip to content

Observability stack

pgway pushes OTLP/gRPC metrics (and optionally traces) — there is no Prometheus /metrics scrape endpoint. To view the metrics locally without assembling your own pipeline, the repo ships a bundled Docker Compose stack under docker/observability/:

Component Purpose Image
OTel Collector receives OTLP (gRPC :4317, HTTP :4318), re-exports metrics as Prometheus and traces → Tempo otel/opentelemetry-collector:0.161.0
Prometheus scrapes the collector; file-based TSDB, 15d retention prom/prometheus:v3.14.0
Tempo trace backend; receives spans from the collector, file-based local storage grafana/tempo:3.0.3
Grafana dashboards + trace exploration; Prometheus and Tempo datasources auto-provisioned grafana/grafana:13.2.2

The scrape config sets honor_labels: true, so the OTLP service.name resource attribute survives as the Prometheus job label (pgway, pgway-cp, pgway-dp). All published ports bind to 127.0.0.1 only.

Run the stack

docker compose -f docker/observability/compose.yaml up -d
Endpoint URL
Grafana http://localhost:3000 (admin / admin)
Prometheus UI http://localhost:9091
Tempo HTTP API http://localhost:3200 (queries, /ready; the trace UI is Grafana Explore)
OTLP gRPC / HTTP localhost:4317 / localhost:4318
Collector health check http://localhost:13133

Prometheus is published on 9091 because pgway's default gRPC listen port is :9090.

Prometheus TSDB, Tempo blocks, and Grafana state live in named volumes (prom-data, tempo-data, grafana-data) and survive docker compose down; run down -v to wipe them.

Point pgway at it

[otel]
enabled = true
endpoint = "localhost:4317"
insecure = true   # the bundled collector listens on plaintext OTLP/gRPC
# Optional traces (requires enabled = true):
# traces_enabled = true
# trace_sample_ratio = 0.1

The same keys work as PGWAY_OTEL_* environment variables. See Configuration for all keys and the metrics reference for what gets exported.

Traces

With otel.enabled = true and otel.traces_enabled = true, pgway exports spans over the same OTLP/gRPC endpoint:

  • Proxy hot path: pgway.proxy.request (parent) and pgway.proxy.upstream (Dial / RoundTrip child)
  • CP unary gRPC: pgway.cp.rpc

Sampling uses ParentBased + TraceIDRatioBased (otel.trace_sample_ratio, default 0.1). Span attributes mirror metrics (entrypoint / result / protocol, or method / result) — no raw URLs, headers, or tokens.

Traces land in the bundled Tempo and are queryable in Grafana via Explore → Tempo (datasource auto-provisioned, filter by service.name). To use a different backend instead, point pgway at it directly or swap the collector's otlp/tempo exporter.

Import the dashboard

A ready-made dashboard export ships with the stack:

pgway-otel-metrics.json

If you have the repo checked out, the file also lives at docker/observability/grafana/dashboards/pgway-otel-metrics.json.

In Grafana: Dashboards → New → Import, then upload the file or paste its contents. The dashboard's datasource variable falls back to your default Prometheus datasource, and the job variable filters by service.name.

Note

The export uses the Grafana dashboards v2 schema; import it into Grafana 13 or newer.