grafana/tempo

Grafana Tempo is an open source, easy-to-use, and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Prometheus, and Loki.

Business value of distributed tracing

Distributed tracing helps teams quickly pinpoint performance issues and understand the flow of requests across services. The Traces Drilldown UI simplifies this process by offering a user-friendly interface to view and analyze trace data, making it easier to identify and resolve issues without needing to write complex queries.

Refer to Use traces to find solutions to learn more about how you can use distributed tracing to investigate and solve issues.

Traces Drilldown UI: A better way to get value from your tracing data

We are excited to introduce the Traces Drilldown (formerly Explore Traces) app as part of the Grafana Explore suite. This app provides a queryless and intuitive experience for analyzing tracing data, allowing teams to quickly identify performance issues, latency bottlenecks, and errors without needing to write complex queries or use TraceQL.

Key Features:

• Intuitive Trace Analysis: Spot slow or error-prone traces with easy, point-and-click interactions.
• RED Metrics Overview: Use Rate, Errors, and Duration metrics to highlight performance issues.
• Automated Comparison: Identify problematic attributes with automatic trace comparison.
• Simplified Visualizations: Access rich visual data without needing to construct TraceQL queries.

To learn more see the following links:

• Traces Drilldown repo
• Traces Drilldown documentation
• Demo video

TraceQL

Tempo implements TraceQL, a traces-first query language inspired by LogQL and PromQL, which enables targeted queries or rich UI-driven analyses.

TraceQL metrics

TraceQL metrics is an experimental feature in Grafana Tempo that creates metrics from traces. Metric queries extend trace queries by applying a function to trace query results. This powerful feature allows for ad hoc aggregation of any existing TraceQL query by any dimension available in your traces, much in the same way that LogQL metric queries create metrics from logs.

Tempo is Jaeger, Zipkin, Kafka and OpenTelemetry compatible. It ingests batches in any of the mentioned formats, buffers them, and then writes them to Azure, GCS, S3, or local disk. As such, it's robust, cheap, and easy to operate.

Getting started with Tempo

• Get started documentation
• Deployment Examples
  • Docker Compose
  • Helm
  • Jsonnet

Further reading

To learn more about Tempo, consult the following documents & talks:

• How to get started with Tempo with Joe Elliott (video)
• Grafana blog posts about Tempo
• New in Grafana Tempo 2.0: Apache Parquet as the default storage format, support for TraceQL
• Get to know TraceQL: A powerful new query language for distributed tracing

Getting help

If you have any questions or feedback regarding Tempo:

• Grafana Labs hosts a forum for Tempo. This is a great place to post questions and search for answers.
• Ask a question on the Tempo Slack channel.
• File an issue for bugs, issues and feature suggestions.
• UI issues should be filed with Grafana.

OpenTelemetry

Tempo's receiver layer, wire format and storage format are all based directly on standards and code established by OpenTelemetry. We support open standards at Grafana!

Check out the Integration Guides to see examples of OpenTelemetry instrumentation with Tempo.

Other components

tempo-vulture

tempo-vulture is Tempo's bird themed consistency checking tool. It writes traces to Tempo and then queries them back in a variety of ways.

tempo-cli

tempo-cli is the place to put any utility functionality related to Tempo. See Documentation for more info.

License

Grafana Tempo is distributed under AGPL-3.0-only. For Apache-2.0 exceptions, see LICENSING.md.