Add span tags and filter and group your application performance
7 minutes to complete
Datadog APM allows you to customize your traces to include any additional information you might need to maintain observability into your business. You can use this to identify a spike in the throughput of a certain enterprise customer, or the user suffering the highest latency, or to pinpoint the database shard generating the most errors.
In this example, a customer ID is added to traces allowing the customers that have the slowest performance to be identified. Customization of traces is based on tags that seamlessly integrate APM with the rest of Datadog and come in the form of key:value
pairs of metadata added to spans.
- Follow the example to get your code instrumented.
Depending on the programming language you are you using, you’ll need to set the tags to add to your spans differently.
Note: take note of the service and resource names you are working on, these will come in handy later. In this example, the service is the Ruby server web-store
and the resource (endpoint) is ShoppingCartController#checkout
.
The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with setTag
method.
import io.opentracing.Span;
import io.opentracing.util.GlobalTracer;
@WebServlet
class ShoppingCartServlet extends AbstractHttpServlet {
@Override
void doGet(HttpServletRequest req, HttpServletResponse resp) {
// Get the active span
final Span span = GlobalTracer.get().activeSpan();
if (span != null) {
// customer_id -> 254889
span.setTag("customer.id", customer_id);
}
// [...]
}
}
The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with set_tag
method.
from ddtrace import tracer
@app.route('/shopping_cart/<int:customer_id>')
@login_required
def shopping_cart(customer_id):
# Get the active span
current_span = tracer.current_span()
if current_span:
# customer_id -> 254889
current_span.set_tag('customer.id', customer_id)
# [...]
The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with set_tag
method.
require 'ddtrace'
# get '/shopping_cart/:customer_id', to: 'shopping_cart#index'
class ShoppingCartController < ApplicationController
# GET /shopping_cart
def index
# Get the active span and set customer_id -> 254889
Datadog::Tracing.active_span&.set_tag('customer.id', params.permit([:customer_id]))
# [...]
end
# POST /shopping_cart
def create
# [...]
end
end
The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with SetTag
method.
package main
import (
muxtrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/gorilla/mux"
"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)
func handler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
// Get the active span from a Go Context
if span, ok := tracer.SpanFromContext(r.Context()); ok {
// customer_id -> 254889
span.SetTag("customer.id", vars["customerID"])
}
// [...]
}
func main() {
tracer.Start(tracer.WithServiceName("web-store"))
defer tracer.Stop()
// Use auto-instrumentation
mux := muxtrace.NewRouter()
mux.HandleFunc("/shopping_cart/{customerID}", handler)
http.ListenAndServe(":8080", mux)
}
The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with setTag
method.
app.get('/shopping_cart/:customer_id', (req, res) => {
// Get the active span
const span = tracer.scope().active()
if (span !== null) {
// customer_id -> 254889
span.setTag('customer.id', req.params.customer_id)
}
// [...]
})
Add tags directly to a Datadog.Trace.Span
object by calling Span.SetTag()
. For example:
public class ShoppingCartController : Controller
{
private IShoppingCartRepository _shoppingCartRepository;
[HttpGet]
public IActionResult Index(int customerId)
{
// Access the active scope through the global tracer (can return null)
var scope = Tracer.Instance.ActiveScope;
if (scope != null)
{
// Add a tag to the span for use in the datadog web UI
scope.Span.SetTag("customer.id", customerId.ToString());
}
var cart = _shoppingCartRepository.Get(customerId);
return View(cart);
}
}
Note: Datadog.Trace.Tracer.Instance.ActiveScope
returns null
if there is no active span.
The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag in the meta
array.
<?php
namespace App\Http\Controllers;
class ShoppingCartController extends Controller
{
public shoppingCartAction (Request $request) {
// Get the currently active span
$span = \DDTrace\active_span();
if (null !== $span) {
// customer_id -> 254889
$span->meta['customer_id'] = $request->get('customer_id');
}
// [...]
}
}
?>
You might have to wait a few minutes between deploying your updated code and seeing the new tags in the Datadog UI
- Go to the Services page and click on the service that you added tags to. Scroll down and click on the specific resource where the tag was added in the Resource table. Scroll down to the Traces table
The Trace table shows you both the overall latency distribution of all traces in the current scope (service, resource, and timeframe) and links to individual traces. You can sort this table by duration or error code to identify erroneous operation or opportunities for optimization.
- Click into one of your traces
In this view you can see the flame graph on top and the additional information windows beneath it. The Datadog flame graph allows you to have an at a glance view of the duration and status of every logical unit (span) that impacts a request. The flame graph is fully interactive and you can pan it (by dragging) or zoom in and out (by scrolling). Clicking on any span provides more information about that span in particular in the bottom part of the view.
The bottom part of the view includes additional information about the trace or any selected span. Here you can see all default tags as well as the ones you manually include. In addition to these, you can also switch to view associated Host and Log information.
- Navigate to the Trace Explorer page.
The Trace Search page allows you to identify specific Traces and Indexed Spans you are interested in. Here you can filter by time a set of default tags (such as Env
,Service
, Resource
and many more).
Find a trace that has the new tag. To do this use the facet explorer on the left to find the Resource name you set at the beginning of this guide and click into one of the rows you see there.
Find the new tag that you added to the trace. Click on it and select Create facet for @[your facet name]
(remember, this is customer_id in our example)
You can now determine the displayed name of your facet and where to place it in the facet explorer.
You should now be able to see the facet you created in the Facet Explorer. The fastest way to find it is by using the Search facets
box.
- Navigate to the Analytics page
The Analytics page is a visual query building tool that allows you to conduct an investigation into your traces with infinite cardinality. It relies on facets to filter and scope the query, read more in the Trace Explorer overview.
- Choose the service you’ve been working on from the service facet list, choose Error from the status facet and select
customer_id
(or any other tags you added to your spans) from the group by field.
- Remove Error from the query, change the
count *
measure to Duration
and change the graph type to Top List
.
You can now see the customers that have the slowest average requests. Note: If you’d like to make sure your customers never pass a certain threshold of performance, you can export this query to a monitor, alternatively, you can save this visualization to a dashboard and keep an eye over it over time.
Finally, you can also see all the traces relevant to your query by clicking the visualization and selecting View traces
.
Further Reading
Additional helpful documentation, links, and articles: