Add span tags and slice and dice your application performance
New announcements from Dash: Incident Management, Continuous Profiler, and more! New announcements from Dash!

Add span tags and slice and dice your application performance

7 minutes to complete

Datadog APM allows you to customize your traces to include any additional information you might need to maintain observability into your business. You can use this to identify a spike in the throughput of a certain enterprise customer, or the user suffering the highest latency, or to pinpoint the database shard generating the most errors.

In this example, a customer ID is added to traces allowing the customers that have the slowest performance to be identified. Customization of traces is based on tags that seamlessly integrate APM with the rest of Datadog and come in the form of key:value pairs of metadata added to spans.

Instrument your code with custom span tags

1) Follow the example to get your code instrumented.

Depending on the programming language you are you using, you’ll need to set the tags to add to your spans differently.

Note: take note of the service and resource names you are working on, these will come in handy later. In this example, the service is the Ruby server web-store and the resource (endpoint) is ShoppingCartController#checkout.

The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with setTag method.

import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;

@WebServlet
class ShoppingCartServlet extends AbstractHttpServlet {
  @Override
  void doGet(HttpServletRequest req, HttpServletResponse resp) {
    // Get the active span
    final Span span = GlobalTracer.get().activeSpan();
    if (span != null) {
      // customer_id -> 254889
      span.setTag("customer.id", customer_id);
    }

    // [...]
  }
}

The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with set_tag method.

from ddtrace import tracer

@app.route('/shopping_cart/<int:customer_id>')
@login_required
def shopping_cart(customer_id):
    # Get the active span
    current_span = tracer.current_span()
    if current_span:
        # customer_id -> 254889
        current_span.set_tag('customer.id', customer_id)

    # [...]

The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with set_tag method.

require 'ddtrace'

# get '/shopping_cart/:customer_id', to: 'shopping_cart#index'
class ShoppingCartController < ApplicationController
  # GET /shopping_cart
  def index
    # Get the active span
    current_span = Datadog.tracer.active_span
    # customer_id -> 254889
    current_span.set_tag('customer.id', params.permit([:customer_id])) unless current_span.nil?

    # [...]
  end

  # POST /shopping_cart
  def create
    # [...]
  end
end

The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with SetTag method.

package main

import (
    muxtrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/gorilla/mux"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    // Get the active span from a Go Context
    if span, ok := tracer.SpanFromContext(r.Context()); ok {
      // customer_id -> 254889
      span.SetTag("customer.id", vars["customerID"])
    }

    // [...]
}

func main() {
    tracer.Start(tracer.WithServiceName("web-store"))
    defer tracer.Stop()
    // Use auto-instrumentation
    mux := muxtrace.NewRouter()
    mux.HandleFunc("/shopping_cart/{customerID}", handler)
    http.ListenAndServe(":8080", mux)
}

The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with setTag method.

app.get('/shopping_cart/:customer_id', (req, res) => {
  // Get the active span
  const span = tracer.scope().active()
  if (span !== null) {
    // customer_id -> 254889
    span.setTag('customer.id', req.params.customer_id)
  }

  // [...]
})

Add tags directly to a Datadog.Trace.Span object by calling Span.SetTag(). For example:

public class ShoppingCartController : Controller
{
    private IShoppingCartRepository _shoppingCartRepository;

    [HttpGet]
    public IActionResult Index(int customerId)
    {
        // Access the active scope through the global tracer (can return null)
        var scope = Tracer.Instance.ActiveScope;

        if (scope != null)
        {
            // Add a tag to the span for use in the datadog web UI
            scope.Span.SetTag("customer.id", customerId.ToString());
        }

        var cart = _shoppingCartRepository.Get(customerId);

        return View(cart);
    }
}

Note: Datadog.Trace.Tracer.Instance.ActiveScope returns null if there is no active span.

The Datadog UI uses tags to set span level metadata. Custom tags may be set for auto-instrumentation by grabbing the active span from the global tracer and setting a tag with setTag method.

<?php
  namespace App\Http\Controllers;

  use DDTrace\GlobalTracer;

  class ShoppingCartController extends Controller
  {
      public shoppingCartAction (Request $request) {
          // Get the currently active span
          $span = GlobalTracer::get()->getActiveSpan();
          if (null !== $span) {
              // customer_id -> 254889
              $span->setTag('customer_id', $request->get('customer_id'));
          }

          // [...]
      }
  }
?>
You might have to wait a few minutes between deploying your updated code and seeing the new tags in the Datadog UI

Leverage the Datadog UI to search for your custom span tags

2) Go to the Services page and click on the service that you added tags to. Scroll down and click on the specific resource where the tag was added in the Resource table. Scroll down to the Traces table

The Trace table shows you both the overall latency distribution of all traces in the current scope (service, resource and timeframe) and links to individual traces. You can sort this table by duration or error code to easily identify erroneous operation or opportunities for optimization.

3) Click into one of your traces

In this view you can see the flamegraph on top and the additional information windows beneath it. The Datadog flamegraph allows you to have an at a glance view of the duration and status of every logical unit (span) that impacts a request. The flamegraph is fully interactive and you can pan it (by dragging) or zoom in and out (by scrolling). Clicking on any span provides more information about that span in particular in the bottom part of the view.

The bottom part of the view includes additional information about the trace or any selected span. Here you can see all default tags as well as the ones you manually include. In addition to these, you can also switch to view associated Host and Log information.

In order to enable Logs in this view you need to have Logs collection enabled and then to connect Logs and Traces

Leverage your custom span tags with App Analytics

This section assumes that you have enabled App Analytics

4) Navigate to the Trace Search page.

The Trace Search page allows you to identify specific Traces and Analyzed Spans you are interested in. Here you can filter by time a set of default tags (such as Env,Service, Resource and many more).

5) Find a trace that has the new tag. To do this use the facet explorer on the left to find the Resource name you set at the beginning of this guide and click into one of the rows you see there.

6) Find the new tag that you added to the trace. Click on it and select Create facet for @[your facet name] (remember, this is customer_id in our example)

You can now determine the displayed name of your facet and where to place it in the facet explorer.

You should now be able to see the facet you created in the Facet Explorer. The fastest way to find it is by using the Search facets box.

6) Navigate to the App Analytics page

The App Analytics page is a visual query building tool that allows you to conduct an investigation into your traces with infinite cardinality. It relies on facets to filter and scope the query, read more in the App Analytics overview.

7) Choose the service you’ve been working on from the service facet list, choose Error from the status facet and select customer_id (or any other tags you added to your spans) from the group by field.

8) Remove Error from the query, change the count * measure to Duration and change the graph type to Top List.

You can now see the customers that have the slowest average requests. Note: If you’d like to make sure your customers never pass a certain threshold of performance, you can export this query to a monitor, alternatively, you can save this visualization to a dashboard and keep an eye over it over time.

Finally, you can also see all the traces relevant to your query by clicking the visualization and selecting View traces.

Further Reading