Add span tags and filter and group your application performance

7 minutes to complete

Datadog APM allows you to customize your traces to include any additional information you might need to maintain observability into your business. You can use this to identify a spike in the throughput of a certain enterprise customer, or the user suffering the highest latency, or to pinpoint the database shard generating the most errors.

In this example, a customer ID is added to traces allowing the customers that have the slowest performance to be identified. Customization of traces is based on tags that seamlessly integrate APM with the rest of Datadog and come in the form of key:value pairs of metadata added to spans.

Instrument your code with custom span tags

1. Follow the example to get your code instrumented.

Depending on the programming language you are you using, you’ll need to set the tags to add to your spans differently.

Note: take note of the service and resource names you are working on, these will come in handy later. In this example, the service is the Ruby server web-store and the resource (endpoint) is ShoppingCartController#checkout.

import io.opentracing.Span;
import io.opentracing.util.GlobalTracer;

@WebServlet
class ShoppingCartServlet extends AbstractHttpServlet {
  @Override
  void doGet(HttpServletRequest req, HttpServletResponse resp) {
    // Get the active span
    final Span span = GlobalTracer.get().activeSpan();
    if (span != null) {
      // customer_id -> 254889
      span.setTag("customer.id", customer_id);
    }

    // [...]
  }
}

from ddtrace import tracer

@app.route('/shopping_cart/<int:customer_id>')
@login_required
def shopping_cart(customer_id):
    # Get the active span
    current_span = tracer.current_span()
    if current_span:
        # customer_id -> 254889
        current_span.set_tag('customer.id', customer_id)

    # [...]

require 'ddtrace'

# get '/shopping_cart/:customer_id', to: 'shopping_cart#index'
class ShoppingCartController < ApplicationController
  # GET /shopping_cart
  def index
    # Get the active span and set customer_id -> 254889
    Datadog::Tracing.active_span&.set_tag('customer.id', params.permit([:customer_id]))

    # [...]
  end

  # POST /shopping_cart
  def create
    # [...]
  end
end

package main

import (
    muxtrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/gorilla/mux"

    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func handler(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    // Get the active span from a Go Context
    if span, ok := tracer.SpanFromContext(r.Context()); ok {
      // customer_id -> 254889
      span.SetTag("customer.id", vars["customerID"])
    }

    // [...]
}

func main() {
    tracer.Start(tracer.WithServiceName("web-store"))
    defer tracer.Stop()
    // Use auto-instrumentation
    mux := muxtrace.NewRouter()
    mux.HandleFunc("/shopping_cart/{customerID}", handler)
    http.ListenAndServe(":8080", mux)
}

app.get('/shopping_cart/:customer_id', (req, res) => {
  // Get the active span
  const span = tracer.scope().active()
  if (span !== null) {
    // customer_id -> 254889
    span.setTag('customer.id', req.params.customer_id)
  }

  // [...]
})

public class ShoppingCartController : Controller
{
    private IShoppingCartRepository _shoppingCartRepository;

    [HttpGet]
    public IActionResult Index(int customerId)
    {
        // Access the active scope through the global tracer (can return null)
        var scope = Tracer.Instance.ActiveScope;

        if (scope != null)
        {
            // Add a tag to the span for use in the datadog web UI
            scope.Span.SetTag("customer.id", customerId.ToString());
        }

        var cart = _shoppingCartRepository.Get(customerId);

        return View(cart);
    }
}

<?php
  namespace App\Http\Controllers;

  class ShoppingCartController extends Controller
  {
      public shoppingCartAction (Request $request) {
          // Get the currently active span
          $span = \DDTrace\active_span();
          if (null !== $span) {
              // customer_id -> 254889
              $span->meta['customer_id'] = $request->get('customer_id');
          }

          // [...]
      }
  }
?>

Leverage the Datadog UI to search for your custom span tags

2. Go to the Services page and click on the service that you added tags to. Scroll down and click on the specific resource where the tag was added in the Resource table. Scroll down to the Traces table

The Trace table shows you both the overall latency distribution of all traces in the current scope (service, resource, and timeframe) and links to individual traces. You can sort this table by duration or error code to identify erroneous operation or opportunities for optimization.

3. Click into one of your traces

In this view you can see the flame graph on top and the additional information windows beneath it. The Datadog flame graph allows you to have an at a glance view of the duration and status of every logical unit (span) that impacts a request. The flame graph is fully interactive and you can pan it (by dragging) or zoom in and out (by scrolling). Clicking on any span provides more information about that span in particular in the bottom part of the view.

The bottom part of the view includes additional information about the trace or any selected span. Here you can see all default tags as well as the ones you manually include. In addition to these, you can also switch to view associated Host and Log information.

Leverage your custom span tags with Analytics

4. Navigate to the Trace Explorer page.

The Trace Search page allows you to identify specific Traces and Indexed Spans you are interested in. Here you can filter by time a set of default tags (such as Env,Service, Resource and many more).

5. Find a trace that has the new tag. To do this use the facet explorer on the left to find the Resource name you set at the beginning of this guide and click into one of the rows you see there.

6. Find the new tag that you added to the trace. Click on it and select Create facet for @[your facet name] (remember, this is customer_id in our example)

You can now determine the displayed name of your facet and where to place it in the facet explorer.

You should now be able to see the facet you created in the Facet Explorer. The fastest way to find it is by using the Search facets box.

6. Navigate to the Analytics page

The Analytics page is a visual query building tool that allows you to conduct an investigation into your traces with infinite cardinality. It relies on facets to filter and scope the query, read more in the Trace Explorer overview.

7. Choose the service you’ve been working on from the service facet list, choose Error from the status facet and select customer_id (or any other tags you added to your spans) from the group by field.

8. Remove Error from the query, change the count * measure to Duration and change the graph type to Top List.

You can now see the customers that have the slowest average requests. Note: If you’d like to make sure your customers never pass a certain threshold of performance, you can export this query to a monitor, alternatively, you can save this visualization to a dashboard and keep an eye over it over time.

Finally, you can also see all the traces relevant to your query by clicking the visualization and selecting View traces.

Further Reading

Additional helpful documentation, links, and articles:

Alert on anomalous p99 latency of a database service3 MINS Compare a service's latency to the previous week2 MINS Create a Dashboard to track and correlate APM metrics4 MINS Debug the slowest trace on the slowest endpoint of a web service3 MINS All guides