Instrument a custom method to get deep visibility into your business logic
Datadog's Research Report: The State of Serverless Report: The State of Serverless

Instrument a custom method to get deep visibility into your business logic

8 minutes to complete

To provide you with deep visibility into your business logic, Datadog APM allows you to customize the spans that make up your traces based on your needs and implementation. This empowers you to trace any method in your codebase and even specific components within methods. You can use this to optimize and monitor critical areas of your application at the granularity that works for you.

Datadog instruments many frameworks out-of-the-box, such as web services, databases, and caches, and enables you to instrument your own business logic to have the exact visibility you need. By creating spans for methods, you can optimize timing and track errors using the APM flame graph and monitors.

Instrumenting your code

Follow the example to get your code instrumented.

These examples walk through tracing the entire BackupLedger.write method to measure its execution time and status. BackupLedger.write is an action that saves the current state of a transaction ledger in memory before making a call to a payments database to post a new customer charge. This happens when the charge endpoint of the payments service is hit:

The http.request POST /charge/ span is taking a lot of time without having any direct child spans. This is a clue that this request requires further instrumentation to gain better insights into its behavior. Depending on the programming language you are using, you need to decorate your functions differently:

In Java, Datadog APM allows you to instrument your code to generate custom spans—either by using method decorators, or by instrumenting specific code blocks.

Instument a method with a decorator:

This example adds a span to the BackupLedger.write method, which adds new rows to a transaction ledger. One span is added to track all posted transactions as a single unit.

import datadog.trace.api.Trace

public class BackupLedger {

  // Use @Trace annotation to trace custom methods
  @Trace
  public void write(List<Transaction> transactions) {
    for (Transaction transaction : transactions) {
      ledger.put(transaction.getId(), transaction);
    }

    // [...]
  }
}

Instrument a specific code block:

This example adds child spans to the BackupLedger.write span created above. This method adds a child span for every transaction in the ledger and a custom tag with the specific transaction ID.

import datadog.trace.api.Trace;
import io.opentracing.Scope;
import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;

public class BackupLedger {

  // Use `@Trace` annotation to trace custom methods
  @Trace
  public void write(List<Transaction> transactions) {
    for (Transaction transaction : transactions) {
      // Use `GlobalTracer` to trace blocks of inline code
      Tracer tracer = GlobalTracer.get();
      try (Scope scope = tracer.buildSpan("BackupLedger.persist").startActive(true)) {
        // Add custom metadata to the span
        scope.span().setTag("transaction.id", transaction.getId());
        ledger.put(transaction.getId(), transaction);
      }
    }

    // [...]
  }
}

In Python, Datadog APM allows you to instrument your code to generate custom spans—either by using method decorators, or by instrumenting specific code blocks.

Instument a method with a decorator:

This example adds a span to the BackupLedger.write method, which adds new rows to a transaction ledger. One span is added to track all posted transactions as a single unit.

from ddtrace import tracer

class BackupLedger:

    # Use `tracer.wrap` decorator to trace custom methods
    @tracer.wrap()
    def write(self, transactions):
        for transaction in transactions:
            self.ledger[transaction.id] = transaction

        # [...]

Instrument a specific code block:

This example adds child spans to the BackupLedger.write span created above. This method adds a child span for every transaction in the ledger and a custom tag with the specific transaction ID.

from ddtrace import tracer

class BackupLedger:

    # Use `tracer.wrap` decorator to trace custom methods
    @tracer.wrap()
    def write(self, transactions):
        for transaction in transactions:
            # Use `tracer.trace` context manager to trace blocks of inline code
            with tracer.trace('BackupLedger.persist') as span:
                # Add custom metadata to the "persist_transaction" span
                span.set_tag('transaction.id', transaction.id)
                self.ledger[transaction.id] = transaction

        # [...]

In Ruby, Datadog APM allows you to instrument your code to generate custom spans by instrumenting specific code blocks.

This example creates a new span for the call to the BackupLedger.write method and a child span for every transaction posted to the ledger with a custom tag with the specific transaction ID.

require 'ddtrace'

class BackupLedger

  def write(transactions)
    # Use global `Datadog.tracer.trace` to trace blocks of inline code
    Datadog.tracer.trace('BackupLedger.write') do |method_span|
      transactions.each do |transaction|
        Datadog.tracer.trace('BackupLedger.persist') do |span|
          # Add custom metadata to the "persist_transaction" span
          span.set_tag('transaction.id', transaction.id)
          ledger[transaction.id] = transaction
        end
      end
    end

    # [...]
  end
end

In Go, Datadog APM allows you to instrument your code to generate custom spans by instrumenting specific code blocks.

This example creates a new span for every transaction posted to the ledger and adds a custom tag with the specific transaction ID to the span.

package ledger

import "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

// [...]

func (bl *BackupLedger) write(ctx context.Context, transactions []*Transaction) (err error) {
  // Trace the `write` function and capture the error if present
  span, ctx := tracer.StartSpanFromContext(ctx, "BackupLedger.write")
  defer func() {
    span.Finish(tracer.WithError(err))
  }()

  for _, t := range transactions {
    if err := bl.persistTransaction(ctx, t); err != nil {
      return err
    }
  }
  return nil
}

// persistTransaction is an inner function you may want to Trace. You can use the
// same approach as before because the `ctx` you pass down includes out-of-the-box span
// references to create a parent/child relationships.
func (bl *BackupLedger) persistTransaction(ctx context.Context, transaction *Transaction) error {
  id := transaction.ID
  span, _ := tracer.StartSpanFromContext(ctx, "BackupLedger.persist", tracer.Tag("transaction_id", id))
  defer span.Finish()

  if t, ok := bl.transactions[id]; ok {
    return errors.New("duplicate entry")
  }
  bl.transactions[id] = transaction
  return nil
}

In Node.js, Datadog APM allows you to instrument your code to generate custom spans by instrumenting specific code blocks.

This example creates a new span for the call to the BackupLedger.write method and a child span for every transaction posted to the ledger with a custom tag with the specific transaction ID.

const tracer = require('dd-trace')

function write (transactions) {
  // Use `tracer.trace` context manager to trace blocks of inline code
  tracer.trace('BackupLedger.write', () => {
    for (const transaction of transactions) {
      // Add custom metadata to the "persist_transaction" span
      span.setTag('transaction.id', transaction.id)
      this.ledger[transaction.id] = transaction
    }
  })

  // [...]
}

In .NET, Datadog APM allows you to instrument your code to generate custom spans by instrumenting specific code blocks.

This example creates a new span for every transaction posted to the ledger and adds a custom tag with the specific transaction ID to the span.

using Datadog.Trace;

public void Write(List<Transaction> transactions)
{
    // Use global tracer to trace blocks of inline code
    using (var scope = Tracer.Instance.StartActive("BackupLedger.write"))
    {
        foreach (var transaction in transactions)
        {
            using (var scope = Tracer.Instance.StartActive("BackupLedger.persist"))
            {
                // Add custom metadata to the span
                scope.Span.SetTag("transaction.id", transaction.Id);
                this.ledger[transaction.Id] = transaction;
            }
        }
    }

    // [...]
}

In PHP, Datadog APM allows you to instrument your code to generate custom spans—either by using method wrappers, or by instrumenting specific code blocks.

Instument a method with a wrapper:

This example adds a span to the BackupLedger.write method, which adds new rows to a transaction ledger. One span is added to track all posted transactions as a single unit by using the dd_trace() function.

<?php
  class BackupLedger {

    public function write(array $transactions) {
      foreach ($transactions as $transaction) {
        $this->transactions[$transaction->getId()] = $transaction;
      }

      # [...]
    }
  }

  // Use dd_trace() to trace custom methods
  dd_trace('BackupLedger', 'write', function () {
    $tracer = \DDTrace\GlobalTracer::get();
    $scope = $tracer->startActiveSpan('BackupLedger.write');
    dd_trace_forward_call();
    $scope->close();
    return $result;
  });
?>

Instrument a specific code block:

This example adds child spans to the BackupLedger.write span created above. This method adds a child span for every transaction in the ledger and a custom tag with the specific transaction ID.

<?php
  class BackupLedger {

    public function write(array $transactions) {
      foreach ($transactions as $transaction) {
        // Use global tracer to trace blocks of inline code
        $scope = \DDTrace\GlobalTracer::get()->startActiveSpan('BackupLedger.persist');

        // Add custom metadata to the span
        $scope->getSpan()->setTag('transaction.id', $transaction->getId());
        $this->transactions[$transaction->getId()] = $transaction;

        // Close the span
        $scope->close();
      }

      # [...]
    }
  }

  // Use dd_trace() to trace custom methods
  dd_trace('BackupLedger', 'write', function () {
    $tracer = \DDTrace\GlobalTracer::get();
    $scope = $tracer->startActiveSpan('BackupLedger.write');
    dd_trace_forward_call();
    $scope->close();
    return $result;
  });
?>

Leverage the Datadog UI to see your new custom spans

Now that you have instrumented your business logic, it’s time to see the results in the Datadog APM UI.

  1. Go to the Service List, identify the service you added custom spans to, and then go to the Service Page. On the service page, click on the specific resource you added, change the time filter to The past 15 minutes, and scroll down to the span summary table:

    You should now be able to find the new spans you’ve added

The span summary table provides aggregate information about the spans that make up your traces. Here you can identify spans that repeat an abnormal amount of times indicating some looping or database access inefficiency (like the n+1 issue).

  1. Scroll down to the Traces list and click into one of your traces.

You’ve now successfully added custom spans to your codebase, making them available on the flame graph and in App Analytics. This is the first step towards taking full advantage of Datadog’s tools. You can now add custom tags to your spans to make them even more powerful.

Further Reading