Inferred Service dependencies

Opt in to the private beta!

Inferred service dependencies are in private beta. To request access, complete the form.

Request Access

Overview

Datadog can automatically discover the dependencies for an instrumented service, such as a database, a queue, or a third-party API, even if that dependency hasn’t been instrumented yet. By analyzing outbound requests from your instrumented services, Datadog infers the presence of these dependencies and collects associated performance metrics.

With the new inferred entities experience, you can filter Service Catalog entries by entity type, such as database, queue, or third-party API. This allows you to better visualize service dependencies using the Service Page dependency map and APM features.

To determine the names and types of the inferred service dependencies, Datadog uses standard span attributes and maps them to peer.* attributes. For the full list of peer.* attributes, see Inferred service dependencies nomenclature. Inferred external APIs use the default naming scheme net.peer.name. For example, api.stripe.com, api.twilio.com, us6.api.mailchimp.com. Inferred databases use the default naming scheme db.instance.

If you’re using the Go, Java, NodeJS, PHP, .NET, or Ruby tracer, you can customize the default names for inferred entities.

Note: If you configure monitors, dashboards, or notebooks for a given inferred service during the beta, you may need to update them if the naming scheme changes. Read more about migration steps in the opt-in instructions.

Service page Dependency map

Use the dependency map to visualize service-to-service communication and gain insight into system components such as databases, queues, and third-party dependencies. You can group dependencies by type and filter by Requests, Latency, or Errors to identify slow or failing connections.

Service page service dependency map

Opt in

Only go through migration steps once Datadog support confirmed the feature is enabled for you on the Datadog side.

To opt in, Datadog recommends you adjust your:

Datadog Agent configuration

Requirements:

  • Datadog Agent version >= 7.50.3.

Set the following environment variables in your Datadog Agent launch configuration:

datadog.yaml

DD_APM_COMPUTE_STATS_BY_SPAN_KIND=true 
DD_APM_PEER_TAGS_AGGREGATION=true
DD_APM_PEER_TAGS='["_dd.base_service","amqp.destination","amqp.exchange","amqp.queue","aws.queue.name","aws.s3.bucket","bucketname","cassandra.keyspace","db.cassandra.contact.points","db.couchbase.seed.nodes","db.hostname","db.instance","db.name","db.namespace","db.system","grpc.host","hostname","http.host","http.server_name","messaging.destination","messaging.destination.name","messaging.kafka.bootstrap.servers","messaging.rabbitmq.exchange","messaging.system","mongodb.db","msmq.queue.path","net.peer.name","network.destination.name","peer.hostname","peer.service","queuename","rpc.service","rpc.system","server.address","streamname","tablename","topicname"]'

Helm

Include the same set of environment variables in your values.yaml file.

OpenTelemetry Collector

Minimum version recommended: opentelemetry-collector-contrib >= v0.95.0.

Example collector.yaml.

connectors:
  datadog/connector:
    traces:
      compute_stats_by_span_kind: true
      peer_tags_aggregation: true
      peer_tags: ["_dd.base_service","amqp.destination","amqp.exchange","amqp.queue","aws.queue.name","aws.s3.bucket","bucketname","db.cassandra.contact.points","db.couchbase.seed.nodes","db.hostname","db.instance","db.name","db.namespace","db.system","grpc.host","hostname","http.host","http.server_name","messaging.destination","messaging.destination.name","messaging.kafka.bootstrap.servers","messaging.rabbitmq.exchange","messaging.system","mongodb.db","msmq.queue.path","net.peer.name","network.destination.name","peer.hostname","peer.service","queuename","rpc.service","rpc.system","server.address","streamname","tablename","topicname"]

If your collector version is below v0.95.0, use an exporter configuration with the following peer_tags:

exporters:
  datadog:
    traces:
      compute_stats_by_span_kind: true
      peer_tags_aggregation: true
      peer_tags: ["_dd.base_service","amqp.destination","amqp.exchange","amqp.queue","aws.queue.name","aws.s3.bucket","bucketname","db.cassandra.contact.points","db.couchbase.seed.nodes","db.hostname","db.instance","db.name","db.namespace","db.system","grpc.host","hostname","http.host","http.server_name","messaging.destination","messaging.destination.name","messaging.kafka.bootstrap.servers","messaging.rabbitmq.exchange","messaging.system","mongodb.db","msmq.queue.path","net.peer.name","network.destination.name","peer.hostname","peer.service","queuename","rpc.service","rpc.system","server.address","streamname","tablename","topicname"]   

APM tracing libary configuration

The following steps introduce a breaking change: Datadog will change the way service names are captured by default. Refer to Global default service naming migration, to determine if you need to take any migration actions.

The minimum Java tracer version required is 1.16.0. Regular updates to the latest version are recommended to access changes and bug fixes.

Download the latest Java tracer version.

To opt in, add the following environment variables or system properties to your tracer settings:

Environment variableSystem property
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=true-Ddd.trace.remove.integration-service-names.enabled=true

Remove the following settings from your configuration:

Environment variableReason for removal
DD_SERVICE_MAPPINGAll service names default to DD_SERVICE.
DD_TRACE_SPLIT_BY_TAGSInferred services are automatically displayed with the introduction of the peer.service tag.
DD_TRACE_DB_CLIENT_SPLIT_BY_INSTANCEDB instances are inferred based on the on the peer.service tag.

The minimum Go tracer version required is v1.52.0. Regular updates to the latest version are recommended to access changes and bug fixes.

To opt in, add the following environment variables or system properties to your tracer settings:

Environment variableSystem property
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=trueWithGlobalServiceName(true)

The minimum NodeJS tracer versions required are 2.44.0, 3.31.0, or 4.10.0. Regular updates to the latest version are recommended to access changes and bug fixes.

To opt in, add the following environment variables or system properties to your tracer settings:

Environment variableSystem property
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=truespanRemoveIntegrationFromService=true

The minimum PHP tracer version required is 0.90.0. Regular updates to the latest version are recommended to access changes and bug fixes.

To opt in, add the following environment variables or system properties to your tracer settings:

Environment variableSystem property
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=truedatadog.trace.remove_integration_service_names_enabled=true

The minimum .NET tracer version required is v2.35.0. Regular updates to the latest version are recommended to access changes and bug fixes.

To opt in, add the following environment variable to your tracer settings or system properties:

  • DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=true

The minimum Python tracer version required is v1.16.0. Regular updates to the latest version are recommended to access changes and bug fixes.

To opt in, add the following environment variables to your tracer settings or system properties:

Add the following environment variables to your tracer settings or system properties:

  • DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=true

As of tracer version v1.16.0 all libraries are supported except for Boto2.

The minimum Ruby tracer version required is v1.13.0. Regular updates to the latest version are recommended to access changes and bug fixes.

To opt in, add the following environment variables to your tracer settings or system properties:

  • DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=true

The new nomenclature: What is changing

List of newly introduced peer.* tags

Global default service naming migration

When you enable the DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED environment variable, it improves how service-to-service connections and inferred services are represented in Datadog visualizations, across all supported tracing library languages and integrations.

Previously, some tracing libraries included the name of the associated integration in service name tagging. For example, .NET tagged gRPC calls as service:<DD_SERVICE>-grpc-client while Python tagged them as service:grpc-client. With this option enabled, all supported tracing libraries tag spans from the downstream services with the calling service’s name, service:<DD_SERVICE>, thereby providing a global default service name.

_BeforeAfter
Service nameservice:my-service-grpc-client or service:grpc-clientservice:myservice
additional peer.* attributesNo peer.* tags set@peer.service:otherservice (otherservice being the name of the remote service being called with gRPC)

Similarly, for a span representing a call to a mySQL database:

_BeforeAfter
Service nameservice:my-service-mysql or service:mysqlservice:myservice
additional peer.* attributesNo peer.* tags set@peer.db.name:user-db, @peer.db.system:mysql

Consequently, if you have existing:

  • APM metrics
  • APM custom span metrics
  • Trace analytics
  • Retention filters
  • Sensitive data scans
  • Monitors, dashboards, or notebooks

And these target similar service names, update those items to use the global default service tag (service:<DD_SERVICE>) instead.

Further reading