APM collects metrics on your app’s performance at four levels of granularity: services, resources, traces, and spans level.
A service is a set of processes that do the same job.
For instance, a simple web application may consist of two services:
webappservice and a single
While a more complex environment may break it out into 6 services:
Note: Services must have a type attached, APM automatically assigns services one of four types: web, database, cache, custom.
A Resource is a particular action for a service.
/user/homeor a handler function like
web.user.home(often referred to as “routes” in MVC frameworks).
SELECT * FROM users WHERE id = ?.
Resources should be grouped together under a canonical name, like
/user/home rather than have
/user/home?id=200 as separate resources. APM automatically assigns names to your resources; however you can also name them explicitly. See instructions for: Go, Java, Python, Ruby.
These resources can be found after clicking on a particular service.
When a resource such as URL or SQL query cannot be aggregated, it significantly increases the cardinality of resources (ie. the number of unique aggregate resources) to be stored in Datadog.
Having a very high cardinality of resources makes Datadog less usable:
As a result we have a hard limit on the cardinality of resources for a given service.
A trace is used to track the time spent by an application processing a single operation, each trace consists of one or more spans.
For example, a trace can be used to track the entire time spent processing a complicated web request. Even though the request may require multiple resources and machines to handle the request, all of these function calls and sub-requests would be encapsulated within a single trace.
A span represents a logical unit of work in the system.
Spans are associated with a service and optionally a resource. Each span consists of a start time, a duration, and optional tags. For example, a span can describe the time spent on a distributed call on a separate machine, or the time spent in a small component within a larger operation. Spans can be nested within each other, and in those instances will have a parent-child relationship.
Additional helpful documentation, links, and articles: