The APM UI provides many tools to troubleshoot application performance and correlate it throughout the product, which helps you find and resolve issues in highly distributed systems.
| Concept | Description |
|---|---|
| Service | Services are the building blocks of modern microservice architectures - broadly a service groups together endpoints, queries, or jobs for the purposes of scaling instances. |
| Resource | Resources represent a particular domain of a customer application - they are typically an instrumented web endpoint, database query, or background job. |
| Trace | A trace is used to track the time spent by an application processing a request and the status of this request. Each trace consists of one or more spans. |
| Span | A span represents a logical unit of work in a distributed system for a given time period. Multiple spans construct a trace. |
| Trace metrics | Trace metrics are automatically collected and kept with a 15-month retention policy similar to other Datadog metrics. They can be used to identify and alert on hits, errors, or latency. |
| Trace Search & Analytics | Trace Search & Analytics is used to filter APM events by user-defined tags (customer_id, error_type, app_name, etc.) or infrastructure tags. |
| APM event | APM events represent 100% throughput of a request and can be used to search, query, and monitor in Trace Search & Analytics. |
| Span tags | Tag spans in the form of key-value pairs to correlate a request in the Trace View or filter in Trace Search & Analytics. |
After instrumenting your application, the Services List is your main landing page for APM data.
Services are the building blocks of modern microservice architectures - broadly a service groups together endpoints, queries, or jobs for the purposes of scaling instances. Some examples:
The screenshot below is a microservice distributed system for an e-commerce site builder. There’s a web-store, ad-server, payment-db, and auth-service all represented as services in APM.
All services can be found in the Service List and visually represented on the Service Map. Each service has its own Service page where trace metrics like throughput, latency, and error rates can be viewed and inspected. Use these metrics to create dashboard widgets, create monitors, and see the performance of every resource such as a web endpoint or database query belonging to the service.
Resources represent a particular domain of a customer application. They could typically be an instrumented web endpoint, database query, or background job. For a web service, these resources can be dynamic web endpoints that are grouped by a static span name - web.request. In a database service, these would be database queries with the span name db.query. For example the web-store service has automatically instrumented resources - web endpoints - which handle checkouts, updating_carts, add_item, etc. Each resource has its own Resource page with trace metrics scoped to the specific endpoint. Trace metrics can be used like any other Datadog metric - they are exportable to a dashboard or can be used to create monitors. The Resource page also shows the span summary widget with an aggregate view of spans for all traces, latency distribution of requests, and traces which show requests made to this endpoint.
A trace is used to track the time spent by an application processing a request and the status of this request. Each trace consists of one or more spans. During the lifetime of the request, you can see distributed calls across services (because a trace-id is injected/extracted through HTTP headers), automatically instrumented libraries, and manual instrumentation using open-source tools like OpenTracing in the flamegraph view. In the Trace View page, each trace collects information that connects it to other parts of the platform, including connecting logs to traces, adding tags to spans, and collecting runtime metrics.
A span represents a logical unit of work in the system for a given time period. Each span consists of a span.name, start time, duration, and span tags. For example, a span can describe the time spent on a distributed call on a separate machine, or the time spent in a small component within a larger request. Spans can be nested within each other, which creates a parent-child relationship between the spans.
For the example below, the span rack.request is the entry-point span of the trace. This means the web-store service page is displaying resources that consist of traces with an entry-point span named rack.request. The example also shows the tags added application side (merchant.name, merchant.tier, etc). These user-defined tags can be used to search and analyze APM data in Trace Search & Analytics.
Trace metrics are automatically collected and kept at a 15-month retention policy similar to any other Datadog metric. They can be used to identify and alert on hits, errors, or latency. Trace metrics are tagged by the host receiving traces along with the service or resource. For example, after instrumenting a web service trace metrics are collected for the entry-point span web.request in the Metric Summary.
Trace metrics can be exported to a dashboard from the Service or Resource page. Additionally, trace metrics can be queried from an existing dashboard.
Trace metrics are useful for monitoring. APM monitors can be set up on the New Monitors, Service, or Resource page. A set of suggested monitors is available on the Service, or Resource page.
Trace Search & Analytics is used to filter APM Events by user-defined tags (customer_id, error_type, app_name, etc.) or infrastructure tags. This allows deep exploration of the web requests flowing through your service along with being able to search, graph, and monitor on 100% throughput of hits, errors, and latency. This feature can be enabled with automatic configuration.
APM events represent 100% throughput of a request and can be used to search, query, and monitor in Trace Search & Analytics by the tags included on the span. After enabling Trace Search & Analytics, the tracing client analyzes an entry-point span for web services by default, with the ability to configure additional services in your application. For example, a Java service with 100 requests generates 100 APM events from its servlet.request spans. If you set DD_TRACE_ANALYTICS_ENABLED=true the web-store service analyzes all rack.request spans and makes them available in Trace Search & Analytics. For this example, you can graph the top 10 merchants highest latency in the 99th percentile. merchant_name is a user defined tag that was applied to the span in the application.
Tag spans in the form of key-value pairs to correlate a request in the Trace View or filter in Trace Search & Analytics. Tags can be added to a single span or globally to all spans. For the example below, the requests (merchant.store_name, merchant.tier, etc.) have been added as tags to the span.
To get started tagging spans in your application, check out this walkthrough.
After a tag has been added to a span, search and query on the tag in Trace Search & Analytics by clicking on the tag to add it as a facet. Once this is done, the value of this tag is stored for all new traces and can be used in the search bar, facet panel, and trace graph query.
Additional helpful documentation, links, and articles:
