Overview
Centralizing logs from various technologies and applications can generate tens or hundreds of different attributes in a Log Management environment, especially when many teams are working within the same environment.
For instance, a client IP may have various log attributes, such as clientIP, client_ip_address, remote_address, client.ip, and so on. The execution time of a request may be referred to as exec_time, request_latency, request.time_elapsed, and so on.
Use attributes and aliasing to unify your Logs environment.
Attribute types and aliasing
Attributes prescribe logs facets and tags, which are used for filtering and searching in Log Explorer.
Reserved attributes are automatically ingested.
Standard attributes are the backbone of the naming convention for your organization. There is a default set of standard attributes available in the app. However, this list can be customized to create a naming convention for your team.
Use Aliasing once you’ve implemented a naming convention with standard attributes or if you’re trying to create a unique standard facet from multiple log sources. For example, follow the clients most impacted by latencies on a hybrid Apache and Amazon Cloud Front infrastructure, using the standard Network Client IP facet alongside the standard duration. Aliasing allows implementation of a naming convention without having to change a team’s technical stack.
Reserved attributes
Below is a list of reserved attributes that are automatically ingested with logs.
Note: If you’re also collecting traces or metrics, it is recommended to configure unified service tagging. This configuration ties Datadog telemetry together through the use of three standard tags: env, service, and version. Refer to the dedicated unified service tagging documentation for more information.
| Attribute | Description |
|---|
host | The name of the originating host as defined in metrics. Datadog automatically retrieves corresponding host tags from the matching host in Datadog and applies them to your logs. The Agent sets this value automatically. |
source | This corresponds to the integration name, the technology from which the log originated. When it matches an integration name, Datadog automatically installs the corresponding parsers and facets. For example, nginx, postgresql, and so on. |
status | This corresponds to the level/severity of a log. It is used to define patterns and has a dedicated layout in the Datadog Log UI. |
service | The name of the application or service generating the log events. It is used to switch from Logs to APM, so make sure you define the same value when you use both products. |
trace_id | This corresponds to the Trace ID used for traces. It is used to correlate your log with its trace. |
message | By default, Datadog ingests the value of the message attribute as the body of the log entry. That value is then highlighted and displayed in Live Tail, where it is indexed for full text search. |
Standard attributes
Log integrations natively rely on a default set of standard attributes.
The standard attribute table comes with a set of predefined standard attributes. You can append that list with your own attributes, and edit or delete existing standard attributes.
Create a new standard attribute
Admin users can curate the standard attribute list:
- Navigate to the standard attribute configuration page.
- Click New Standard Attribute.
- Define the standard attribute:
Path: The path of the standard attributes as you would find it in your JSON (for example, network.client.ip).Type: (string, integer, double, boolean): The type of the attribute, which is used to cast elements of the remapping list.Description: Human-readable description of the attribute.- (Optional)
Remapping list: Comma-separated list of non-compliant attributes that should be remapped to the standard attribute.
Default standard attribute list
See the full list of Log Management Default Standard Attributes, which is split into the functional domains:
| Standard Attribute | Description |
|---|
| Network/communications | These attributes are related to the data used in network communication. All fields and metrics are prefixed by network. |
| Geolocation | These attributes are related to the geolocation of IP addresses used in network communication. All fields are prefixed by network.client.geoip or network.destination.geoip. |
| HTTP Requests | These attributes are related to data commonly used in HTTP requests and accesses. All attributes are prefixed by http. Typical integrations relying on these attributes include Apache, Rails, AWS CloudFront, web applications servers, and so forth. URL details attributes are prefixed by http.url_details. These attributes provide details about the parsed parts of the HTTP URL. They are generated by the URL parser. |
| Source code | These attributes are related to data used when a log or an error is generated using a logger in a custom application. All attributes are prefixed either by logger or error. Typical integrations relying on these attributes are Java, Node.js, .NET, Golang, Python, and so on. |
| Database | Typical integrations relying on these attributes are Cassandra, MySQL, RDS, Elasticsearch, and so on. |
| Performance | These attributes are related to performance metrics. Datadog recommends remapping any durations within your logs on this attribute since they are displayed and used as a default measure for trace search. |
| User related attributes | All attributes and measures are prefixed by usr. |
| Syslog and log shippers | These attributes are related to the data added by a syslog or a log-shipper agent. All fields and metrics are prefixed by syslog. Integrations that rely on these include Rsyslog, NxLog, Syslog-ng, Fluentd, and Logstash. |
| DNS | All attributes and measures are prefixed by dns. |
| Events | All attributes are prefixed by evt. |
Aliasing
Creating an alias for a source attribute that maps to a destination attribute allows logs to carry both the source and destination attributes.
Users can interact with either the aliased (source) or standard (destination) faceted attribute. However, users are encouraged to use the standard facet rather than the aliased one. This provides guidance towards the naming convention, and discourages users from building assets (such as saved views or dashboards) based on non-standard content.
Additional details regarding aliasing:
- Aliasing happens after logs are processed by pipelines. Any extracted or processed attribute can be used as a source for aliasing.
- Datadog enforces the type of an aliased attribute. If this is not possible, the aliasing is skipped.
- In the case of a log that already carries the destination attribute, aliasing overrides the value of that log.
- For a standard attribute to which multiple attributes are aliased, if a log carries several of these source attributes, only one of these source attributes is aliased.
- Any updates or additions to standard attributes are only applied to newly ingested logs.
- Standard attributes cannot be aliased.
- Attributes can only be aliased to standard attributes.
- To respect the JSON structure of logs, it is not possible to have one standard attribute as the child of another (for example,
user and user.name cannot both be standard attributes).
See Alias Facets for additional information.
Further Reading
Additional helpful documentation, links, and articles:
