Overview
To collect logs from files, configure the filelog receiver in your Datadog Exporter.
For more information, see the OpenTelemetry project documentation for the filelog receiver.
Setup
For a collector deployed on the same host as the log files to be collected, specify the paths of the log files to collect in your Collector configuration:
receivers:
filelog:
include_file_path: true
poll_interval: 500ms
include:
- /var/log/*/app.log
operators:
- type: json_parser
# Layout must match log timestamp format. If this section is removed, timestamp will correspond to the time of log intake by Datadog.
- type: time_parser
parse_from: attributes.time
layout: '%Y-%m-%dT%H:%M:%S%z'
Add the following lines to values.yaml:
presets:
logsCollection:
enabled: true
includeCollectorLogs: true
The filelog receiver needs access to the file paths. The preset mounts the necessary volumes to the collector container for /var/log/pods and collects all logs from /var/log/pods/*/*/*.log. See Important components for Kubernetes for a full list of settings set by the preset.
Collector configuration sets up a list of operators to parse the logs based on different formats:
filelog:
include:
- /var/log/pods/*/*/*.log
exclude:
- /var/log/pods/abc/*.log
operators:
- type: json_parser
- type: trace_parser
trace_id:
parse_from: attributes.trace_id
span_id:
parse_from: attributes.span_id
trace_flags:
parse_from: attributes.trace_flags
- type: time_parser
parse_from: attributes.time
layout: '%Y-%m-%dT%H:%M:%S%z'
Data collected
Logs from the configured files.
Full example configuration
For a full working example configuration with the Datadog exporter, see logs.yaml.
Example logging output
ResourceLog #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1
Resource attributes:
-> k8s.container.name: Str(loadgenerator)
-> k8s.namespace.name: Str(otel-staging)
-> k8s.pod.name: Str(opentelemetry-demo-loadgenerator-d8c4d699d-ztt98)
-> k8s.container.restart_count: Str(1)
-> k8s.pod.uid: Str(92bf09ed-0db9-4f69-a9d6-1dadf12e01aa)
-> k8s.pod.ip: Str(192.168.55.78)
-> cloud.provider: Str(aws)
-> cloud.platform: Str(aws_ec2)
-> cloud.region: Str(us-east-1)
-> cloud.account.id: Str(XXXXXXXXX)
-> cloud.availability_zone: Str(us-east-1c)
-> host.id: Str(i-0368add8e328c28f7)
-> host.image.id: Str(ami-08a2e6a8e82737230)
-> host.type: Str(m5.large)
-> host.name: Str(ip-192-168-53-115.ec2.internal)
-> os.type: Str(linux)
-> k8s.daemonset.uid: Str(6d6fef61-d4c7-4226-9b7b-7d6b893cb31d)
-> k8s.daemonset.name: Str(opentelemetry-collector-agent)
-> k8s.node.name: Str(ip-192-168-53-115.ec2.internal)
-> kube_app_name: Str(opentelemetry-collector)
-> kube_app_instance: Str(opentelemetry-collector)
-> k8s.pod.start_time: Str(2023-11-20T12:53:23Z)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2023-11-20 13:02:04.332021519 +0000 UTC
Timestamp: 2023-11-20 13:01:46.095736502 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str( return wrapped_send(self, request, **kwargs))
Attributes:
-> log.file.path: Str(/var/log/pods/otel-staging_opentelemetry-demo-loadgenerator-d8c4d699d-ztt98_92bf09ed-0db9-4f69-a9d6-1dadf12e01aa/loadgenerator/1.log)
-> time: Str(2023-11-20T13:01:46.095736502Z)
-> logtag: Str(F)
-> log.iostream: Str(stderr)
Trace ID:
Span ID:
Flags: 0
