There are a number of common issues that can get in the way when sending new logs to Datadog via the log collector in the dd-agent. If you experience issues sending new logs to Datadog, this list helps you troubleshoot. If you continue to have trouble, contact Datadog support for further assistance.
The Datadog Agent sends its logs to Datadog over TCP using port 10516. If that connection is not available, logs fail to be sent and an error is recorded in the agent.log file to that effect.
You can manually test your connection using OpenSSL, GnuTLS, or another SSL/TLS client. For OpenSSL, run the following command:
If opening the port 10516 is not an option, it is possible to configure the Datadog Agent to send logs through HTTPS by adding the following in datadog.yaml:
The Datadog Agent only collects logs that have been written after it has started trying to collect them (whether it be tailing or listening for them). In order to confirm whether log collection has been successfully set up, make sure that new logs have been written.
The Datadog Agent does not run as root (and running as root is not recommended, as a general best practice). When you configure your Agent to tail log files for custom logs or for integrations, you need to take special care to ensure the Agent user has the correct access to the log files.
The default Agent user per operating system:
Operating system
Default Agent user
Linux
datadog-agent
MacOS
datadog-agent
Windows
ddagentuser
If the Agent does not have the correct permissions, you might see one of the following error messages when checking the Agent status:
The file does not exist.
Access is denied.
Could not find any file matching pattern <path/to/filename>, check that all its subdirectories are executable.
To fix the error, give the Datadog Agent user read and execute permissions to the log file and subdirectories.
Note: Make sure that these permissions are correctly set in your log rotation configuration. Otherwise, on the next log rotate, the Datadog Agent might lose its read permissions. Set permissions as 644 in the log rotation configuration to make sure the Agent has read access to the files.
Use the icacls command on the log folder to obtain more information about the file permissions:
icacls path/to/logs/file /t
The /t flag runs the command recursively on files and sub-folders.
In the following example, the test directory and its children are not accessible to ddagentuser:
In case the application uses log rotation, (OI) and (CI) inheritance rights ensure that any future log files created in the directory inherit the parent folder permissions.
Run icacls again to check that ddagentuser has the correct permissions:
icaclspath/to/logs/file/t
In the following example, ddagentuser is listed in the file permissions:
When collecting logs from Journald, make sure that the Datadog Agent user is added in the systemd group as shown in the Journald integration.
Note: Journald sends an empty payload if the file permissions are incorrect. Accordingly, it is not possible to raise or send an explicit error message in this case.
These are a few of the common configuration issues that are worth triple-checking in your datadog-agent setup:
Check if the api_key is defined in datadog.yaml.
Check if you have logs_enabled: true in your datadog.yaml
By default the Agent does not collect any logs, make sure there is at least one .yaml file in the Agent’s conf.d/ directory that includes a logs section and the appropriate values.
You may have some .yaml parsing errors in your configuration files. YAML can be finicky, so when in doubt rely on a YAML validator.
If they appear in the Live Tail, check the Indexes configuration page for any exclusion filters that could match your logs.
If they do not appear in the Live Tail, they might have been dropped if their timestamp was further than 18 hours in the past. You can check which service and source may be impacted with the datadog.estimated_usage.logs.drop_count metric.
Logs above 1MB are truncated. You can check which service and source are impacted with the datadog.estimated_usage.logs.truncated_count and datadog.estimated_usage.logs.truncated_bytes metrics.