The Service Map for APM is here!

Agent Troubleshooting

If you have not yet installed the Datadog Agent, go to the dedicated Agent integration page for installation instructions. If you just installed the Agent, it may take a few moments before you start seeing metrics appear. The first place you should check for metrics is the Metrics Explorer.

If you think you might be experiencing issues, the first thing to do is run the info command and check the Agent logs.

If you’re still unsure about the issue, you may reach out to Datadog support team along with a flare of your Agent.

Get more logging from the Agent

To enable the full debug mode:

  1. Modify your local datadog.yaml file (see this page to locate this configuration file on your instance)

  2. Replace # log_level: INFO with log_level: DEBUG (remove # to uncomment the line).

  3. Restart the Datadog Agent. See the Agent Commands page for OS-specific details.

  4. Wait a few minutes to generate some logs. See the Agent docuementation for the location of the logs.

Obtaining debug logs from the container Agent

Set the DD_LOG_LEVEL=debug environment variable when starting your Agent.

If your container is already running:

  1. To avoid that your process is restarted by S6, run:

    rm /var/run/s6/services/agent/finish

  2. Then stop the Agent:

    s6-svc -d /var/run/s6/services/agent/
    
  3. Restart then the Agent with debug log level by running:

    DD_LOG_LEVEL=debug agent start
    

When run in a container, the Agent cannot be restarted via service datadog-agent restart (or similar) as this will cause the container to be killed by Docker. Use supervisor to restart a containerised Agent:

/opt/datadog-agent/bin/supervisorctl -c /etc/dd-agent/supervisor.conf restart all

The following commands enable debug logging, restart the Agent, wait 60 seconds, then send a flare, in that order:

sed -i '/\[Main\]/a LOG_LEVEL=DEBUG' /etc/dd-agent/datadog.conf
/opt/datadog-agent/bin/supervisorctl -c /etc/dd-agent/supervisor.conf restart all
sleep 60
/etc/init.d/datadog-agent flare <CASE_ID>

Debug logs can be disabled with:

sed -i '/LOG_LEVEL=DEBUG/d' /etc/dd-agent/datadog.conf
/opt/datadog-agent/bin/supervisorctl -c /etc/dd-agent/supervisor.conf restart all

Or the container can be restarted.

Send a flare

If you are running the 5.3 version (or higher) of the Agent, you’re able to send all necessary troubleshooting information to the Datadog Support Team, with one flare command!

flare gathers all of the Agent’s configuration files and logs into an archive file. It removes sensitive information including passwords, API keys, Proxy credentials, and SNMP community strings.
Confirm the upload of the archive to immediately send it to Datadog support.
Since the Datadog Agent is completely open source, you can verify the code’s behavior. You can also review the archive prior to sending as the flare prompts a confirmation before uploading it.

In the commands below, replace <CASE_ID> with your Datadog support case ID, if you don’t specify a case ID, the command asks for an email address that is used to login in your organization and creates a new support case.

Platform Command
Linux sudo -u dd-agent -- datadog-agent flare <CASE_ID>
Docker docker exec -it datadog-agent agent flare <CASE_ID>
macOS datadog-agent flare <CASE_ID> or web [web GUI][8]
CentOS sudo datadog-agent flare <CASE_ID>
Debian sudo datadog-agent flare <CASE_ID>
Kubernetes kubectl exec <pod-name> -it agent flare <CASE_ID>
Fedora sudo datadog-agent flare <CASE_ID>
Redhat sudo datadog-agent flare <CASE_ID>
Suse sudo datadog-agent flare <CASE_ID>
Source sudo datadog-agent flare <CASE_ID>
Windows Consult the dedicated Windows documentation
Platform Command
Linux sudo /etc/init.d/datadog-agent flare <CASE_ID>
Docker docker exec -it dd-agent /etc/init.d/datadog-agent flare <CASE_ID>
macOS datadog-agent flare <CASE_ID>
CentOS sudo service datadog-agent flare <CASE_ID>
Debian sudo service datadog-agent flare <CASE_ID>
Kubernetes kubectl exec <pod-name> -it /etc/init.d/datadog-agent flare <CASE_ID>
Fedora sudo service datadog-agent flare <CASE_ID>
Redhat sudo service datadog-agent flare <CASE_ID>
Suse sudo service datadog-agent flare <CASE_ID>
Source sudo ~/.datadog-agent/bin/agent flare <CASE_ID>
Windows Consult the dedicated Windows documentation

Note: If you are using a Linux based system and the service wrapper command is not available, consult the list of alternatives.

Platform Command
Kubernetes kubectl exec <pod-name> -it datadog-cluster-agent flare <CASE_ID>

Status of an Agent check

Agent checks must be called by the Agent. To test an Agent check, run:

sudo -u dd-agent datadog-agent check <CHECK_NAME>

If you want to include rate metrics, add --check-rate to your command, for instance for Agent v6.x run:

sudo -u dd-agent datadog-agent check <CHECK_NAME> --check-rate
sudo -u dd-agent dd-agent check <CHECK_NAME>

If you want to include rate metrics, add --check-rate to your command, for instance for Agent v6.x run:

sudo -u dd-agent dd-agent check <CHECK_NAME> --check-rate

If your issue continues, reach out to the Datadog Support team with a flare.

Windows

Run the following script, with the proper <CHECK_NAME>:

C:\Program Files\Datadog\Datadog agent\embedded\agent.exe check <CHECK_NAME>

The Agent install includes a file called shell.exe in your \Program Files\ directory for the Datadog Agent which you can use to run Python within the Agent environment. Once your check (called <CHECK_NAME>) is written and you have the .py and .yaml files in their correct places, you can run the following in shell.exe:

from checks import run_check
run_check('<CHECK_NAME>')

This outputs any metrics or events that the check returns.

Run the following script, with the proper <CHECK_NAME>:

<INSTALL_DIR>/embedded/python.exe <INSTALL_DIR>agent/agent.py check <CHECK_NAME>

For example, to run the disk check:

C:\Program' 'Files\Datadog\Datadog' 'Agent\embedded\python.exe C:\Program' 'Files\Datadog\Datadog' 'Agent\agent\agent.py check disk

Systemd

For systems using systemd, use journalctl to assist with debugging.

The following command shows the status of the Datadog Agent.

sudo systemctl status datadog-agent

If the Agent failed to start, and no further information is provided, use the following command to display all logs for the Datadog Agent service. If needed, use -r to print logs in reverse order.

sudo journalctl -u datadog-agent.service

The following command shows the status of the Datadog Agent.

sudo systemctl status dd-agent

If the Agent failed to start, and no further information is provided, use the following command to display all logs for the Datadog Agent service. If needed, use -r to print logs in reverse order.

sudo journalctl -u dd-agent.service

Further Reading