Getting Started with AWS
This guide provides an overview of the process for integrating an Amazon Web Services(AWS) account with Datadog using Datadog’s CloudFormation template.
At a high level, this involves creating an IAM role and associated policy to enable Datadog’s AWS account to make API calls into your AWS account for collecting or pushing data. The template also deploys the Datadog Forwarder Lambda function for sending logs to Datadog. Using the CloudFormation template provides all the tools needed to send this data to your Datadog account, and Datadog maintains the CloudFormation template to provide the latest functionality.
After the initial connection is established, you can enable individual AWS service integrations relevant to your AWS environment. With a single click, Datadog provisions the necessary resources in your AWS account and begins querying metrics and events for the services you use. For popular AWS services you are using, Datadog provisions out-of-the-box dashboards, providing immediate and customizable visibility. This guide demonstrates setting up the integration, sending logs from CloudTrail and the Forwarder Lambda function, and installing the Datadog Agent on an Amazon Linux EC2 instance. See the Enable integrations for individual AWS service section for a list of the available sub-integrations.
This process can be repeated for as many AWS accounts as necessary, or you can also use the API, AWS CLI, or Terraform to set up multiple accounts at once. For more information, read the Datadog-Amazon CloudFormation guide.
Before getting started, ensure you have the following prerequisites:
An AWS account. Your AWS user needs the following IAM permissions to successfully run the CloudFormation template:
From the AWS tile on the Integrations page in your Datadog account, select the Datadog products you wish to integrate with this AWS Account. This selects the correct default settings for integrating data from this AWS account for those products. These settings can be changed in the future if needed.
Select the AWS Region where the CloudFormation stack will be launched. This also sets where to create the Datadog Lambda Forwarder for sending AWS logs to Datadog (if you selected Log Management).
Note: CloudWatch metrics are collected from ALL AWS regions you are using regardless of the region you select.
Select or create the Datadog API Key used to send data from your AWS account to Datadog.
Click “Launch CloudFormation Template”. This opens the AWS Console and loads the CloudFormation stack. All the parameters are filled in based on your selections in the prior Datadog form, so you do not need to edit those unless desired.
DatadogAppKey parameter enables the CloudFormation stack to make API calls to Datadog to add and edit the Datadog configuration for this AWS account. The key is automatically generated and tied to your Datadog account.
Check the required boxes from AWS and click This launches the creation process for the Datadog stack along with three nested stacks. This could take several minutes. Ensure that the stack is successfully created before proceeding.
After the Stack is created, go back to the AWS integration tile in Datadog and find the box for the new account you created. Click “Refresh to Check Status” to see a success message at the top of the page, along with the new account visible on the page with the relevant details.
Depending on which AWS services you use and your use case for monitoring, there are multiple options within the integration tile to specify the data to be collected. For example, you can limit data collection based on AWS service, namespace, or tags. Additionally, you can choose to mute monitor notifications. For example, terminations triggered manually or by autoscaling with EC2 automuting enabled. If needed, enable Alarm Collection to send your CloudWatch alarms to the Datadog Event Stream and choose whether to collect custom metrics.
Wait up to 10 minutes for data to start being collected, and then view the out-of-the-box AWS overview dashboard to see metrics sent by your AWS services and infrastructure:
Enable integrations for individual AWS service
See the Integrations page for a full listing of the available sub-integrations. Many of these integrations are installed by default when Datadog recognizes data coming in from your AWS account.
For a full list of ways you can send your AWS logs to Datadog, see Enable logging for your AWS service.
Once you have enabled logs, find them in the Logs Explorer using either the
service facets from the facet panel, such as this example from S3:
Get more from the Datadog platform
Deeper visibility with the Datadog Agent on EC2
By default the Datadog AWS integration crawls the CloudWatch API for AWS-provided metrics, but you can gain even deeper visibility into your EC2 instances with the Datadog Agent. The Agent is a lightweight daemon that reports metrics and events, and can also be configured for logs and traces. The Agent Installation section of the Datadog application provides instructions for installing the Agent on a wide variety of operating systems. Many operating systems (for example, Amazon Linux) have one-step installation commands that you can run from the instance terminal to install the Agent:
Once the Agent is installed, it’s graphically represented within the Infrastructure List with a bone icon:
The screen shot above shows the host with the Datadog Agent reporting data from the System and NTP checks. The System check provides metrics around CPU, memory, filesystem, and I/O, providing additional insights into the host. You can enable additional integrations to suit the environment and use case, or additionally use DogStatsD to send custom metrics directly to Datadog.
See the FAQ on why you should install the Datadog Agent on your cloud instances for more information about the benefits of this approach.
Using the Datadog Agent with Amazon Container Services
For containerized environments, you can use the Datadog Agent, whether you’re managing your instances or leveraging Fargate for a serverless environment.
ECS with EC2 launch type
Use the Amazon ECS documentation to run the Datadog Docker Agent on the EC2 instances in your ECS cluster. Review the Amazon ECS Data Collection documentation to see the metrics and events reported to your Datadog account.
ECS with Fargate launch type
Use the Amazon ECS on AWS Fargate documentation to run the Agent as a container in the same task definition as your application. Note: Datadog Agent version 6.1.1 or higher is needed to take full advantage of the Fargate integration.
You don’t need any specific configuration for Amazon Elastic Kubernetes Service (EKS), as mentioned in the Kubernetes Distributions documentation. Use the dedicated Kubernetes documentation to deploy the Agent in your EKS cluster.
EKS with Fargate
Because Fargate pods are managed by AWS, they exclude host-based system checks like CPU and memory. To collect data from your AWS Fargate pods, use the Amazon EKS on AWS Fargate documentation to run the Agent as a sidecar of your application pod with custom role-based access control (RBAC). Note: This requires Datadog Agent version 7.17 or higher.
Use the EKS Anywhere documentation for on-premises Kubernetes clusters.
Create additional Datadog resources
In addition to using the Datadog UI or API, you can create many Datadog resources with the CloudFormation Registry. For visibility and troubleshooting, use dashboards to display key data, apply Functions, and find Metric Correlations.
To get notified of any unwanted or unexpected behavior in your account, create monitors. Monitors consistently evaluate the data reported to your account, and send Notifications to ensure that the right information gets to the right team members. Review the List of Notification Integrations for all the ways to notify your team.
You can unify the metrics, traces, and logs from your AWS Lambda functions running serverless applications in Datadog. Check out Serverless for instructions on instrumenting your application, installing Serverless Libraries and Integrations, implementing Distributed Tracing with Serverless Applications, or Serverless Troubleshooting.
To dig even deeper and gather more data from your applications and AWS services, enable collecting distributed traces from either the AWS X-Ray integration or from a host with the Datadog Agent using APM. Then, read Explore Datadog APM for a better understanding of how to use this data to gain insights into your application performance.
Additionally, you can use Watchdog, an algorithmic feature for APM performance and infrastructure metrics, to automatically detect and be notified about potential application issues.
Review Getting Started with Cloud SIEM to evaluate your logs against the out-of-the-box Log Detection Rules. These rules are customizable, and when threats are detected, they generate security signals which can be accessed on the Security Signals Explorer. To ensure that the correct team is notified, use Notification Rules to configure notification preferences across multiple rules.
Use the Getting Started with CSPM guide to learn about detecting and assessing misconfigurations in your cloud environment. Resource configuration data is evaluated against the out-of-the-box Posture Management Cloud and Infrastructure Detection Rules to flag attacker techniques and potential misconfigurations, allowing for fast response and remediation.
If you encounter any issues, be sure to check out the Troubleshooting section.
Additional helpful documentation, links, and articles: