Getting Started with AWS

Overview

This guide provides an overview of the process for integrating an Amazon Web Services(AWS) account with Datadog using Datadog’s CloudFormation template.

At a high level, this involves creating an IAM role and associated policy to enable Datadog’s AWS account to make API calls into your AWS account for collecting or pushing data. The template also deploys the Datadog Forwarder Lambda function for sending logs to Datadog. Using the CloudFormation template provides all the tools needed to send this data to your Datadog account, and Datadog maintains the CloudFormation template to provide the latest functionality.

After the initial connection is established, you can enable individual AWS service integrations relevant to your AWS environment. With a single click, Datadog provisions the necessary resources in your AWS account and begins querying metrics and events for the services you use. For popular AWS services you are using, Datadog provisions out-of-the-box dashboards, providing immediate and customizable visibility. This guide demonstrates setting up the integration and installing the Datadog Agent on an Amazon Linux EC2 instance, as well as providing a broad overview of the integration’s capabilities. See the Enable integrations for individual AWS service section for a list of the available sub-integrations.

This process can be repeated for as many AWS accounts as necessary, or you can also use the API, AWS CLI, or Terraform to set up multiple accounts at once. For more information, read the Datadog-Amazon CloudFormation guide.

Prerequisites

Before getting started, ensure you have the following prerequisites:

  1. An AWS account. Your AWS user needs the following IAM permissions to successfully run the CloudFormation template:

    • cloudformation:CreateStack
    • ec2:DescribeSecurityGroups
    • ec2:DescribeSubnets
    • ec2:DescribeVpcs
    • iam:AttachRolePolicy
    • iam:CreatePolicy
    • iam:CreateRole
    • iam:PassRole
    • iam:PutRolePolicy
    • iam:UpdateAssumeRolePolicy
    • kms:Decrypt
    • lambda:AddPermission
    • lambda:CreateFunction
    • lambda:GetCodeSigningConfig
    • lambda:GetFunction
    • lambda:GetFunctionCodeSigningConfig
    • lambda:InvokeFunction
    • lambda:PutFunctionConcurrency
    • logs:CreateLogGroup
    • logs:DescribeLogGroups
    • logs:PutRetentionPolicy
    • s3:CreateBucket
    • s3:GetObject
    • s3:GetObjectVersion
    • secretsmanager:CreateSecret
    • secretsmanager:GetSecretValue
    • secretsmanager:PutSecretValue
    • serverless:CreateCloudFormationTemplate

Setup

  1. Go to the the AWS integration configuration page in Datadog and click Add AWS Account.

  2. Configure the integration’s settings under the Automatically using CloudFormation option.
    a. Select the AWS regions to integrate with.
    b. Add your Datadog API key.
    c. Optionally, send logs and other data to Datadog with the Datadog Forwarder Lambda.
    d. Optionally, enable Cloud Security Posture Management (CSPM) to scan your cloud environment, hosts, and containers for misconfigurations and security risks.

  3. Click Launch CloudFormation Template. This opens the AWS Console and loads the CloudFormation stack. All the parameters are filled in based on your selections in the prior Datadog form, so you do not need to edit those unless desired.
    Note: The DatadogAppKey parameter enables the CloudFormation stack to make API calls to Datadog to add and edit the Datadog configuration for this AWS account. The key is automatically generated and tied to your Datadog account.

  4. Check the required boxes from AWS and click Create stack. This launches the creation process for the Datadog stack along with three nested stacks. This could take several minutes. Ensure that the stack is successfully created before proceeding.

  5. After the stack is created, go back to the AWS integration tile in Datadog and click Ready!

  6. Wait up to 10 minutes for data to start being collected, and then view the out-of-the-box AWS overview dashboard to see metrics sent by your AWS services and infrastructure:

    The AWS overview dashboard in the Datadog account. On the left is the AWS logo and an AWS events graph showing 'No matching entries found'. In the center are graphs related to EBS volumes with numerical data displayed and a heat map showing consistent data. Along the right are graphs related to ELBs showing numerical data as well as a timeseries graph showing spiky data from three sources.

Enable integrations for individual AWS services

See the Integrations page for a full listing of the available sub-integrations. Many of these integrations are installed by default when Datadog recognizes data coming in from your AWS account.

Send logs

There are two ways of sending AWS service logs to Datadog:

  • Kinesis Firehose destination: Use the Datadog destination in your Kinesis Firehose delivery stream to forward logs to Datadog. It is recommended to use this approach when sending logs from CloudWatch in a very high volume.
  • Forwarder Lambda function: Deploy the Datadog Forwarder Lambda function, which subscribes to S3 buckets or your CloudWatch log groups and forwards logs to Datadog. You must use this approach to send traces, enhanced metrics, or custom metrics from Lambda functions asynchronously through logs. Datadog also recommends you use this approach to sending logs from S3 or other resources that cannot directly stream data to Kinesis.

Read the Enable logging for your AWS service section to get logs flowing for the most-used AWS services.

Validation

Once you have enabled logs, find them in the Logs Explorer using either the source or service facets from the facet panel, such as this example from S3:

The Logs Explorer page of the Datadog account. Along the left the image displays the Source and Service facets, both checked with 's3'. Along the right, some log entries are displayed in a list format.

Get more from the Datadog platform

Deeper visibility with the Datadog Agent on EC2

By default the Datadog AWS integration crawls the CloudWatch API for AWS-provided metrics, but you can gain even deeper visibility into your EC2 instances with the Datadog Agent. The Agent is a lightweight daemon that reports metrics and events, and can also be configured for logs and traces. The Agent Installation section of the Datadog application provides instructions for installing the Agent on a wide variety of operating systems. Many operating systems (for example, Amazon Linux) have one-step installation commands that you can run from the instance terminal to install the Agent:

The 'Agent' section of the 'Integrations' tab in Datadog. Along the left are displayed a list of supported operating systems for the Datadog Agent. 'Amazon Linux' is highlighted from this list. On the right is displayed 'Use our easy one-step install'. The command for installing the Agent is displayed below this, with the DD_API_KEY section obfuscated.

Once the Agent is installed, it’s graphically represented within the Infrastructure List with a bone icon:

The infrastructure list showing two hosts in a list format. Both hosts show the AWS icon for the AWS integration and 'aws' shown in a blue box to show they are associated with the AWS integration. One host also shows a dog-bone icon and blue boxes for 'ntp' and 'system'.

The screen shot above shows the host with the Datadog Agent reporting data from the System and NTP checks. The System check provides metrics around CPU, memory, filesystem, and I/O, providing additional insights into the host. You can enable additional integrations to suit the environment and use case, or additionally use DogStatsD to send custom metrics directly to Datadog.

See the FAQ on why you should install the Datadog Agent on your cloud instances for more information about the benefits of this approach.

Using the Datadog Agent with Amazon Container Services

For containerized environments, you can use the Datadog Agent, whether you’re managing your instances or leveraging Fargate for a serverless environment.

ECS with EC2 launch type

Use the Amazon ECS documentation to run the Datadog Docker Agent on the EC2 instances in your ECS cluster. Review the Amazon ECS Data Collection documentation to see the metrics and events reported to your Datadog account.

ECS with Fargate launch type

Use the Amazon ECS on AWS Fargate documentation to run the Agent as a container in the same task definition as your application. Note: Datadog Agent version 6.1.1 or higher is needed to take full advantage of the Fargate integration.

EKS

You don’t need any specific configuration for Amazon Elastic Kubernetes Service (EKS), as mentioned in the Kubernetes Distributions documentation. Use the dedicated Kubernetes documentation to deploy the Agent in your EKS cluster.

EKS with Fargate

Because Fargate pods are managed by AWS, they exclude host-based system checks like CPU and memory. To collect data from your AWS Fargate pods, use the Amazon EKS on AWS Fargate documentation to run the Agent as a sidecar of your application pod with custom role-based access control (RBAC). Note: This requires Datadog Agent version 7.17 or higher.

EKS Anywhere

Use the EKS Anywhere documentation for on-premises Kubernetes clusters.

Create additional Datadog resources

In addition to using the Datadog UI or API, you can create many Datadog resources with the CloudFormation Registry. For visibility and troubleshooting, use dashboards to display key data, apply Functions, and find Metric Correlations.

To get notified of any unwanted or unexpected behavior in your account, create monitors. Monitors consistently evaluate the data reported to your account, and send Notifications to ensure that the right information gets to the right team members. Review the List of Notification Integrations for all the ways to notify your team.

Serverless

You can unify the metrics, traces, and logs from your AWS Lambda functions running serverless applications in Datadog. Check out Serverless for instructions on instrumenting your application, installing Serverless Libraries and Integrations, implementing Distributed Tracing with Serverless Applications, or Serverless Troubleshooting.

APM

To dig even deeper and gather more data from your applications and AWS services, enable collecting distributed traces from either the AWS X-Ray integration or from a host with the Datadog Agent using APM. Then, read Explore Datadog APM for a better understanding of how to use this data to gain insights into your application performance.

Additionally, you can use Watchdog, an algorithmic feature for APM performance and infrastructure metrics, to automatically detect and be notified about potential application issues.

Security

Cloud SIEM

Review Getting Started with Cloud SIEM to evaluate your logs against the out-of-the-box Log Detection Rules. These rules are customizable, and when threats are detected, they generate security signals which can be accessed on the Security Signals Explorer. To ensure that the correct team is notified, use Notification Rules to configure notification preferences across multiple rules.

CSPM

Use the Getting Started with CSPM guide to learn about detecting and assessing misconfigurations in your cloud environment. Resource configuration data is evaluated against the out-of-the-box Posture Management Cloud and Infrastructure Detection Rules to flag attacker techniques and potential misconfigurations, allowing for fast response and remediation.

Troubleshooting

If you encounter any issues, be sure to check out the Troubleshooting section.

Further Reading