Amazon VPC

Overview

Amazon Virtual Private Cloud (Amazon VPC) enables you to launch AWS resources into your virtual network. VPC Flow Logs is a feature that allows you to capture information about the IP traffic going to and from network interfaces in your VPC.

Setup

Installation

If you haven’t already, set up the Amazon Web Services integration first.

Metric collection

There are no additional steps required to collect non-aws.vpc.flowlogs.* Amazon VPC metrics. Metrics prefixed with aws.vpc.flowlogs.* are generated by the Datadog VPC Flow Logs integration. See the log collection section below to enable flow logs metrics collection.

For aws.vpc.subnet.* metrics:

Ensure that the Amazon EC2 integration is installed and EC2 metric collection is enabled under the Metric Collection tab in the AWS integration page.
Contact Datadog support to enable collection for your account.

Log collection

Find or create the destination resource in AWS for your VPC flow logs

VPC flow logs must first be sent to an intermediate destination before being sent to Datadog. You can send them directly to a Amazon Data Firehose, or they can be stored in an S3 bucket or a CloudWatch Log group.

Amazon Data Firehose is the recommended option for sending VPC flow logs to Datadog because it has less operational overhead and can be more cost-effective. Read Introducing Amazon VPC Flow Logs to Kinesis Data Firehose for more information.

Create a new or choose an existing:
- Amazon Data Firehose (recommended). If you don’t already have an existing Amazon Data Firehose delivery stream for sending logs to Datadog, follow the instructions in the Send AWS services logs with the Datadog Amazon Firehose Destination guide to create one. Note: You can optionally choose a delivery stream in another AWS account separate from your VPC for centralized log collection and delivery.
- S3 bucket or folder path.
- CloudWatch Log group.

Note: Specify vpc as the prefix for the S3 path or CloudWatch log group name to have the Lambda automatically tag the vpc source on the logs.

Enable VPC flow log logging

In the AWS console, go to the VPC you want to monitor.
Go to the Flow logs tab.
Click Create flow log.
Select the All filter to get both accepted and rejected connections.
Select the desired destination type (Amazon Data Firehose, S3 bucket, or CloudWatch log group) for the logs.
Fill in the details for the destination resource.
Click Create flow log.

Send logs to Datadog

If you selected Amazon Data Firehose as your destination, you are done!

If you selected an S3 bucket or the CloudWatch log group as your destination:

If you haven’t already, set up the Datadog Forwarder Lambda function in your AWS account.
Once set up, go to the Datadog Forwarder Lambda function. In the Function Overview section, click Add Trigger.
Select the S3 or CloudWatch Logs trigger for the Trigger Configuration.
Select the S3 bucket or CloudWatch log group that contains your VPC logs.
For S3, leave the event type as All object create events.
Click Add to add the trigger to your Lambda.

Go to the Log Explorer to start exploring your logs.

For more information on collecting AWS Services logs, see Send AWS Services Logs with the Datadog Lambda Function.

Data collected

Metrics


aws.transitgateway.bytes_in (count)	The number of bytes received by the transit gateway. Shown as byte
aws.transitgateway.bytes_out (count)	The number of bytes sent from the transit gateway. Shown as byte
aws.transitgateway.packet_drop_count_blackhole (count)	The number of packets dropped because they matched a blackhole route. Shown as packet
aws.transitgateway.packet_drop_count_no_route (count)	The number of packets dropped because they did not match a route. Shown as packet
aws.transitgateway.packets_in (count)	The number of packets received by the transit gateway. Shown as packet
aws.transitgateway.packets_out (count)	The number of packets sent by the transit gateway. Shown as packet
aws.vpc.flowlogs.action (count)	ACCEPT or REJECT if the traffic was permitted or not by the securtiy groups or network ACLs
aws.vpc.flowlogs.bytes.per_request.max (gauge)	The maximum number of bytes transferred per request during the capture window Shown as byte
aws.vpc.flowlogs.bytes.per_request.median (gauge)	The median number of bytes transferred per request during the capture window Shown as byte
aws.vpc.flowlogs.bytes.per_request.min (gauge)	The minimum number of bytes transferred per request during the capture window Shown as byte
aws.vpc.flowlogs.bytes.per_request.p90 (gauge)	The 90th percentile number of bytes transferred per request during the capture window Shown as byte
aws.vpc.flowlogs.bytes.per_request.p95 (gauge)	The 95th percentile number of bytes transferred per request during the capture window Shown as byte
aws.vpc.flowlogs.bytes.per_request.p99 (gauge)	The 99th percentile number of bytes transferred per request during the capture window Shown as byte
aws.vpc.flowlogs.bytes.total (count)	The total number of bytes transferred during the capture window Shown as byte
aws.vpc.flowlogs.duration.per_request.max (gauge)	The maximum duration per request during the capture window Shown as second
aws.vpc.flowlogs.duration.per_request.median (gauge)	The median duration per request during the capture window Shown as second
aws.vpc.flowlogs.duration.per_request.min (gauge)	The minimum duration per request during the capture window Shown as second
aws.vpc.flowlogs.duration.per_request.p90 (gauge)	The 90th percentile duration per request during the capture window Shown as second
aws.vpc.flowlogs.duration.per_request.p95 (gauge)	The 95th percentile duration per request during the capture window Shown as second
aws.vpc.flowlogs.duration.per_request.p99 (gauge)	The 99th percentile duration per request during the capture window Shown as second
aws.vpc.flowlogs.log_status (count)	The logging status of the flow log: OK NODATA or SKIPDATA
aws.vpc.flowlogs.packets.per_request.max (gauge)	The maximum number of packets transferred per request during the capture window Shown as packet
aws.vpc.flowlogs.packets.per_request.median (gauge)	The median number of packets transferred per request during the capture window Shown as packet
aws.vpc.flowlogs.packets.per_request.min (gauge)	The minimum number of packets transferred per request during the capture window Shown as packet
aws.vpc.flowlogs.packets.per_request.p90 (gauge)	The 90th percentile number of packets transferred per request during the capture window Shown as packet
aws.vpc.flowlogs.packets.per_request.p95 (gauge)	The 95th percentile number of packets transferred per request during the capture window Shown as packet
aws.vpc.flowlogs.packets.per_request.p99 (gauge)	The 99th percentile number of packets transferred per request during the capture window Shown as packet
aws.vpc.flowlogs.packets.total (count)	The total number of packets transferred during the capture window Shown as packet
aws.vpc.subnet.available_ip_address_count (gauge)	The number of available IP addresses in the subnet
aws.vpc.subnet.total_ip_address_count (gauge)	The total number of IP addresses contained within the subnet

Each of the metrics retrieved from AWS is assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.