Amazon Firehose is the easiest way to load streaming data into AWS.
Enable this integration to see in Datadog all your Firehose metrics.
If you haven’t already, set up the Amazon Web Services integration first.
Firehose is checked under metric collection.Configure Amazon Firehose to send logs either to a S3 bucket or to Cloudwatch.
Note: If you log to a S3 bucket, make sure that amazon_firehose is set as Target prefix.
If you haven’t already, set up the Datadog log collection AWS Lambda function.
Once the lambda function is installed, manually add a trigger on the S3 bucket or Cloudwatch log group that contains your Amazon Firehose logs in the AWS console:
| aws.firehose.delivery_to_elasticsearch_bytes (count) | The total number of bytes indexed to Amazon ElasticSearch. Shown as byte |
| aws.firehose.delivery_to_elasticsearch_records (count) | The total number of records indexed to Amazon ElasticSearch. |
| aws.firehose.delivery_to_elasticsearch_success (gauge) | Fraction of records successfully indexed to Amazon ElasticSearch. Shown as fraction |
| aws.firehose.delivery_to_elasticsearch_success.sum (count) | Total number of records successfully indexed to Amazon ElasticSearch. |
| aws.firehose.delivery_to_redshift_bytes (gauge) | The average number of bytes copied to Amazon Redshift per delivery. Shown as byte |
| aws.firehose.delivery_to_redshift_bytes.sum (count) | The total number of bytes copied to Amazon Redshift. Shown as byte |
| aws.firehose.delivery_to_redshift_records (count) | The total number of records copied to Amazon Redshift. |
| aws.firehose.delivery_to_redshift_records.average (gauge) | The average number of records copied to Amazon Redshift per delivery. |
| aws.firehose.delivery_to_redshift_success (gauge) | Fraction of successful Redshift COPY commands Shown as fraction |
| aws.firehose.delivery_to_redshift_success.sum (count) | Total number of successful Redshift COPY commands |
| aws.firehose.delivery_to_s_3bytes (gauge) | The average number of bytes delivered to Amazon S3 per delivery. Shown as byte |
| aws.firehose.delivery_to_s_3bytes.sum (count) | The total number of bytes delivered to Amazon S3. Shown as byte |
| aws.firehose.delivery_to_s_3data_freshness (gauge) | Age (from getting into Firehose to now) of the oldest record in Firehose. Any record older than this age has been delivered to the S3 bucket. Shown as second |
| aws.firehose.delivery_to_s_3records (count) | The total number of records delivered to Amazon S3. |
| aws.firehose.delivery_to_s_3records.average (gauge) | The average number of records delivered to Amazon S3 per delivery. |
| aws.firehose.delivery_to_s_3success (gauge) | Fraction of successful deliveries to S3 Shown as fraction |
| aws.firehose.delivery_to_s_3success.sum (count) | Sum of successful deliveries to S3 |
| aws.firehose.delivery_to_splunk_bytes (gauge) | The average number of bytes delivered to Splunk per delivery. Shown as byte |
| aws.firehose.delivery_to_splunk_bytes.sum (count) | The total number of bytes delivered to Splunk. Shown as byte |
| aws.firehose.delivery_to_splunk_data_ack_latency (gauge) | The average approximate duration it takes to receive an acknowledgement from Splunk after Kinesis Data Firehose sends it data. Shown as second |
| aws.firehose.delivery_to_splunk_data_ack_latency.maximum (gauge) | The maximum approximate duration it takes to receive an acknowledgement from Splunk after Kinesis Data Firehose sends it data. Shown as second |
| aws.firehose.delivery_to_splunk_data_freshness (gauge) | Average age (from getting into Firehose to now) of the oldest record in Firehose. Any record older than this age has been delivered to Splunk. Shown as second |
| aws.firehose.delivery_to_splunk_data_freshness.maximum (gauge) | Maximum age (from getting into Firehose to now) of the oldest record in Firehose. Any record older than this age has been delivered to Splunk. Shown as second |
| aws.firehose.delivery_to_splunk_records (count) | The total number of records delivered to Splunk. |
| aws.firehose.delivery_to_splunk_records.average (gauge) | The average number of records delivered to Splunk per delivery. |
| aws.firehose.delivery_to_splunk_success (gauge) | Fraction of successfully indexed records Shown as fraction |
| aws.firehose.delivery_to_splunk_success.maximum (gauge) | Maximum value for fraction of successfully indexed records Shown as fraction |
| aws.firehose.delivery_to_splunk_success.minimum (gauge) | Minimum value for fraction of successfully indexed records Shown as fraction |
| aws.firehose.describe_delivery_stream_latency (gauge) | The average time taken per DescribeDeliveryStream operation. Shown as millisecond |
| aws.firehose.describe_delivery_stream_latency.maximum (gauge) | The maximum time taken per DescribeDeliveryStream operation. Shown as millisecond |
| aws.firehose.describe_delivery_stream_requests (count) | Total number of DescribeDeliveryStream requests. |
| aws.firehose.incoming_bytes (gauge) | The number of bytes ingested into the Firehose delivery stream. Shown as byte |
| aws.firehose.incoming_records (count) | The number of records ingested into the Firehose delivery stream. |
| aws.firehose.list_delivery_stream_latency (gauge) | The average time taken per ListDeliveryStream operation. Shown as millisecond |
| aws.firehose.list_delivery_stream_latency.maximum (gauge) | The maximum time taken per ListDeliveryStream operation. Shown as millisecond |
| aws.firehose.list_delivery_stream_requests (count) | Total number of ListDeliveryStream requests. |
| aws.firehose.list_delivery_streams_latency (gauge) | The average time taken per ListDeliveryStreams operation. Shown as millisecond |
| aws.firehose.list_delivery_streams_latency.maximum (gauge) | The maximum time taken per ListDeliveryStreams operation. Shown as millisecond |
| aws.firehose.list_delivery_streams_requests (count) | Total number of ListDeliveryStreams requests. |
| aws.firehose.put_record_bytes (count) | The total number of bytes put to the Firehose delivery stream using PutRecord. Shown as byte |
| aws.firehose.put_record_bytes.average (gauge) | The average number of bytes per PutRecord request Shown as byte |
| aws.firehose.put_record_latency (gauge) | The average time taken per PutRecord operation. Shown as millisecond |
| aws.firehose.put_record_latency.maximum (gauge) | The maximum time taken per PutRecord operation. Shown as millisecond |
| aws.firehose.put_record_requests (count) | Total number of PutRecord requests. |
| aws.firehose.put_record_batch_bytes (count) | The total number of bytes put to the Firehose delivery stream using PutRecordBatch. Shown as byte |
| aws.firehose.put_record_batch_bytes.average (gauge) | The average number of bytes per PutRecordBatch request. Shown as byte |
| aws.firehose.put_record_batch_latency (gauge) | The average time taken per PutRecordBatch operation. Shown as millisecond |
| aws.firehose.put_record_batch_latency.maximum (gauge) | The maximum time taken per PutRecordBatch operation. Shown as millisecond |
| aws.firehose.put_record_batch_records (count) | Total number of records from PutRecordBatch operations. |
| aws.firehose.put_record_batch_records.average (gauge) | Average number of records per PutRecordBatch operation. |
| aws.firehose.put_record_batch_requests (count) | Total number of PutRecordBatch requests. |
| aws.firehose.throttled_describe_stream (count) | Total number of times the DescribeStream operation is throttled when the data source is a Kinesis data stream. Shown as millisecond |
| aws.firehose.throttled_get_records (count) | The total number of times the GetRecords operation is throttled when the data source is a Kinesis data stream. Shown as millisecond |
| aws.firehose.throttled_get_shard_iterator (count) | The total number of times the GetShardIterator operation is throttled when the data source is a Kinesis data stream. Shown as millisecond |
| aws.firehose.update_delivery_stream_latency (gauge) | The average time taken per UpdateDeliveryStream operation. Shown as millisecond |
| aws.firehose.update_delivery_stream_latency.maximum (gauge) | The maximum time taken per UpdateDeliveryStream operation. Shown as millisecond |
| aws.firehose.update_delivery_stream_requests (count) | Total number of UpdateDeliveryStream requests. |
| aws.firehose.update_firehose_latency (gauge) | The average time taken per UpdateFirehose operation. Shown as millisecond |
| aws.firehose.update_firehose_latency.maximum (gauge) | The maximum time taken per UpdateFirehose operation. Shown as millisecond |
| aws.firehose.update_firehose_requests (count) | Total number of UpdateFirehose requests. |
Each of the metrics retrieved from AWS will be assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.
The AWS Kinesis Firehose integration does not include any events.
The AWS Kinesis Firehose integration does not include any service checks.
Need help? Contact Datadog support.