(LEGACY) Route Logs in Datadog-Rehydratable Format to Amazon S3

The Observability Pipelines Datadog Archives destination is in beta.

Overview

The Observability Pipelines datadog_archives destination formats logs into a Datadog-rehydratable format and then routes them to Log Archives. These logs are not ingested into Datadog, but are routed directly to the archive. You can then rehydrate the archive in Datadog when you need to analyze and investigate them.

The Observability Pipelines Datadog Archives destination is useful when:

  • You have a high volume of noisy logs, but you may need to index them in Log Management ad hoc.
  • You have a retention policy.

For example in this first diagram, some logs are sent to a cloud storage for archiving and others to Datadog for analysis and investigation. However, the logs sent directly to cloud storage cannot be rehydrated in Datadog when you need to investigate them.

A diagram showing logs going to cloud storage and Datadog.

In this second diagram, all logs are going to the Datadog Agent, including the logs that went to a cloud storage in the first diagram. However, in the second scenario, before the logs are ingested into Datadog, the datadog_archives destination formats and routes the logs that would have gone directly to a cloud storage to Datadog Log Archives instead. The logs in Log Archive can be rehydrated in Datadog when needed.

A diagram showing all logs going to Datadog.

This guide walks you through how to:

datadog_archives is available for Observability Pipelines Worker version 1.5 and later.

Configure a Log Archive

Create an Amazon S3 bucket

See AWS Pricing for inter-region data transfer fees and how cloud storage costs may be impacted.

  1. Navigate to Amazon S3 buckets.
  2. Click Create bucket.
  3. Enter a descriptive name for your bucket.
  4. Do not make your bucket publicly readable.
  5. Optionally, add tags.
  6. Click Create bucket.

Set up an IAM policy that allows Workers to write to the S3 bucket

  1. Navigate to the IAM console.
  2. Select Policies in the left side menu.
  3. Click Create policy.
  4. Click JSON in the Specify permissions section.
  5. Copy the below policy and paste it into the Policy editor. Replace <MY_BUCKET_NAME> and <MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1> with the information for the S3 bucket you created earlier.
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DatadogUploadAndRehydrateLogArchives",
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:GetObject"],
            "Resource": "arn:aws:s3:::<MY_BUCKET_NAME_1_/_MY_OPTIONAL_BUCKET_PATH_1>/*"
        },
        {
            "Sid": "DatadogRehydrateLogArchivesListBucket",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<MY_BUCKET_NAME>"
        }
    ]
}
  6. Click Next.
  7. Enter a descriptive policy name.
  8. Optionally, add tags.
  9. Click Create policy.

    Create an IAM user

    Create an IAM user and attach the IAM policy you created earlier to it.

    1. Navigate to the IAM console.
    2. Select Users in the left side menu.
    3. Click Create user.
    4. Enter a user name.
    5. Click Next.
    6. Select Attach policies directly.
    7. Choose the IAM policy you created earlier to attach to the new IAM user.
    8. Click Next.
    9. Optionally, add tags.
    10. Click Create user.

    Create access credentials for the new IAM user. Save these credentials as AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY.

    Create a service account

    Create a service account to use the policy you created above. In the Helm configuration, replace ${DD_ARCHIVES_SERVICE_ACCOUNT} with the name of the service account.

    Create an IAM user

    Create an IAM user and attach the IAM policy you created earlier to it.

    1. Navigate to the IAM console.
    2. Select Users in the left side menu.
    3. Click Create user.
    4. Enter a user name.
    5. Click Next.
    6. Select Attach policies directly.
    7. Choose the IAM policy you created earlier to attach to the new IAM user.
    8. Click Next.
    9. Optionally, add tags.
    10. Click Create user.

    Create access credentials for the new IAM user. Save these credentials as AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY.

    Create an IAM user

    Create an IAM user and attach the IAM policy you created earlier to it.

    1. Navigate to the IAM console.
    2. Select Users in the left side menu.
    3. Click Create user.
    4. Enter a user name.
    5. Click Next.
    6. Select Attach policies directly.
    7. Choose the IAM policy you created earlier to attach to the new IAM user.
    8. Click Next.
    9. Optionally, add tags.
    10. Click Create user.

    Create access credentials for the new IAM user. Save these credentials as AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY.

    Attach the policy to the IAM instance profile

    Attach the policy to the IAM Instance Profile that is created with Terraform, which you can find under the iam-role-name output.

    Connect the S3 bucket to Datadog Log Archives

    1. Navigate to Datadog Log Forwarding.
    2. Click Add a new archive.
    3. Enter a descriptive archive name.
    4. Add a query that filters out all logs going through log pipelines so that none of those logs go into this archive. For example, add the query observability_pipelines_read_only_archive, assuming no logs going through the pipeline have that tag added.
    5. Select AWS S3.
    6. Select the AWS Account that your bucket is in.
    7. Enter the name of the S3 bucket.
    8. Optionally, enter a path.
    9. Check the confirmation statement.
    10. Optionally, add tags and define the maximum scan size for rehydration. See Advanced settings for more information.
    11. Click Save.

    See the Log Archives documentation for additional information.

    Configure the datadog_archives destination

    You can configure the datadog_archives destination using the configuration file or the pipeline builder UI.

    If the Worker is ingesting logs that are not coming from the Datadog Agent and are routed to the Datadog Archives destination, those logs are not tagged with reserved attributes. This means that you lose Datadog telemetry and the benefits of unified service tagging. For example, say your syslogs are sent to datadog_archives and those logs have the status tagged as severity instead of the reserved attribute of status and the host tagged as hostname instead of the reserved attribute host. When these logs are rehydrated in Datadog, the status for the logs are all set to info and none of the logs will have a hostname tag.

    Configuration file

    For manual deployments, the sample pipelines configuration file for Datadog includes a sink for sending logs to Amazon S3 under a Datadog-rehydratable format.

      In the sample pipelines configuration file, replace AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with the AWS credentials you created earlier.

      In the sample pipelines configuration file, replace AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with the AWS credentials you created earlier.

      In the sample pipelines configuration file, replace AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with the AWS credentials you created earlier.

      In the sample pipelines configuration file, replace AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with the AWS credentials you created earlier.

      Replace ${DD_ARCHIVES_BUCKET} and ${DD_ARCHIVES_REGION} parameters based on your S3 configuration.

      Pipeline builder UI

      1. Navigate to your Pipeline.
      2. (Optional) Add a remap transform to tag all logs going to datadog_archives.
        a. Click Edit and then Add More in the **Add Transforms.
        b. Click the Remap tile.
        c. Enter a descriptive name for the component.
        d. In the Inputs field, select the source to connect this destination to.
        e. Add .sender = "observability_pipelines_worker" in the Source section.
        f. Click Save.
        g. Navigate back to your pipeline.
      3. Click Edit.
      4. Click Add More in the Add Destination tile.
      5. Click the Datadog Archives tile.
      6. Enter a descriptive name for the component.
      7. Select the sources or transforms to connect this destination to.
        1. In the Bucket field, enter the name of the S3 bucket you created earlier.
        2. Enter aws_s3 in the Service field.
        3. Toggle AWS S3 to enable those specific configuration options.
        4. In the Storage Class field, select the storage class in the dropdown menu.
        5. Set the other configuration options based on your use case.
        6. Click Save.
        1. In the Bucket field, enter the name of the S3 bucket you created earlier.
        2. Enter azure_blob in the Service field.
        3. Toggle Azure Blob to enable those specific configuration options.
        4. Enter the Azure Blob Storage Account connection string.
        5. Set the other configuration options based on your use case.
        6. Click Save.
        1. In the Bucket field, enter the name of the S3 bucket you created earlier.
        2. Enter gcp_cloud_storage in the Service field.
        3. Toggle GCP Cloud Storage to enable those specific configuration options.
        4. Set the configuration options based on your use case.
        5. Click Save.

        If you are using Remote Configuration, deploy the change to your pipeline in the UI. For manual configuration, download the updated configuration and restart the worker.

        See Datadog Archives reference for details on all configuration options.

        Rehydrate your archive

        See Rehydrating from Archives for instructions on how to rehydrate your archive in Datadog so that you can start analyzing and investigating those logs.

        Further reading

        Additional helpful documentation, links, and articles:

        Working with data in Observability PipelinesDOCUMENTATION more more Learn more about Log ArchivesDOCUMENTATION more more Learn more about rehydrating log archivesDOCUMENTATION more more