Use Observability Pipelines’ Amazon Security Lake destination to send logs to Amazon Security Lake.

Prerequisites

You need to do the following before setting up the Amazon Security Lake destination:

1. Follow the Getting Started with Amazon Security Lake to set up Amazon Security Lake, and make sure to:
   • Enable Amazon Security Lake for the AWS account.
   • Select the AWS regions where S3 buckets will be created for OCSF data.
   • Take note of the Amazon Security Lake S3 bucket name. The bucket name is used when you set up the Amazon Security Lake destination in Observability Pipelines.
2. Follow Collecting data from custom sources in Security Lake to create a custom source in Amazon Security Lake.
   • When you configure a custom log source in Security Lake in the AWS console:
     • Enter a source name.
     • Select the OCSF event class for the log source and type.
     • Enter the account details for the AWS account that will write logs to Amazon Security Lake:
       • AWS account ID
       • External ID
   • Select Create and use a new service for service access.
3. Set up AWS authentication using AWS_PROFILE and AWS_CONFIG FILE environment variables. Observability Pipelines uses credentials associated with those environment variables to send logs to Amazon Security Lake. See AWS Authentication for more information.

Setup

Set up the Amazon Security Lake destination and its environment variables when you set up a pipeline. The information below is configured in the pipelines UI.

Set up the destination

1. Enter your S3 bucket name.
2. Enter the AWS region.
3. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required:
   • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
   • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
   • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

Notes:

• When you add the Amazon Security Lake destination, the OCSF processor is automatically added so that you can convert your logs to Parquet before they are sent to Amazon Security Lake. See Remap to OCSF documentation for setup instructions.
• Only logs formatted by the OCSF processor are converted to Parquet.

Set the environment variables

There are no environment variables to configure for the Amazon Security Lake destination.

AWS Authentication

To use the Amazon Security Lake destination, you need to set up AWS credential files and environment variables. Observability Pipelines uses those credentials to send logs to Amazon Security Lake. Datadog recommends setting up a specific AWS profile that can be used by Observability Pipelines.

To set up AWS authentication:

1. Create an IAM role if you don’t have one already. The role needs, at a minimum, these permissions to interact with the component. See Create a role to delegate permissions to an IAM user for more information.
2. In your AWS configuration file, create a new profile using the role_arn from the role you created in step 1.
3. When installing the Observability Pipelines Worker, ensure you set the AWS_PROFILE and AWS_CONFIG_FILE environment variables. The AWS_CONFIG_FILE variable is the path to your AWS configuration file. Set AWS_PROFILE to the name of the profile you created in step 2. See Configuration and credential file setting in the AWS CLI for more information. This is an example of a profile configuration:

   [profile profile_name]
   region = us-east-1
   output = json
   role_arn = arn:aws:iam::123456789:role/MyRole
   source_profile = default
   

Permissions

For Observability Pipelines to send logs to Amazon Security Lake, the following policy permissions are required:

• s3:ListBucket
• s3:PutObject

How the destination works

Event batching

A batch of events is flushed when one of these parameters is met. See event batching for more information.