Descripción general

Utilice el destino de Amazon Security Lake de Observability Pipelines para enviar registros a Amazon Security Lake.

Requisitos previos

Debe hacer lo siguiente antes de configurar el destino de Amazon Security Lake:

1. Follow the Getting Started with Amazon Security Lake to set up Amazon Security Lake, and make sure to:
   • Enable Amazon Security Lake for the AWS account.
   • Select the AWS regions where S3 buckets will be created for OCSF data.
2. Follow Collecting data from custom sources in Security Lake to create a custom source in Amazon Security Lake.
   • When you configure a custom log source in Security Lake in the AWS console:
     • Enter a source name.
     • Select the OCSF event class for the log source and type.
     • Enter the account details for the AWS account that will write logs to Amazon Security Lake:
       • AWS account ID
       • External ID
   • Select Create and use a new service for service access.
   • Take note of the name of the bucket that is created because you need it when you set up the Amazon Security Lake destination later on.
     • To find the bucket name, navigate to Custom Sources. The bucket name is in the location for your custom source. For example, if the location is s3://aws-security-data-lake-us-east-2-qjh9pr8hy/ext/op-api-activity-test, the bucket name is aws-security-data-lake-us-east-2-qjh9pr8hy.

Configuración

Configure el destino de Amazon Security Lake cuando configure un pipeline. Puede configurar un pipeline en la UI, utilizando la API o con Terraform. Los pasos en esta sección se configuran en la interfaz de usuario.

Notas:

• Cuando agrega el destino de Amazon Security Lake, el procesador OCSF se agrega automáticamente para que pueda convertir sus registros a Parquet antes de que se envíen a Amazon Security Lake. Consulte la documentación de Remap to OCSF para obtener instrucciones de configuración.
• Solo los registros formateados por el procesador OCSF se convierten a Parquet.

Después de seleccionar el destino de Amazon Security Lake en la interfaz de usuario del pipeline:

1. Ingrese el nombre de su bucket de S3.
2. Ingrese la región de AWS.
3. Ingrese el nombre de fuente personalizado.

Configuración opcional

Autenticación de AWS

1. Seleccione una opción de autenticación de AWS.
2. Ingrese el ARN del rol de IAM que desea asumir.
3. Opcionalmente, ingrese el nombre de la sesión del rol asumido y el ID externo.

Habilitar TLS

Toggle the switch to Enable TLS.

• If you are using Secrets Management, enter the identifier for the key pass. See Secret defaults for the default used if the field is left blank.
• Enter the following certificate and key files:
  • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • (Optional) CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • (Optional) Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
  • Notes:
    • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
    • The file must be readable by the observability-pipelines-worker group and user.

Almacenamiento en búfer

Toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing data to disk, ensuring buffered data persists through a Worker restart. See Destination buffers for more information.

• If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
• To configure a buffer on your destination:
  1. Select the buffer type you want to set (Memory or Disk).
  2. Enter the buffer size and select the unit.
     1. Maximum memory buffer size is 128 GB.
     2. Maximum disk buffer size is 5 TB.
        • Note: For Worker versions 2.20.x and older, the maximum disk buffer size is 500 GB.
  3. In the Behavior on full buffer dropdown menu, select whether you want to block events or drop new events when the buffer is full.

Valores predeterminados de Secret

These are the defaults used for secret identifiers and environment variables.

Cómo funciona el destino

Autenticación de AWS

The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See AWS SDKs and Tools standardized credential providers for more information.

Permisos

For Observability Pipelines to send logs to Amazon Security Lake, the following policy permissions are required:

• s3:ListBucket
• s3:PutObject

Procesamiento por lotes de eventos

Un lote de eventos se vacía cuando se cumple uno de estos parámetros. Consulte Agrupamiento de eventos de destino para obtener más información.