Utiliza la fuente de Amazon S3 de Observability Pipelines para recibir logs de Amazon S3. Selecciona y configura esta fuente cuando configures un pipeline.

Requisitos previos

To use Observability Pipelines’ Amazon S3 source, you must configure a SQS queue to receive your S3 bucket notifications.

Configurar la fuente en la interfaz de usuario del pipeline

Selecciona y configura esta fuente cuando configures un pipeline. La siguiente información se refiere a la configuración de la fuente en la interfaz de usuario del pipeline.

1. Enter the AWS region.
2. Optionally, select an AWS authentication option. If you select Assume role:
   1. Enter the ARN of the IAM role you want to assume.
   2. Optionally, enter the assumed role session name and external ID.
3. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.
   Note: All file paths are made relative to the configuration data directory, which is /var/lib/observability-pipelines-worker/config/ by default. See Advanced Configurations for more information. The file must be owned by the observability-pipelines-worker group and observability-pipelines-worker user, or at least readable by the group or user.
   • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
   • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
   • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

Configurar las variables de entorno

• Amazon S3 SQS URL
  • The URL of the SQS queue to which the S3 bucket sends the notification events.
  • Stored as the environment variable: DD_OP_SOURCE_AWS_S3_SQS_URL
• AWS_CONFIG_FILE path
  • The path to the AWS configuration file local to this node.
  • Stored as the environment variable: AWS_CONFIG_FILE.
• AWS_PROFILE name
  • The name of the profile to use within these files.
  • Stored as the environment variable: AWS_PROFILE.

Autenticación de AWS

The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See AWS SDKs and Tools standardized credential providers for more information.

Permisos

For Observability Pipelines to collect logs from Amazon S3, the following policy permissions are required:

• s3:GetObject
• sqs:ReceiveMessage
• sqs:DeleteMessage