Fuente de Amazon Data Firehose

Este producto no es compatible con el sitio Datadog seleccionado. ().
Disponible para:

Registros

Descripción general

Utilice la fuente de Amazon Data Firehose de Observability Pipelines para recibir registros de Amazon Data Firehose.

Requisitos previos

To use Observability Pipelines’ Amazon Data Firehose:

  • Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See Understand HTTP endpoint delivery request and response specifications for more information.
  • If your forwarders are globally configured to enable SSL, you need the appropriate TLS certificates and the password you used to create your private key.

Configuración

Para la gestión de secretos: solo ingrese los identificadores para la dirección de Amazon Data Firehose y, si corresponde, la frase de contraseña de la clave TLS. No ingrese los valores reales.

Configure esta fuente cuando configure una canalización. Puede configurar una canalización en la interfaz de usuario, utilizando la API o con Terraform. Las instrucciones de esta sección son para configurar la fuente en la interfaz de usuario.

Después de seleccionar la fuente de Amazon Data Firehose en la pipeline UI, ingrese el identificador para su dirección de Amazon Data Firehose. Si lo deja en blanco, se utiliza el predeterminado.

If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP_. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

Configuración opcional

Autenticación de AWS

Seleccione una opción de AWS authentication. Si selecciona Assume role:

  1. Ingrese el ARN del rol de IAM que desea asumir.
  2. Opcionalmente, ingrese el nombre de la sesión del rol asumido y el ID externo.

Habilitar TLS

Toggle the switch to Enable TLS.

  • If you are using Secrets Management, enter the identifier for the key pass. See Secret defaults for the default used if the field is left blank.
  • Enter the following certificate and key files:
    • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • (Optional) CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • (Optional) Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
    • Notes:
      • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
      • The file must be readable by the observability-pipelines-worker group and user.

Valores predeterminados de secretos

These are the defaults used for secret identifiers and environment variables.

    • Identificador de la dirección de Amazon Data Firehose:
      • Hace referencia a la dirección del socket en la que Observability Pipelines Worker escucha para recibir registros.
      • El identificador predeterminado es SOURCE_AWS_DATA_FIREHOSE_ADDRESS.
    • Identificador de la frase de contraseña TLS de Amazon Data Firehose (cuando TLS está habilitado):
      • El identificador predeterminado es SOURCE_AWS_DATA_FIREHOSE_KEY_PASS.
    • Amazon Data Firehose address:
      • The Observability Pipelines Worker listens to this socket address to receive logs from Amazon Data Firehose.
      • The default environment variable is DD_OP_SOURCE_AWS_DATA_FIREHOSE_ADDRESS.
    • Amazon Data Firehose TLS passphrase (when enabled):
      • The default environment variable is DD_OP_SOURCE_AWS_DATA_FIREHOSE_KEY_PASS.

    Envíe registros al Observability Pipelines Worker a través de Amazon Data Firehose

    Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See Understand HTTP endpoint delivery request and response specifications for more information.

    To send logs to the Observability Pipelines Worker, set up an Amazon Data Firehose stream with an HTTP endpoint destination in the region where your logs are. Configure the endpoint URL to the endpoint where OPW is deployed.

    Amazon Data Firehose may send log events nested in an array, such as the structure shown below. To extract the records as individual events, use the Split Array processor and target the array. For example, logEvents. The Split Array processor enables you to break log data from arrays into individual events, making it easier for users to filter, query, and visualize data that was previously buried in layers.

    {
    "logEvents": [
        {log 1},
        {log 2},
        {log 3},
        {log n}
  ]
}

    Autenticación de AWS

    The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See AWS SDKs and Tools standardized credential providers for more information.

    Permisos

    For Observability Pipelines to collect logs from Amazon S3, the following policy permissions are required:

    • s3:GetObject
    • sqs:ReceiveMessage
    • sqs:DeleteMessage

    Métricas de estado

    Para métricas de componente y métricas de búfer de fuente emitidas por todas las fuentes, consulte la documentación de Pipelines Usage Metrics. Para filtrar o agrupar por las métricas de la fuente de Amazon Data Firehose, utilice la etiqueta component_type:aws_kinesis_firehose.