Source Amazon Data Firehose

Disponible pour:

Logs

Présentation

Utilisez la source Amazon Data Firehose d’Observability Pipelines pour recevoir des logs depuis Amazon Data Firehose.

Prérequis

To use Observability Pipelines’ Amazon Data Firehose:

  • Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See Understand HTTP endpoint delivery request and response specifications for more information.
  • If your forwarders are globally configured to enable SSL, you need the appropriate TLS certificates and the password you used to create your private key.

Configuration

Pour la gestion des secrets : saisissez uniquement les identifiants de l'adresse Amazon Data Firehose et, le cas échéant, de la clé TLS. Ne saisissez pas les valeurs réelles.

Configurez cette source lorsque vous configurez un pipeline. Vous pouvez configurer un pipeline dans le UI, en utilisant l’API ou avec Terraform. Les instructions de cette section concernent la configuration de la source dans l’IU.

Après avoir sélectionné la source Amazon Data Firehose dans l’interface utilisateur du pipeline, saisissez l’identifiant de votre adresse Amazon Data Firehose. Si vous le laissez vide, le default est utilisé.

If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP_. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

Paramètres optionnels

Authentification AWS

Sélectionnez une option AWS authentication . Si vous sélectionnez Assume role :

  1. Saisissez l’ARN du rôle IAM que vous souhaitez assumer.
  2. Facultativement, saisissez le nom de session du rôle assumé et l’ID externe.

Activer TLS

Toggle the switch to Enable TLS.

  • If you are using Secrets Management, enter the identifier for the key pass. See Secret defaults for the default used if the field is left blank.
  • Enter the following certificate and key files:
    • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • (Optional) CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
    • (Optional) Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
    • Notes:
      • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
      • The file must be readable by the observability-pipelines-worker group and user.

Valeurs par défaut des secrets

These are the defaults used for secret identifiers and environment variables.

  • Identifiant de l’adresse Amazon Data Firehose :
    • Référence l’adresse de socket sur laquelle l’Observability Pipelines Worker écoute pour recevoir des logs.
    • L’identifiant par défaut est SOURCE_AWS_DATA_FIREHOSE_ADDRESS.
  • Identifiant de la phrase secrète TLS Amazon Data Firehose (lorsque TLS est activé) :
    • L’identifiant par défaut est SOURCE_AWS_DATA_FIREHOSE_KEY_PASS.
  • Amazon Data Firehose address:
    • The Observability Pipelines Worker listens to this socket address to receive logs from Amazon Data Firehose.
    • The default environment variable is DD_OP_SOURCE_AWS_DATA_FIREHOSE_ADDRESS.
  • Amazon Data Firehose TLS passphrase (when enabled):
    • The default environment variable is DD_OP_SOURCE_AWS_DATA_FIREHOSE_KEY_PASS.

Envoyer des logs à l’Observability Pipelines Worker via Amazon Data Firehose

Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See Understand HTTP endpoint delivery request and response specifications for more information.

To send logs to the Observability Pipelines Worker, set up an Amazon Data Firehose stream with an HTTP endpoint destination in the region where your logs are. Configure the endpoint URL to the endpoint where OPW is deployed.

Amazon Data Firehose may send log events nested in an array, such as the structure shown below. To extract the records as individual events, use the Split Array processor and target the array. For example, logEvents. The Split Array processor enables you to break log data from arrays into individual events, making it easier for users to filter, query, and visualize data that was previously buried in layers.

{
    "logEvents": [
        {log 1},
        {log 2},
        {log 3},
        {log n}
  ]
}

Authentification AWS

The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See AWS SDKs and Tools standardized credential providers for more information.

Autorisations

For Observability Pipelines to collect logs from Amazon S3, the following policy permissions are required:

  • s3:GetObject
  • sqs:ReceiveMessage
  • sqs:DeleteMessage

Métriques de santé

Pour les métriques de composant et les métriques de tampon de source émises par toutes les sources, consultez la documentation sur les Métriques d’utilisation des pipelines. Pour filtrer ou regrouper par les métriques de la source Amazon Data Firehose, utilisez le tag component_type:aws_kinesis_firehose.