概要

Observability Pipelines の Amazon Data Firehose ソースを使用して、Amazon Data Firehose からログを受信します。

前提条件

To use Observability Pipelines’ Amazon Data Firehose:

• Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See Understand HTTP endpoint delivery request and response specifications for more information.
• If your forwarders are globally configured to enable SSL, you need the appropriate TLS certificates and the password you used to create your private key.

セットアップ

パイプラインをセットアップする際に、このソースを設定します。パイプラインは、UI、API、または Terraform を使用してセットアップできます。このセクションの手順は、UI でソースをセットアップするためのものです。

パイプライン UI で Amazon Data Firehose ソースを選択した後、Amazon Data Firehose アドレスの識別子を入力します。空白のままにすると、デフォルトが使用されます。

オプション設定

AWS 認証

AWS authentication オプションを選択します。Assume role を選択した場合、

1. 引き受ける IAM ロールの ARN を入力します。
2. 必要に応じて、引き受けロールのセッション名と外部 ID を入力します。

TLS を有効にする

Toggle the switch to Enable TLS.

• If you are using Secrets Management, enter the identifier for the key pass. See Secret defaults for the default used if the field is left blank.
• Enter the following certificate and key files:
  • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • (Optional) CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • (Optional) Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
  • Notes:
    • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
    • The file must be readable by the observability-pipelines-worker group and user.

シークレットのデフォルト値

These are the defaults used for secret identifiers and environment variables.

Amazon Data Firehose 経由で Observability Pipelines Worker にログを送信する

Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See Understand HTTP endpoint delivery request and response specifications for more information.

To send logs to the Observability Pipelines Worker, set up an Amazon Data Firehose stream with an HTTP endpoint destination in the region where your logs are. Configure the endpoint URL to the endpoint where OPW is deployed.

Amazon Data Firehose may send log events nested in an array, such as the structure shown below. To extract the records as individual events, use the Split Array processor and target the array. For example, logEvents. The Split Array processor enables you to break log data from arrays into individual events, making it easier for users to filter, query, and visualize data that was previously buried in layers.

{
    "logEvents": [
        {log 1},
        {log 2},
        {log 3},
        {log n}
  ]
}

AWS 認証

The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See AWS SDKs and Tools standardized credential providers for more information.

権限

For Observability Pipelines to collect logs from Amazon S3, the following policy permissions are required:

• s3:GetObject
• sqs:ReceiveMessage
• sqs:DeleteMessage

Health メトリクス

すべてのソースから出力されるコンポーネントメトリクスおよびソースバッファメトリクスについては、Pipelines 使用状況メトリクスのドキュメントを参照してください。Amazon Data Firehose ソースメトリクスでフィルタリングまたはグループ化するには、タグ component_type:aws_kinesis_firehose を使用してください。