概要

Observability Pipelines の Amazon Security Lake 送信先を使用して、Amazon Security Lake にログを送信します。

前提条件

Amazon Security Lake 送信先をセットアップする前に、以下の操作を行う必要があります。

1. Follow the Getting Started with Amazon Security Lake to set up Amazon Security Lake, and make sure to:
   • Enable Amazon Security Lake for the AWS account.
   • Select the AWS regions where S3 buckets will be created for OCSF data.
2. Follow Collecting data from custom sources in Security Lake to create a custom source in Amazon Security Lake.
   • When you configure a custom log source in Security Lake in the AWS console:
     • Enter a source name.
     • Select the OCSF event class for the log source and type.
     • Enter the account details for the AWS account that will write logs to Amazon Security Lake:
       • AWS account ID
       • External ID
   • Select Create and use a new service for service access.
   • Take note of the name of the bucket that is created because you need it when you set up the Amazon Security Lake destination later on.
     • To find the bucket name, navigate to Custom Sources. The bucket name is in the location for your custom source. For example, if the location is s3://aws-security-data-lake-us-east-2-qjh9pr8hy/ext/op-api-activity-test, the bucket name is aws-security-data-lake-us-east-2-qjh9pr8hy.

セットアップ

パイプラインをセットアップする際に、Amazon Security Lake 送信先を設定します。パイプラインは、UI、API、または Terraform を使用してセットアップできます。このセクションの手順は、UI で設定します。

注記:

• Amazon Security Lake 送信先を追加すると、OCSF プロセッサが自動的に追加され、ログを Amazon Security Lake に送信する前に Parquet 形式に変換できるようになります。セットアップ手順については、OCSF ドキュメントへの再マッピングを参照してください。
• OCSF プロセッサによってフォーマットされたログのみが Parquet 形式に変換されます。

パイプライン UI で Amazon Security Lake 送信先を選択した後:

1. S3 バケット名を入力します。
2. AWS リージョンを入力します。
3. カスタムソース名を入力します。

オプション設定

AWS 認証

1. AWS 認証オプションを選択します。
2. 引き受ける IAM ロールの ARN を入力します。
3. 必要に応じて、引き受けるロールのセッション名と外部 ID を入力します。

TLS の有効化

Toggle the switch to Enable TLS.

• If you are using Secrets Management, enter the identifier for the key pass. See Secret defaults for the default used if the field is left blank.
• Enter the following certificate and key files:
  • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • (Optional) CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • (Optional) Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
  • Notes:
    • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
    • The file must be readable by the observability-pipelines-worker group and user.

バッファリング

Toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing data to disk, ensuring buffered data persists through a Worker restart. See Destination buffers for more information.

• If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
• To configure a buffer on your destination:
  1. Select the buffer type you want to set (Memory or Disk).
  2. Enter the buffer size and select the unit.
     1. Maximum memory buffer size is 128 GB.
     2. Maximum disk buffer size is 5 TB.
        • Note: For Worker versions 2.20.x and older, the maximum disk buffer size is 500 GB.
  3. In the Behavior on full buffer dropdown menu, select whether you want to block events or drop new events when the buffer is full.

シークレットのデフォルト

These are the defaults used for secret identifiers and environment variables.

送信先の仕組み

AWS 認証

The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See AWS SDKs and Tools standardized credential providers for more information.

権限

For Observability Pipelines to send logs to Amazon Security Lake, the following policy permissions are required:

• s3:ListBucket
• s3:PutObject

イベントのバッチ処理

イベントのバッチは、次のパラメータのいずれかが満たされたときにフラッシュされます。詳細については、送信先のイベントのバッチ処理を参照してください。