개요

Observability Pipelines의 Amazon Data Firehose 소스를 사용하여 Amazon Data Firehose에서 로그를 수신하세요.

전제 조건

To use Observability Pipelines’ Amazon Data Firehose:

• Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See Understand HTTP endpoint delivery request and response specifications for more information.
• If your forwarders are globally configured to enable SSL, you need the appropriate TLS certificates and the password you used to create your private key.

설정

파이프라인을 설정할 때 이 소스를 설정하세요. 파이프라인은 UI, API 또는 Terraform을 사용하여 설정할 수 있습니다. 이 섹션의 지침은 UI에서 소스를 설정하는 방법을 설명합니다.

파이프라인 UI에서 Amazon Data Firehose 소스를 선택한 후, Amazon Data Firehose 주소의 식별자를 입력하세요. 비워두면 기본값이 사용됩니다.

선택적 설정

AWS 인증

AWS authentication 옵션을 선택하세요. Assume role을 선택한 경우:

1. 맡으려는 IAM 역할의 ARN을 입력합니다.
2. 필요시 맡은 역할의 세션 이름과 외부 ID를 입력합니다.

TLS 활성화

Toggle the switch to Enable TLS.

• If you are using Secrets Management, enter the identifier for the key pass. See Secret defaults for the default used if the field is left blank.
• Enter the following certificate and key files:
  • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • (Optional) CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • (Optional) Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
  • Notes:
    • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
    • The file must be readable by the observability-pipelines-worker group and user.

시크릿 기본값

These are the defaults used for secret identifiers and environment variables.

Amazon Data Firehose를 통해 Observability Pipelines Worker로 로그 전송

Since Amazon Data Firehose can only deliver data over HTTP to an HTTPS URL, when you deploy the Observability Pipelines Worker, you need to deploy it with a publicly exposed endpoint and solve TLS termination. To solve TLS termination, you can front OPW with a load balancer or configure TLS options. See Understand HTTP endpoint delivery request and response specifications for more information.

To send logs to the Observability Pipelines Worker, set up an Amazon Data Firehose stream with an HTTP endpoint destination in the region where your logs are. Configure the endpoint URL to the endpoint where OPW is deployed.

Amazon Data Firehose may send log events nested in an array, such as the structure shown below. To extract the records as individual events, use the Split Array processor and target the array. For example, logEvents. The Split Array processor enables you to break log data from arrays into individual events, making it easier for users to filter, query, and visualize data that was previously buried in layers.

{
    "logEvents": [
        {log 1},
        {log 2},
        {log 3},
        {log n}
  ]
}

AWS 인증

The Observability Pipelines Worker uses the standard AWS credential provider chain for authentication. See AWS SDKs and Tools standardized credential providers for more information.

권한

For Observability Pipelines to collect logs from Amazon S3, the following policy permissions are required:

• s3:GetObject
• sqs:ReceiveMessage
• sqs:DeleteMessage

상태 메트릭

모든 소스에서 내보내는 구성 요소 메트릭 및 소스 버퍼 메트릭은 파이프라인 사용량 메트릭 설명서를 참조하세요. Amazon Data Firehose 소스 메트릭별로 필터링하거나 그룹화하려면 component_type:aws_kinesis_firehose 태그를 사용하세요.