- 필수 기능
- 앱 내
- 서비스 관리
- 인프라스트럭처
- 애플리케이션 성능
- 디지털 경험
- 소프트웨어 제공
- 보안
- 로그 관리
- 관리
- 인프라스트럭처
- ci
- containers
- csm
- ndm
- otel_guides
- overview
- slos
- synthetics
- tests
- 워크플로
The Observability Pipelines Worker can ingest logs from many different sources. If you have an Amazon S3 bucket that is receiving logs from an external system, such as AWS CloudTrail or CloudWatch, you can configure the Worker to ingest those logs. The setup uses Observability Pipelines Worker’s Amazon S3 source, which requires configuring an Amazon SQS queue to receive event notifications from the S3 bucket. The event notification then informs the Worker to collect the new log events in the S3 bucket.
This guide walks you through the following steps:
In the Amazon SQS console, provision a new queue specific to this configuration. This keeps any changes you make to it separate from any other log analysis tools that you are using.
${REGION}
, ${AWS_ACCOUNT_ID}
, ${QUEUE_NAME}
, and ${BUCKET_NAME}
with the relevant AWS account information, the queue name, and the bucket name you just entered. {
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": "SQS:SendMessage",
"Resource": "arn:aws:sqs:${REGION}:${AWS_ACCOUNT_ID}:${QUEUE_NAME}",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "${AWS_ACCOUNT_ID}"
},
"StringLike": {
"aws:SourceArn": "arn:aws:s3:*:*:${BUCKET_NAME}"
}
}
}
]
}
The SQS queue should now be receiving messages for the Worker to process.
If you encounter the “Unable to validate the following destination configurations” error, check that the SQS access policy is set up correctly.
Create a separate IAM role for the Worker so that only the necessary permissions are provided.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sqs:DeleteMessage",
"s3:GetObject",
"sqs:ReceiveMessage",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::${BUCKET_NAME}/*",
"arn:aws:s3:::${BUCKET_NAME}",
"arn:aws:sqs:${REGION}:${ACCOUNT_ID}:${QUEUE_NAME}"
]
}
]
}
${REGION
}, ${AWS_ACCOUNT_ID}
, ${QUEUE_NAME}
, and ${BUCKET_NAME}
with the relevant AWS account information and the queue and bucket names that you are using. You need to further modify the role permissions if you want the role to be attachable to EC2 instances, assumable by users, etc.Apply the role to the running Observability Pipelines process. You can do this by attaching the role to an EC2 instance or assuming a role from a given user profile.
sources:
cloudtrail:
type: aws_s3
region: ${REGION}
sqs:
queue_url: ${SQS_URL}
${REGION}
with the AWS account region. Replace ${SQS_URL}
with the HTTP URL provided in the SQS queue’s Details section in the console.See Amazon S3 source documentation for more options.
With the Amazon S3 source set up, you can now add transforms to manipulate the data and sinks to output the logs to destinations based on your use case. See Configurations for more information on sources, transforms, and sinks.
Most services (for example, CloudTrail) send logs to S3 in batches, which means that each event that the Worker receives is composed of multiple logs. In the below example, Records
is an array of three log events that are batched together.
{
"Records": [
{
"log event 1": "xxxx"
},
{
"log event 2": "xxxx"
},
{
"log event 3": "xxxx"
}
]
}
Add the following explode
and map
transforms to separate the batched log events into individual events for correct processing for sinks:
transforms:
explode:
type: remap
inputs:
- cloudtrail
source: |-
.message = parse_json!(.message)
. = unnest!(.message.Records)
map:
type: remap
inputs:
- explode
source: |-
merge!(., .message.Records)
del(.message)
In this example, the parse_json
function parses the string into JSON.
The unnest
function separates the batched log events into an array of individual log events.
[
{"Records": {"log event 1": "xxx"}},
{"Records": {"log event 2": "xxx"}},
{"Records": {"log event 3": "xxx"}}
]
Then the merge
function collapses the data in .Records
to the top level so that each log event is an individual log line. The del
function removes the extraneous field.
{"log event 1": "xxx"}
{"log event 2": "xxx"}
{"log event 3": "xxx"}
Additional helpful documentation, links, and articles: