- 重要な情報
- はじめに
- 用語集
- ガイド
- エージェント
- インテグレーション
- OpenTelemetry
- 開発者
- API
- CoScreen
- アプリ内
- Service Management
- インフラストラクチャー
- アプリケーションパフォーマンス
- 継続的インテグレーション
- ログ管理
- セキュリティ
- UX モニタリング
- 管理
The Observability Pipelines Worker can collect, process, and route logs and metrics from any source to any destination. Using Datadog, you can build and manage all of your Observability Pipelines Worker deployments at scale.
There are several ways to get started with the Observability Pipelines Worker.
This document walks you through the quickstart installation steps and then provides resources for next steps.
If you are enrolled in the private beta of Remote Configuration, you can remotely roll out changes to your Workers from the Datadog UI, rather than make updates to your pipeline configuration in a text editor and then manually rolling out your changes. Choose your deployment method when you create a pipeline and install your Workers.
See Updating deployment modes on how to change the deployment mode after a pipeline is deployed.
To install the Observability Pipelines Worker, you need the following:
To generate a new API key and pipeline:
Follow the below instructions to install the Worker and deploy a sample pipeline configuration that uses demo data.
The Observability Pipelines Worker Docker image is published to Docker Hub here.
Download the sample pipeline configuration file. This configuration emits demo data, parses and structures the data, and then sends them to the console and Datadog. See Configurations for more information about the source, transform, and sink used in the sample configuration.
Run the following command to start the Observability Pipelines Worker with Docker:
docker run -i -e DD_API_KEY=<API_KEY> \
-e DD_OP_PIPELINE_ID=<PIPELINE_ID> \
-e DD_SITE=<SITE> \
-p 8282:8282 \
-v ./pipeline.yaml:/etc/observability-pipelines-worker/pipeline.yaml:ro \
datadog/observability-pipelines-worker run
Replace <API_KEY>
with your Datadog API key, <PIPELINES_ID>
with your Observability Pipelines configuration ID, and <SITE>
with . Note:
./pipeline.yaml
must be the relative or absolute path to the configuration you downloaded in step 1.
Download the Helm chart for AWS EKS. See Configurations for more information about the source, transform, and sink used in the sample configuration.
In the Helm chart, replace the datadog.apiKey
and datadog.pipelineId
values to match your pipeline and use for the
site
value. Then, install it in your cluster with the following commands:
helm repo add datadog https://helm.datadoghq.com
helm repo update
helm upgrade --install \
opw datadog/observability-pipelines-worker \
-f aws_eks.yaml
Download the Helm chart for Azure AKS. See Configurations for more information about the source, transform, and sink used in the sample configuration.
In the Helm chart, replace the datadog.apiKey
and datadog.pipelineId
values to match your pipeline and use for the
site
value. Then, install it in your cluster with the following commands:
helm repo add datadog https://helm.datadoghq.com
helm repo update
helm upgrade --install \
opw datadog/observability-pipelines-worker \
-f azure_aks.yaml
Download the Helm chart for Google GKE. See Configurations for more information about the source, transform, and sink used in the sample configuration.
In the Helm chart, replace the datadog.apiKey
and datadog.pipelineId
values to match your pipeline and use for the
site
value. Then, install it in your cluster with the following commands:
helm repo add datadog https://helm.datadoghq.com
helm repo update
helm upgrade --install \
opw datadog/observability-pipelines-worker \
-f google_gke.yaml
Install the Worker with the one-line install script or manually.
Run the one-line install command to install the Worker. Replace <DD_API_KEY>
with your Datadog API key, <PIPELINES_ID>
with your Observability Pipelines ID, and <SITE>
with .
DD_API_KEY=<DD_API_KEY> DD_OP_PIPELINE_ID=<PIPELINES_ID> DD_SITE=<SITE> bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_op_worker1.sh)"
Download the sample configuration file to /etc/observability-pipelines-worker/pipeline.yaml
on the host. See Configurations for more information about the source, transform, and sink used in the sample configuration.
Start the worker:
sudo systemctl restart observability-pipelines-worker
Run the following commands to set up APT to download through HTTPS:
sudo apt-get update
sudo apt-get install apt-transport-https curl gnupg
Run the following commands to set up the Datadog deb
repo on your system and create a Datadog archive keyring:
sudo sh -c "echo 'deb [signed-by=/usr/share/keyrings/datadog-archive-keyring.gpg] https://apt.datadoghq.com/ stable observability-pipelines-worker-1' > /etc/apt/sources.list.d/datadog-observability-pipelines-worker.list"
sudo touch /usr/share/keyrings/datadog-archive-keyring.gpg
sudo chmod a+r /usr/share/keyrings/datadog-archive-keyring.gpg
curl https://keys.datadoghq.com/DATADOG_APT_KEY_CURRENT.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch
curl https://keys.datadoghq.com/DATADOG_APT_KEY_F14F620E.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch
curl https://keys.datadoghq.com/DATADOG_APT_KEY_C0962C7D.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch
Run the following commands to update your local apt
repo and install the Worker:
sudo apt-get update
sudo apt-get install observability-pipelines-worker datadog-signing-keys
Add your keys and the site () to the Worker’s environment variables:
sudo cat <<-EOF > /etc/default/observability-pipelines-worker
DD_API_KEY=<API_KEY>
DD_OP_PIPELINE_ID=<PIPELINE_ID>
DD_SITE=<SITE>
EOF
Download the sample configuration file to /etc/observability-pipelines-worker/pipeline.yaml
on the host.
Start the Worker:
sudo systemctl restart observability-pipelines-worker
Install the Worker with the one-line install script or manually.
Run the one-line install command to install the Worker. Replace <DD_API_KEY>
with your Datadog API key, <PIPELINES_ID>
with your Observability Pipelines ID, and <SITE>
with .
DD_API_KEY=<DD_API_KEY> DD_OP_PIPELINE_ID=<PIPELINES_ID> DD_SITE=<SITE> bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_op_worker1.sh)"
Download the sample configuration file to /etc/observability-pipelines-worker/pipeline.yaml
on the host. See Configurations for more information about the source, transform, and sink used in the sample configuration.
Run the following command to start the Worker:
sudo systemctl restart observability-pipelines-worker
Run the following commands to set up the Datadog rpm
repo on your system:
cat <<EOF > /etc/yum.repos.d/datadog-observability-pipelines-worker.repo
[observability-pipelines-worker]
name = Observability Pipelines Worker
baseurl = https://yum.datadoghq.com/stable/observability-pipelines-worker-1/\$basearch/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://keys.datadoghq.com/DATADOG_RPM_KEY_CURRENT.public
https://keys.datadoghq.com/DATADOG_RPM_KEY_B01082D3.public
https://keys.datadoghq.com/DATADOG_RPM_KEY_FD4BF915.public
https://keys.datadoghq.com/DATADOG_RPM_KEY_E09422B3.public
EOF
Note: If you are running RHEL 8.1 or CentOS 8.1, use repo_gpgcheck=0
instead of repo_gpgcheck=1
in the configuration above.
Update your packages and install the Worker:
sudo yum makecache
sudo yum install observability-pipelines-worker
Add your keys and the site () to the Worker’s environment variables:
sudo cat <<-EOF > /etc/default/observability-pipelines-worker
DD_API_KEY=<API_KEY>
DD_OP_PIPELINE_ID=<PIPELINE_ID>
DD_SITE=<SITE>
EOF
Download the sample configuration file to /etc/observability-pipelines-worker/pipeline.yaml
on the host. See Configurations for more information about the source, transform, and sink used in the sample configuration.
Run the following command to start the Worker:
sudo systemctl restart observability-pipelines-worker
Set up the Worker module in your existing Terraform using this sample configuration. Update the values in vpc-id
, subnet-ids
, and region
to match your AWS deployment. Update the values in datadog-api-key
and pipeline-id
to match your pipeline.
See Configurations for more information about the source, transform, and sink used in the sample configuration.
module "opw" {
source = "https://github.com/DataDog/opw-terraform//aws"
vpc-id = "{VPC ID}"
subnet-ids = ["{SUBNET ID 1}", "{SUBNET ID 2}"]
region = "{REGION}"
datadog-api-key = "{DATADOG API KEY}"
pipeline-id = "{OP PIPELINE ID}"
pipeline-config = <<EOT
## SOURCES: Data sources that Observability Pipelines Worker collects data from.
## For a Datadog use case, we will receive data from the Datadog agent.
sources:
datadog_agent:
address: 0.0.0.0:8282
type: datadog_agent
multiple_outputs: true
transforms:
## The Datadog Agent natively encodes its tags as a comma-separated list
## of values that are stored in the string `.ddtags`. To work with
## and filter off of these tags, you need to parse that string into
## more structured data.
logs_parse_ddtags:
type: remap
inputs:
- datadog_agent.logs
source: |
.ddtags = parse_key_value!(.ddtags, key_value_delimiter: ":", field_delimiter: ",")
## The `.status` attribute added by the Datadog Agent needs to be deleted, otherwise
## your logs can be miscategorized at intake.
logs_remove_wrong_level:
type: remap
inputs:
- logs_parse_ddtags
source: |
del(.status)
## This is a placeholder for your own remap (or other transform)
## steps with tags set up. Datadog recommends these tag assignments.
## They show which data has been moved over to OP and what still needs
## to be moved.
LOGS_YOUR_STEPS:
type: remap
inputs:
- logs_remove_wrong_level
source: |
.ddtags.sender = "observability_pipelines_worker"
.ddtags.opw_aggregator = get_hostname!()
## Before sending data to the logs intake, you must re-encode the
## tags into the expected format, so that it appears as if the Agent is
## sending it directly.
logs_finish_ddtags:
type: remap
inputs:
- LOGS_YOUR_STEPS
source: |
.ddtags = encode_key_value(.ddtags, key_value_delimiter: ":", field_delimiter: ",")
metrics_add_dd_tags:
type: remap
inputs:
- datadog_agent.metrics
source: |
.tags.sender = "observability_pipelines_worker"
.tags.opw_aggregator = get_hostname!()
## This buffer configuration is split into the following, totaling the 288GB
## provisioned automatically by the Terraform module:
## - 240GB buffer for logs
## - 48GB buffer for metrics
##
## This should work for the vast majority of OP Worker deployments and should rarely
## need to be adjusted. If you do change it, be sure to update the `ebs-drive-size-gb`
## parameter.
sinks:
datadog_logs:
type: datadog_logs
inputs:
- logs_finish_ddtags
default_api_key: "$${DD_API_KEY}"
compression: gzip
buffer:
type: disk
max_size: 257698037760
datadog_metrics:
type: datadog_metrics
inputs:
- metrics_add_dd_tags
default_api_key: "$${DD_API_KEY}"
buffer:
type: disk
max_size: 51539607552
EOT
}
See Working with Data for more information on transforming your data.
After deploying a pipeline, you can also switch deployment methods, such as going from a manually managed pipeline to a remote configuration enabled pipeline or vice versa.
If you want to switch from a remote configuration deployment to a manually managed deployment:
DD_OP_REMOTE_CONFIGURATION_ENABLED
flag to false
and restart the Worker. Workers that are not restarted with this flag continue to be remote configuration enabled, which means that the Workers are not updated manually through a local configuration file.If you want to switch from manually managed deployment to a remote configuration deployment:
DD_OP_REMOTE_CONFIGURATION_ENABLED
flag to true
and restart the Worker. Workers that are not restarted with this flag are not polled for configurations deployed in the UI.The quickstart walked you through installing the Worker and deploying a sample pipeline configuration. For instructions on how to install the Worker to receive and route data from your Datadog Agents to Datadog or to receive and route data from your Splunk HEC to Splunk and Datadog, select your specific use case:
For recommendations on deploying and scaling multiple Workers:
お役に立つドキュメント、リンクや記事: