(LEGACY) Set Up the Observability Pipelines Worker

이 페이지는 아직 한국어로 제공되지 않으며 번역 작업 중입니다. 번역에 관한 질문이나 의견이 있으시면 언제든지 저희에게 연락해 주십시오.

Observability Pipelines is not available on the US1-FED Datadog site.

OP Worker 버전 1.8 이하에서 버전 2.0 이상으로 업그레이드할 경우 기존 파이프라인에 오류가 발생합니다. OP Worker 버전이 1.8 이하이고 해당 버전을 계속 사용하고 싶을 경우에는 버전을 업그레이드하지 마세요. OP Worker 2.0 이상을 사용하고 싶을 경우에는 OP Worker 버전 1.8 이하의 파이프라인을 OP Worker 2.x.로 마이그레이션해야 합니다.

Datadog에서는 OP Worker 버전 2.0 이상으로 업그레이드하는 것을 권고합니다. 주 OP Worker 버전으로 업그레이드하고 업데이트해야 최신 OP Worker의 기능, 버그 수정, 보안 업데이트 서비스를 받을 수 있습니다.

Overview

The Observability Pipelines Worker can collect, process, and route logs from any source to any destination. Using Datadog, you can build and manage all of your Observability Pipelines Worker deployments at scale.

There are several ways to get started with the Observability Pipelines Worker.

  • Quickstart: Install the Worker with a simple pipeline that emits demo data to get started quickly.
  • Datadog setup guide: Install the Worker with an out-of-the-box pipeline for receiving and routing data from your Datadog Agents to Datadog.
  • Datadog archiving setup guide: Install the Worker with an out-of-the-box pipeline for receiving and routing data from your Datadog Agents to Datadog and S3.
  • Splunk setup guide: Install the Worker with an out-of-the-box pipeline for receiving and routing data from Splunk HEC to both Splunk and Datadog.

This document walks you through the quickstart installation steps and then provides resources for next steps. Use and operation of this software is governed by the End User License Agreement.

Deployment Modes

관측 파이프라인용 원격 구성이 프라이빗 베타 서비스 중입니다. 서비스에 액세스하려면 Datadog 지원팀 이나 고객 성공 매니저에게 문의하세요.

원격 구성 프라이빗 베타에 등록하면 텍스트 편집기에서 파이프라인 구성을 업데이트한 후 수동으로 변경 사항을 출시하는 대신 변경 사항을 Datadog UI에서 작업자로 원격 출시할 수 있습니다. 파이프라인을 생성할 때 배포 방법을 선택하고 작업자를 설치하세요.

파이프라인을 배포한 후 배포 모드를 변경하는 방법에 관해서는 배포 모드 업데이트를 참고하세요.

Prerequisites

To install the Observability Pipelines Worker, you need the following:

To generate a new API key and pipeline:

  1. Navigate to Observability Pipelines.
  2. Click New Pipeline.
  3. Enter a name for your pipeline.
  4. Click Next.
  5. Select the template you want and follow the instructions.

Quickstart

Follow the below instructions to install the Worker and deploy a sample pipeline configuration that uses demo data.

Install the Observability Pipelines Worker

    The Observability Pipelines Worker Docker image is published to Docker Hub here.

    1. Download the sample pipeline configuration file. This configuration emits demo data, parses and structures the data, and then sends them to the console and Datadog. See Configurations for more information about the source, transform, and sink used in the sample configuration.

    2. Run the following command to start the Observability Pipelines Worker with Docker:

      docker run -i -e DD_API_KEY=<API_KEY> \
  -e DD_OP_PIPELINE_ID=<PIPELINE_ID> \
  -e DD_SITE=<SITE> \
  -p 8282:8282 \
  -v ./pipeline.yaml:/etc/observability-pipelines-worker/pipeline.yaml:ro \
  datadog/observability-pipelines-worker run

      Replace <API_KEY> with your Datadog API key, <PIPELINES_ID> with your Observability Pipelines configuration ID, and <SITE> with . Note: ./pipeline.yaml must be the relative or absolute path to the configuration you downloaded in step 1.

    1. Download the Helm chart values file for AWS EKS. See Configurations for more information about the source, transform, and sink used in the sample configuration.

    2. In the Helm chart, replace the datadog.apiKey and datadog.pipelineId values to match your pipeline and use for the site value. Then, install it in your cluster with the following commands:

      helm repo add datadog https://helm.datadoghq.com

      helm repo update

      helm upgrade --install \
    opw datadog/observability-pipelines-worker \
    -f aws_eks.yaml

    1. Download the Helm chart values file for Azure AKS. See Configurations for more information about the source, transform, and sink used in the sample configuration.

    2. In the Helm chart, replace the datadog.apiKey and datadog.pipelineId values to match your pipeline and use for the site value. Then, install it in your cluster with the following commands:

      helm repo add datadog https://helm.datadoghq.com

      helm repo update

      helm upgrade --install \
  opw datadog/observability-pipelines-worker \
  -f azure_aks.yaml

    1. Download the Helm chart values file for Google GKE. See Configurations for more information about the source, transform, and sink used in the sample configuration.

    2. In the Helm chart, replace the datadog.apiKey and datadog.pipelineId values to match your pipeline and use for the site value. Then, install it in your cluster with the following commands:

      helm repo add datadog https://helm.datadoghq.com

      helm repo update

      helm upgrade --install \
  opw datadog/observability-pipelines-worker \
  -f google_gke.yaml

    Install the Worker with the one-line install script or manually.

    One-line installation script

    1. Run the one-line install command to install the Worker. Replace <DD_API_KEY> with your Datadog API key, <PIPELINES_ID> with your Observability Pipelines ID, and <SITE> with .

      DD_API_KEY=<DD_API_KEY> DD_OP_PIPELINE_ID=<PIPELINES_ID> DD_SITE=<SITE> bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_op_worker1.sh)"

    2. Download the sample configuration file to /etc/observability-pipelines-worker/pipeline.yaml on the host. See Configurations for more information about the source, transform, and sink used in the sample configuration.

    3. Start the worker:

      sudo systemctl restart observability-pipelines-worker

    Manual installation

    1. Run the following commands to set up APT to download through HTTPS:

      sudo apt-get update
sudo apt-get install apt-transport-https curl gnupg

    2. Run the following commands to set up the Datadog deb repo on your system and create a Datadog archive keyring:

      sudo sh -c "echo 'deb [signed-by=/usr/share/keyrings/datadog-archive-keyring.gpg] https://apt.datadoghq.com/ stable observability-pipelines-worker-1' > /etc/apt/sources.list.d/datadog-observability-pipelines-worker.list"
sudo touch /usr/share/keyrings/datadog-archive-keyring.gpg
sudo chmod a+r /usr/share/keyrings/datadog-archive-keyring.gpg
curl https://keys.datadoghq.com/DATADOG_APT_KEY_CURRENT.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch
curl https://keys.datadoghq.com/DATADOG_APT_KEY_06462314.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch
curl https://keys.datadoghq.com/DATADOG_APT_KEY_F14F620E.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch
curl https://keys.datadoghq.com/DATADOG_APT_KEY_C0962C7D.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch

    3. Run the following commands to update your local apt repo and install the Worker:

      sudo apt-get update
sudo apt-get install observability-pipelines-worker datadog-signing-keys

    4. Add your keys and the site () to the Worker’s environment variables:

      sudo cat <<-EOF > /etc/default/observability-pipelines-worker
DD_API_KEY=<API_KEY>
DD_OP_PIPELINE_ID=<PIPELINE_ID>
DD_SITE=<SITE>
EOF

    5. Download the sample configuration file to /etc/observability-pipelines-worker/pipeline.yaml on the host.

    6. Start the Worker:

      sudo systemctl restart observability-pipelines-worker

    Install the Worker with the one-line install script or manually.

    One-line installation script

    1. Run the one-line install command to install the Worker. Replace <DD_API_KEY> with your Datadog API key, <PIPELINES_ID> with your Observability Pipelines ID, and <SITE> with .

      DD_API_KEY=<DD_API_KEY> DD_OP_PIPELINE_ID=<PIPELINES_ID> DD_SITE=<SITE> bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_op_worker1.sh)"

    2. Download the sample configuration file to /etc/observability-pipelines-worker/pipeline.yaml on the host. See Configurations for more information about the source, transform, and sink used in the sample configuration.

    3. Run the following command to start the Worker:

      sudo systemctl restart observability-pipelines-worker

    Manual installation

    1. Run the following commands to set up the Datadog rpm repo on your system:

      cat <<EOF > /etc/yum.repos.d/datadog-observability-pipelines-worker.repo
[observability-pipelines-worker]
name = Observability Pipelines Worker
baseurl = https://yum.datadoghq.com/stable/observability-pipelines-worker-1/\$basearch/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://keys.datadoghq.com/DATADOG_RPM_KEY_CURRENT.public
       https://keys.datadoghq.com/DATADOG_RPM_KEY_4F09D16B.public
       https://keys.datadoghq.com/DATADOG_RPM_KEY_B01082D3.public
       https://keys.datadoghq.com/DATADOG_RPM_KEY_FD4BF915.public
       https://keys.datadoghq.com/DATADOG_RPM_KEY_E09422B3.public
EOF

      Note: If you are running RHEL 8.1 or CentOS 8.1, use repo_gpgcheck=0 instead of repo_gpgcheck=1 in the configuration above.

    2. Update your packages and install the Worker:

      sudo yum makecache
sudo yum install observability-pipelines-worker

    3. Add your keys and the site () to the Worker’s environment variables:

      sudo cat <<-EOF > /etc/default/observability-pipelines-worker
DD_API_KEY=<API_KEY>
DD_OP_PIPELINE_ID=<PIPELINE_ID>
DD_SITE=<SITE>
EOF

    4. Download the sample configuration file to /etc/observability-pipelines-worker/pipeline.yaml on the host. See Configurations for more information about the source, transform, and sink used in the sample configuration.

    5. Run the following command to start the Worker:

      sudo systemctl restart observability-pipelines-worker
    1. Download the the sample configuration.
    2. Set up the Worker module in your existing Terraform using the sample configuration. Make sure to update the values in vpc-id, subnet-ids, and region to match your AWS deployment in the configuration. Also,update the values in datadog-api-key and pipeline-id to match your pipeline.

    See Configurations for more information about the source, transform, and sink used in the sample configuration.

    See Working with Data for more information on transforming your data.

    Updating deployment modes

    파이프라인을 배포한 후 배포 방법을 변경할 수 있습니다. 예를 들어 수동 관리형 파이프라인에서 원격 구성이 활성화된 파이프라인으로 변경하거나 그 반대 방향으로도 바꿀 수 있습니다.

    원격 구성 배포에서 수동 관리형 배포로 바꾸는 방법:

    1. Observability Pipeline으로 이동해 파이프라인을 선택하세요.
    2. 톱니바퀴 아이콘을 클릭해 설정으로 이동하세요.
    3. Deployment Mode에서 manual을 선택해 활성화하세요.
    4. DD_OP_REMOTE_CONFIGURATION_ENABLED 플래그를 false로 설정하고 작업자를 재시작하세요. 이 플래그로 작업자를 재시작하지 않으면 원격 구성이 활성화된 상태로 계속 진행되며, 작업자가 로컬 구성 파일을 통해 수동으로 업데이트되지 않습니다.

    수동 관리형 배포에서 원격 구성 배포로 바꾸는 방법:

    1. Observability Pipeline으로 이동해 파이프라인을 선택하세요.
    2. 톱니바퀴 아이콘을 클릭해 설정으로 이동하세요.
    3. Deployment Mode에서 Remote Configuration을 선택해 활성화하세요.
    4. DD_OP_REMOTE_CONFIGURATION_ENABLED 플래그를 true로 설정하고 작업자를 재시작하세요. 이 플래그로 작업자를 재시작해야 UI에서 배포된 구성으로 폴링됩니다.
    5. 작업자가 새 버전 구성을 받도록 버전 내역에서 버전 하나를 배포하세요. 버전을 클릭하세요. Edit as Draft를 선택한 후 Deploy를 클릭하세요.

    Next steps

    The quickstart walked you through installing the Worker and deploying a sample pipeline configuration. For instructions on how to install the Worker to receive and route data from your Datadog Agents to Datadog or to receive and route data from your Splunk HEC to Splunk and Datadog, select your specific use case:

    Datadog
    Splunk

    For recommendations on deploying and scaling multiple Workers:

    Further reading

    추가 유용한 문서, 링크 및 기사:

    Working with data in Observability PipelinesDOCUMENTATION more more Deployment Design and Principles for the Observability Pipelines WorkerDOCUMENTATION more more Production deployment design and principles for the Observability Pipelines WorkerDOCUMENTATION more more Safe and Secure Local Processing with Observability PipelinesLEARNING CENTER more more