Update Existing Pipelines

Overview

For existing pipelines in Observability Pipelines, you can update and deploy changes for source settings, destination settings, and processors in the Observability Pipelines UI. But if you want to update source and destination environment variables, you need to manually update the Worker with the new values.

Update an existing pipeline

  1. Navigate to Observability Pipelines.
  2. Select the pipeline you want to update.
  3. Click Edit Pipeline in the top right corner.
  4. Make changes to the pipeline.
    • If you are updating the source or destination settings shown in the tiles, or updating and adding processors, make the changes and then click Deploy Changes.
    • To update source or destination environment variables, click Go to Worker Installation Steps and see Update source or destination variables for instructions.

Update source or destination variables

On the the Worker installation page:

  1. Select your platform in the Choose your installation platform dropdown menu.
  2. If you want to update source environment variables, update the information based on your selected logs source.
      • Datadog Agent address
        • The Observability Pipelines Worker listens to this socket address to receive logs from the Datadog Agent.
        • Stored in the environment variableDD_OP_SOURCE_DATADOG_AGENT_ADDRESS.
      • Splunk HEC address:
        • The bind address that your Observability Pipelines Worker listens on to receive logs originally intended for the Splunk indexer. For example, 0.0.0.0:8088
          Note: /services/collector/event is automatically appended to the endpoint.
        • Stored in the environment variable DD_OP_SOURCE_SPLUNK_HEC_ADDRESS.
      • Splunk TCP address
        • The Observability Pipelines Worker listens to this socket address to receive logs from the Splunk Forwarder. For example, 0.0.0.0:9997.
        • Stored in the environment variable DD_OP_SOURCE_SPLUNK_TCP_ADDRESS.
      • Sumo Logic address
        • The bind address that your Observability Pipelines Worker listens on to receive logs originally intended for the Sumo Logic HTTP Source. For example, 0.0.0.0:80.
          Note: /receiver/v1/http/ path is automatically appended to the endpoint.
        • Stored in the environment variable DD_OP_SOURCE_SUMO_LOGIC_ADDRESS.
    • If you want to update destination environment variables, update the information based on your selected logs destinations.

        • AWS access key ID of your S3 archive

          • The AWS access key ID for the S3 archive bucket.
          • Stored in the environment variable: DD_OP_DESTINATION_DATADOG_ARCHIVES_AWS_ACCESS_KEY_ID

        • AWS secret access key ID of your S3 archive

          • The AWS secret access key ID for the S3 archive bucket.
          • Stored in the environment variable DD_OP_DESTINATION_DATADOG_ARCHIVES_AWS_SECRET_KEY.

        No environment variables required.

        • Splunk HEC token
          • The Splunk HEC token for the Splunk indexer.
          • Stored in the environment variable DD_OP_DESTINATION_SPLUNK_HEC_TOKEN.
        • Base URL of the Splunk instance
          • The Splunk HTTP Event Collector endpoint your Observability Pipelines Worker sends processed logs to. For example, https://hec.splunkcloud.com:8088.
            Note: /services/collector/event path is automatically appended to the endpoint.
          • Stored in the environment variable DD_OP_DESTINATION_SPLUNK_HEC_ENDPOINT_URL.
        • Unique URL generated for the HTTP Logs and Metrics Source to receive log data
          • The Sumo Logic HTTP Source endpoint. The Observability Pipelines Worker sends processed logs to this endpoint. For example, https://<ENDPOINT>.collection.sumologic.com/receiver/v1/http/<UNIQUE_HTTP_COLLECTOR_CODE>, where:
            • <ENDPOINT> is your Sumo collection endpoint.
            • <UNIQUE_HTTP_COLLECTOR_CODE> is the string that follows the last forward slash (/) in the upload URL for the HTTP source.
          • Stored in the environment variable DD_OP_DESTINATION_SUMO_LOGIC_HTTP_COLLECTOR_URL.
      • Follow the instructions for your environment to update the worker:
          1. Click Select API key to choose the Datadog API key you want to use.
          2. Run the command provided in the UI to install the Worker. The command is automatically populated with the environment variables you entered earlier.
            docker run -i -e DD_API_KEY=<DATADOG_API_KEY> \
    -e DD_OP_PIPELINE_ID=<PIPELINE_ID> \
    -e DD_SITE=<DATADOG_SITE> \
    -e <SOURCE_ENV_VARIABLE> \
    -e <DESINATION_ENV_VARIABLE> \
    -p 8088:8088 \
    datadog/observability-pipelines-worker run
            Note: By default, the docker run command exposes the same port the Worker is listening on. If you want to map the Worker’s container port to a different port on the Docker host, use the -p | --publish option:
            -p 8282:8088 datadog/observability-pipelines-worker run
          3. Click Navigate Back to go back to the Observability Pipelines edit pipeline page.
          4. Click Deploy Changes.
          1. Download the Helm chart values file for AWS EKS.
          2. Click Select API key to choose the Datadog API key you want to use.
          3. Update the Datadog Helm chart to the latest version:
            helm repo update
          4. Run the command provided in the UI to install the Worker. The command is automatically populated with the environment variables you entered earlier.
            helm upgrade --install opw \
-f aws_eks.yaml \
--set datadog.apiKey=<DATADOG_API_KEY> \
--set datadog.pipelineId=<PIPELINE_ID> \
--set <SOURCE_ENV_VARIABLES> \
--set <DESTINATION_ENV_VARIABLES> \
--set service.ports[0].protocol=TCP,service.ports[0].port=<SERVICE_PORT>,service.ports[0].targetPort=<TARGET_PORT> \
datadog/observability-pipelines-worker
            Note: By default, the Kubernetes Service maps incoming port <SERVICE_PORT> to the port the Worker is listening on (<TARGET_PORT>). If you want to map the Worker’s pod port to a different incoming port of the Kubernetes Service, use the following service.ports[0].port and service.ports[0].targetPort values:
            --set service.ports[0].protocol=TCP,service.ports[0].port=8088,service.ports[0].targetPort=8282
          5. Click Navigate Back to go back to the Observability Pipelines edit pipeline page.
          6. Click Deploy Changes.
          1. Download the Helm chart values file for Azure AKS.
          2. Click Select API key to choose the Datadog API key you want to use.
          3. Add the Datadog chart repository to Helm:
            helm repo add datadog https://helm.datadoghq.com
            If you already have Datadog chart repository, run the following command to make sure it is up to date:
            helm repo update
          4. Run the command provided in the UI to install the Worker. The command is automatically populated with the environment variables you entered earlier.
            helm upgrade --install opw \
    -f azure_aks.yaml \
    --set datadog.apiKey=<DATADOG_API_KEY> \
    --set datadog.pipelineId=<PIPELINE_ID> \
    --set <SOURCE_ENV_VARIABLES> \
    --set <DESTINATION_ENV_VARIABLES> \
    --set service.ports[0].protocol=TCP,service.ports[0].port=<SERVICE_PORT>,service.ports[0].targetPort=<TARGET_PORT> \
    datadog/observability-pipelines-worker
            Note: By default, the Kubernetes Service maps incoming port <SERVICE_PORT> to the port the Worker is listening on (<TARGET_PORT>). If you want to map the Worker’s pod port to a different incoming port of the Kubernetes Service, use the following service.ports[0].port and service.ports[0].targetPort values:
            --set service.ports[0].protocol=TCP,service.ports[0].port=8088,service.ports[0].targetPort=8282
          5. Click Navigate Back to go back to the Observability Pipelines edit pipeline page.
          6. Click Deploy Changes.
          1. Download the Helm chart values file for Google GKE.
          2. Click Select API key to choose the Datadog API key you want to use.
          3. Add the Datadog chart repository to Helm:
            helm repo add datadog https://helm.datadoghq.com
            If you already have the Datadog chart repository, run the following command to make sure it is up to date:
            helm repo update
          4. Run the command provided in the UI to install the Worker. The command is automatically populated with the environment variables you entered earlier.
            helm upgrade --install opw \
    -f google_gke.yaml \
    --set datadog.apiKey=<DATADOG_API_KEY> \
    --set datadog.pipelineId=<PIPELINE_ID> \
    --set <SOURCE_ENV_VARIABLES> \
    --set <DESTINATION_ENV_VARIABLES> \
    --set service.ports[0].protocol=TCP,service.ports[0].port=<SERVICE_PORT>,service.ports[0].targetPort=<TARGET_PORT> \
    datadog/observability-pipelines-worker
            Note: By default, the Kubernetes Service maps incoming port <SERVICE_PORT> to the port the Worker is listening on (<TARGET_PORT>). If you want to map the Worker’s pod port to a different incoming port of the Kubernetes Service, use the following service.ports[0].port and service.ports[0].targetPort values:
            --set service.ports[0].protocol=TCP,service.ports[0].port=8088,service.ports[0].targetPort=8282
          5. Click Navigate Back to go back to the Observability Pipelines edit pipeline page.
          6. Click Deploy Changes.
          1. Click Select API key to choose the Datadog API key you want to use.
          2. Run the one-step command provided in the UI to re-install the Worker:
            DD_API_KEY=<DATADOG_API_KEY> DD_OP_PIPELINE_ID=<PIPELINES_ID> DD_SITE=<DATADOG_SITE> <SOURCE_ENV_VARIABLES> <DESTINATION_ENV_VARIABLES> bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_op_worker2.sh)"

          If you prefer not to use the one-line installation script, follow these step-by-step instructions:

          1. Run the following commands to update your local apt repo and install the latest Worker version:
            sudo apt-get update
sudo apt-get install observability-pipelines-worker datadog-signing-keys
          2. Add your keys, site (for example datadoghq.com for US1), source, and destination environment variables to the Worker’s environment file:
            sudo cat <<EOF > /etc/default/observability-pipelines-worker
DD_API_KEY=<DATADOG_API_KEY>
DD_OP_PIPELINE_ID=<PIPELINE_ID>
DD_SITE=<DATADOG_SITE>
<SOURCE_ENV_VARIABLES>
<DESTINATION_ENV_VARIABLES>
EOF
          3. Restart the worker:
            sudo systemctl restart observability-pipelines-worker
          4. Click Navigate Back to go back to the Observability Pipelines edit pipeline page.
          5. Click Deploy Changes.
          1. Click Select API key to choose the Datadog API key you want to use.
          2. Run the one-step command provided in the UI to re-install the Worker:
            DD_API_KEY=<DATADOG_API_KEY> DD_OP_PIPELINE_ID=<PIPELINE_ID> DD_SITE=<DATADOG_SITE> <SOURCE_ENV_VARIABLES> <DESTINATION_ENV_VARIABLES> bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_op_worker2.sh)"

          If you prefer not to use the one-line installation script, follow these step-by-step instructions:

          1. Update your packages and install the latest version of Worker:
            sudo yum makecache
sudo yum install observability-pipelines-worker
          2. Add your keys, site (for example datadoghq.com for US1), source, and destination updated environment variables to the Worker’s environment file:
            sudo cat <<-EOF > /etc/default/observability-pipelines-worker
DD_API_KEY=<API_KEY>
DD_OP_PIPELINE_ID=<PIPELINE_ID>
DD_SITE=<SITE>
<SOURCE_ENV_VARIABLES>
<DESTINATION_ENV_VARIABLES>
EOF
          3. Restart the worker:
            sudo systemctl restart observability-pipelines-worker
          4. Click Navigate Back to go back to the Observability Pipelines edit pipeline page.
          5. Click Deploy Changes.
          1. Select the expected log volume for the pipeline from the dropdown.
          2. Select the AWS region you want to use to install the Worker.
          3. Click Select API key to choose the Datadog API key you want to use.
          4. Click Launch CloudFormation Template to navigate to the AWS Console to review the stack configuration and then launch it. Make sure the CloudFormation parameters are set as expected.
          5. Select the VPC and subnet that you want to use to install the Worker.
          6. Review and check the necessary permissions checkboxes for IAM. Click Submit to create the stack. CloudFormation handles the installation at this point; the Worker instances are launched, the necessary software is downloaded, and the Worker starts automatically.
          7. Delete the previous CloudFormation stack and resources associated with it.
          8. Click Navigate Back to go back to the Observability Pipelines edit pipeline page.
          9. Click Deploy Changes.