--- title: Send logs to BYOC Logs with Observability Pipelines description: >- Configure Observability Pipelines to send logs to BYOC Logs with optional dual shipping breadcrumbs: >- Docs > BYOC Logs > Set up Log Ingestion > Send logs to BYOC Logs with Observability Pipelines --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Send logs to BYOC Logs with Observability Pipelines {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} Observability Pipelines provides a flexible intermediary layer between your Datadog Agents and BYOC (Bring Your Own Cloud) Logs, allowing you to process, transform, and route logs before they reach your BYOC Logs deployment. Configure Observability Pipelines to receive logs from the Datadog Agent and forward them to BYOC Logs. For hybrid environments with logs split between on-premises data centers and the cloud, deploy an Observability Pipelines Worker in an on-premises data center and configure it to route those logs to the AWS data center that hosts BYOC Logs. BYOC Logs then collects the forwarded logs through the standard ingestion path. **Note**: This pattern requires an existing cloud footprint and does not support environments that run entirely on premises. To configure this setup: 1. **Create and configure the pipeline** - Define your pipeline configuration (source, processors, destination) in the Observability Pipelines UI. This creates the pipeline definition that will be used by the Worker. 1. **Deploy the Observability Pipelines Worker** - Install the Worker with your pipeline configuration using the Operator or a standalone Helm release. The Worker must be running and listening for logs before the Agent can connect to it. 1. **Configure the Datadog Agent** - Point the Agent to send logs to the deployed Worker. This step must come last because the Agent needs the Worker's address to be available. ## Create and configure an Observability Pipeline{% #create-and-configure-an-observability-pipeline %} 1. Navigate to [Observability Pipelines](https://app.datadoghq.com/observability-pipelines). 1. Select the [**Log Volume Control template**](https://docs.datadoghq.com/observability_pipelines/configuration/explore_templates.md?tab=logs#log-volume-control). 1. Set up your pipeline: 1. Choose the [**Datadog Agent** source](https://docs.datadoghq.com/observability_pipelines/sources/datadog_agent.md). 1. Remove any default processors from the pipeline. 1. Select the [**Datadog BYOC Logs** destination](https://docs.datadoghq.com/observability_pipelines/destinations/datadog_byoc_logs.md) to forward logs to your BYOC Logs instance. Leave the configuration empty. ## Deploy your Observability Pipelines{% #deploy-your-observability-pipelines %} After creating your pipeline in the UI, deploy the Observability Pipelines Worker. The Worker runs your pipeline configuration and listens for logs from the Datadog Agent. Choose the deployment method for your environment. The Operator manages Workers inside the BYOC Kubernetes cluster. ### Datadog Operator{% #datadog-operator %} For a new AWS deployment, follow [Install BYOC with the Datadog Operator](https://docs.datadoghq.com/byoc-logs/install/datadog_operator.md). The Operator does not create the pipeline in Datadog. Create it first and provide its ID in `spec.components.pipelines[].pipelineID`. Add the following entry to `spec.components.pipelines` in your `DatadogBYOCCluster` manifest. Keep the rest of the cluster configuration and any other pipeline entries: ```yaml spec: components: pipelines: - name: agent pipelineID: "" replicas: 2 ports: - name: datadog-agent port: 8282 ``` Use the exact port name `datadog-agent`. It tells the Operator to set `DD_OP_SOURCE_DATADOG_AGENT_ADDRESS` to `0.0.0.0:8282`. A different name does not configure this source address automatically. For other sources, use a [recognized port name](https://docs.datadoghq.com/containers/datadog_operator/crd_byoccluster.md#pipelines-and-ports) or set the source's `DD_OP_SOURCE_*_ADDRESS` environment variable in `spec.components.pipelines[].env`. Apply the complete manifest and wait for the workloads: ```shell kubectl apply -f byoc-cluster.yaml kubectl wait --for=condition=Available \ datadogbyoccluster/ -n --timeout=10m ``` The Operator creates a Worker and Service named `-pipeline-`, where `` is the pipeline entry's `name` (`agent` in this example). It supplies the indexer destination endpoint, Datadog site, and shared API key. Configure the Agent to use `http://-pipeline-..svc.cluster.local:8282`. Do not also install a standalone Worker Helm release for this pipeline entry. ### Standalone Worker Helm chart{% #standalone-worker-helm-chart %} The following Helm command installs or upgrades the Worker, configuring it to listen for logs and forward them to your BYOC Logs indexer. **Note**: You need the `pipelineId` from the pipeline you created in the previous step. This ID links the Worker to your pipeline configuration. ```shell helm upgrade --install opw \ -f values.yaml \ --set datadog.apiKey=XXXXXXX \ --set datadog.pipelineId=XXXXXXX \ --set env[0].name=DD_OP_SOURCE_DATADOG_AGENT_ADDRESS,env[0].value='0.0.0.0:8282' \ --set env[1].name=DD_OP_DESTINATION_CLOUDPREM_ENDPOINT_URL,env[1].value='http://-indexer..svc.cluster.local:7280' \ --set service.ports[0].name=dd-op-source-datadog-agent-address-port,service.ports[0].protocol=TCP,service.ports[0].port=8282,service.ports[0].targetPort=8282 \ datadog/observability-pipelines-worker ``` After a minute, verify that logs are flowing through the pipeline and reaching the BYOC Logs destination. This indicates that the Worker is running and ready to receive logs, and you can proceed to configure the Agent. ## Configure the Datadog Agent{% #configure-the-datadog-agent %} After the Observability Pipelines Worker is deployed and running, configure your Datadog Agent to send logs to it. The Agent connects to the Worker using the Worker's service address. For more information, see [Connect the Datadog Agent to the Observability Pipelines Worker](https://docs.datadoghq.com/observability_pipelines/sources/datadog_agent.md?tab=agenthelmvaluesfile#connect-the-datadog-agent-to-the-observability-pipelines-worker). Replace `` with the Service address for your deployment. For the Operator example in the previous section, use `-pipeline-..svc.cluster.local`. For a standalone Worker, use its Service address. Update your Datadog Agent configuration to send logs to the Observability Pipelines Worker: ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog spec: global: clusterName: your-cluster site: datadoghq.com credentials: apiSecret: secretName: datadog-secret keyName: api-key env: - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED value: "true" - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL value: "http://:8282" features: logCollection: enabled: true containerCollectAll: true ``` ## Verification{% #verification %} For an Operator-managed Worker, inspect its resource and logs: ```shell kubectl get datadogobservabilitypipelinesworker -pipeline- \ -n -o yaml kubectl logs statefulset/-pipeline- -n --tail=100 ``` For a standalone Worker, use its deployment's labels and namespace. For example: ```shell # Check Observability Pipelines worker status kubectl get pods -l app=observability-pipelines-worker # Check worker logs kubectl logs -l app=observability-pipelines-worker # Verify logs are reaching BYOC Logs kubectl exec -it -searcher-0 -n -- curl 'http://localhost:7280/api/v1/datadog/search?query=' ``` Send a log from the configured Agent, select the cluster's BYOC Logs index in Log Explorer, and confirm that the log appears. A ready Worker alone does not verify end-to-end ingestion. ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Datadog Agent Integration](https://docs.datadoghq.com/byoc-logs/ingest_logs/datadog_agent.md) - [REST API Integration](https://docs.datadoghq.com/byoc-logs/ingest_logs/rest_api.md) - [Observability Pipelines Overview](https://docs.datadoghq.com/observability_pipelines.md) - [Datadog BYOC Logs destination for Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/destinations/datadog_byoc_logs.md)