--- title: Send OpenTelemetry logs with Observability Pipelines description: >- A quick guide to send OpenTelemetry logs through Observability Pipelines to CloudPrem in less than 5 minutes breadcrumbs: >- Docs > CloudPrem > CloudPrem Guides > Send OpenTelemetry logs with Observability Pipelines --- # Send OpenTelemetry logs with Observability Pipelines {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). (). {% /alert %} {% /callout %} {% callout %} ##### CloudPrem is in Preview Join the CloudPrem Preview to access new self-hosted log management features. [Request Access](https://www.datadoghq.com/product-preview/cloudprem/) {% /callout %} ## Overview{% #overview %} CloudPrem supports log ingestion from OTEL collectors by using Observability Pipelines as the ingestion layer. This guide provides step-by-step instructions to connect OTEL logs to CloudPrem—without disrupting your existing OTEL configuration. By the end of this guide, you will be able to: 1. Start CloudPrem locally. 1. Create an Observability Pipeline with a custom processor to add tags. 1. Run the Observability Pipelines Worker. 1. Send OpenTelemetry logs using the Python SDK. 1. View tagged logs in Datadog. ## Prerequisites{% #prerequisites %} - [CloudPrem Preview](https://www.datadoghq.com/product-preview/cloudprem/) access. - **Datadog API Key**: [Get your API key](https://docs.datadoghq.com/account_management/api-app-keys/#add-an-api-key-or-client-token). - **Datadog Application Key**: [Get your application key](https://docs.datadoghq.com/account_management/api-app-keys/#add-application-keys). - **Docker**: [Install Docker](https://docs.docker.com/get-docker/). - **Python 3 and pip**: For sending test OTLP logs. ## Step 1: Start CloudPrem{% #step-1-start-cloudprem %} Start a local CloudPrem instance. Replace `` with your Datadog API Key: ```shell export DD_API_KEY="" export DD_SITE="datadoghq.com" docker run -d \ --name cloudprem \ -v $(pwd)/qwdata:/quickwit/qwdata \ -e DD_SITE=${DD_SITE} \ -e DD_API_KEY=${DD_API_KEY} \ -p 127.0.0.1:7280:7280 \ datadog/cloudprem run ``` ## Step 2: Create an Observability Pipeline with the API{% #step-2-create-an-observability-pipeline-with-the-api %} Create a pipeline with an OpenTelemetry source, a filter processor, and a CloudPrem destination. Replace `` with your Datadog Application Key: ```shell export DD_APP_KEY="" curl -s -X POST "https://api.${DD_SITE}/api/v2/obs-pipelines/pipelines" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d '{ "data": { "attributes": { "name": "OTEL to CloudPrem Pipeline", "config": { "sources": [ { "id": "otel-source", "type": "opentelemetry" } ], "processor_groups": [ { "id": "main-processors", "enabled": true, "include": "*", "inputs": ["otel-source"], "processors": [ { "id": "add-tags", "display_name": "Add tags", "enabled": true, "type": "custom_processor", "include": "*", "remaps": [ { "drop_on_error": false, "enabled": true, "include": "*", "name": "ddtags", "source": ".ddtags = [\"pipeline:observability-pipelines\", \"source:opentelemetry\"]" } ] } ] } ], "destinations": [ { "id": "cloudprem-dest", "type": "cloud_prem", "inputs": ["main-processors"] } ] } }, "type": "pipelines" } }' | jq -r '.data.id' ``` This command returns the `pipeline_id`. Save it for the next step. **Note**: The custom processor adds a `ddtags` field with custom tags to all logs through the `remaps` configuration. ## Step 3: Run the Observability Pipelines Worker{% #step-3-run-the-observability-pipelines-worker %} Start the Observability Pipelines Worker using Docker. Replace `` with the ID from Step 2: ```shell export PIPELINE_ID="" docker run -d \ --name opw \ -p 4317:4317 \ -p 4318:4318 \ -e DD_API_KEY=${DD_API_KEY} \ -e DD_SITE=${DD_SITE} \ -e DD_OP_PIPELINE_ID=${PIPELINE_ID} \ -e DD_OP_SOURCE_OTEL_GRPC_ADDRESS="0.0.0.0:4317" \ -e DD_OP_SOURCE_OTEL_HTTP_ADDRESS="0.0.0.0:4318" \ -e DD_OP_DESTINATION_CLOUDPREM_ENDPOINT_URL="http://host.docker.internal:7280" \ datadog/observability-pipelines-worker run ``` **Notes**: - The Worker exposes port 4318 for HTTP and 4317 for gRPC. - On macOS/Windows, use `host.docker.internal` to connect to CloudPrem on the host machine. - On Linux, use `--network host` instead of `-p` flags and `http://localhost:7280` for the endpoint. {% image source="https://datadog-docs.imgix.net/images/cloudprem/guides/otel-op-cloudprem/op-config.e56307c13009897e1cb130c5eb23bfe8.png?auto=format" alt="The Observability Pipelines configuration" /%} ## Step 4: Send logs through Observability Pipelines{% #step-4-send-logs-through-observability-pipelines %} Install the OpenTelemetry SDK and send a test log to the Observability Pipelines Worker: ```shell pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http python3 -c " import time, logging from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler from opentelemetry.sdk._logs.export import BatchLogRecordProcessor from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter from opentelemetry.sdk.resources import Resource exporter = OTLPLogExporter(endpoint='http://localhost:4318/v1/logs') resource = Resource.create({'service.name': 'otel-demo'}) log_provider = LoggerProvider(resource=resource) log_provider.add_log_record_processor(BatchLogRecordProcessor(exporter)) handler = LoggingHandler(logger_provider=log_provider) logging.getLogger().addHandler(handler) logging.getLogger().setLevel(logging.INFO) logging.info('Hello from OpenTelemetry via Observability Pipelines!') time.sleep(2) log_provider.shutdown() print('✓ Log sent successfully!') " ``` For production, configure your OpenTelemetry Collector to forward logs to the Worker: ```yaml exporters: otlphttp: endpoint: http://localhost:4318 service: pipelines: logs: receivers: [otlp] exporters: [otlphttp] ``` ## Verify the pipeline and CloudPrem{% #verify-the-pipeline-and-cloudprem %} Check that all components are running: ```shell # Check CloudPrem status docker logs cloudprem --tail 20 # Check Observability Pipelines Worker status docker logs opw --tail 20 ``` ## Step 5: View logs in Datadog{% #step-5-view-logs-in-datadog %} 1. Go to the [Datadog Log Explorer](https://app.datadoghq.com/logs). 1. In the left facet panel, select your CloudPrem index under **CLOUDPREM INDEXES**. 1. You should see your OpenTelemetry logs from the `otel-demo` service with custom tags: `pipeline:observability-pipelines` and `source:opentelemetry`. {% image source="https://datadog-docs.imgix.net/images/cloudprem/guides/otel-op-cloudprem/cloudprem_logs.f7f7508d934acb3b009f3c2088d3a3b7.png?auto=format" alt="CloudPrem logs available in the Datadog Log Explorer" /%} ## Next steps{% #next-steps %} - Configure your OpenTelemetry Collector or instrumented applications to send logs to the Worker. - Add more processors to your pipeline (sampling, enrichment, transformation). - Scale the Worker deployment for production workloads. - See [Observability Pipelines documentation](https://docs.datadoghq.com/observability_pipelines/) for advanced configurations. ## Cleanup{% #cleanup %} To stop and remove the containers: ```shell docker stop cloudprem opw docker rm cloudprem opw ``` ## Further reading{% #further-reading %} - [CloudPrem Quickstart](https://docs.datadoghq.com/cloudprem/quickstart/) - [OpenTelemetry Source for Observability Pipelines](https://docs.datadoghq.com/observability_pipelines/sources/opentelemetry/) - [Send logs to CloudPrem with Observability Pipelines](https://docs.datadoghq.com/cloudprem/ingest/observability_pipelines/)