--- isPrivate: true title: >- (LEGACY) Set Up Observability Pipelines to Send Logs in Datadog-Rehydratable Format to Amazon S3 and Datadog description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: >- Docs > Observability Pipelines > (LEGACY) Observability Pipelines Documentation > (LEGACY) Set Up the Observability Pipelines Worker > (LEGACY) Set Up Observability Pipelines to Send Logs in Datadog-Rehydratable Format to Amazon S3 and Datadog --- # (LEGACY) Set Up Observability Pipelines to Send Logs in Datadog-Rehydratable Format to Amazon S3 and Datadog {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). (). {% /alert %} {% /callout %} {% alert level="warning" %} If you upgrade your OP Workers version 1.8 or below to version 2.0 or above, your existing pipelines will break. Do not upgrade your OP Workers if you want to continue using OP Workers version 1.8 or below. If you want to use OP Worker 2.0 or above, you must migrate your OP Worker 1.8 or earlier pipelines to OP Worker 2.x.Datadog recommends that you update to OP Worker versions 2.0 or above. Upgrading to a major OP Worker version and keeping it updated is the only supported way to get the latest OP Worker functionality, fixes, and security updates. {% /alert %} ## Overview{% #overview %} The [Observability Pipelines Worker](https://docs.datadoghq.com/observability_pipelines/legacy/) can collect, process, and route logs from any source to any destination. Using Datadog, you can build and manage all of your Observability Pipelines Worker deployments at scale. This guide walks you through deploying the Worker in your common tools cluster and configuring it to send logs in a Datadog-rehydratable format to a cloud storage for archiving. ## Deployment Modes{% #deployment-modes %} {% alert level="danger" %} Remote configuration for Observability Pipelines is in private beta. Contact [Datadog support](https://docs.datadoghq.com/help/) or your Customer Success Manager for access. {% /alert %} If you are enrolled in the private beta of [Remote Configuration](https://docs.datadoghq.com/agent/remote_config), you can remotely roll out changes to your Workers from the Datadog UI, rather than make updates to your pipeline configuration in a text editor and then manually rolling out your changes. Choose your deployment method when you create a pipeline and install your Workers. See Updating deployment modes on how to change the deployment mode after a pipeline is deployed. ## Assumptions{% #assumptions %} - You are already using Datadog and want to use Observability Pipelines. - You have administrative access to the clusters where the Observability Pipelines Worker is going to be deployed, as well as to the workloads that are going to be aggregated. - You have a common tools cluster or security cluster for your environment to which all other clusters are connected. ## Prerequisites{% #prerequisites %} Before installing, make sure you have: - A valid [Datadog API key](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys). - A Pipeline ID. You can generate both of these in [Observability Pipelines](https://app.datadoghq.com/observability-pipelines/create). ### Provider-specific requirements{% #provider-specific-requirements %} {% tab title="Docker" %} Ensure that your machine is configured to run Docker. {% /tab %} {% tab title="AWS EKS" %} To run the Worker on your Kubernetes nodes, you need a minimum of two nodes with one CPU and 512MB RAM available. Datadog recommends creating a separate node pool for the Workers, which is also the recommended configuration for production deployments. - The [EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) is required. To see if it is installed, run the following command and look for `ebs-csi-controller` in the list: ```shell kubectl get pods -n kube-system ``` - A `StorageClass` is required for the Workers to provision the correct EBS drives. To see if it is installed already, run the following command and look for `io2` in the list: ```shell kubectl get storageclass ``` If `io2` is not present, download [the StorageClass YAML](https://docs.datadoghq.com/resources/yaml/observability_pipelines/helm/storageclass.yaml) and `kubectl apply` it. - The [AWS Load Balancer controller](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) is required. To see if it is installed, run the following command and look for `aws-load-balancer-controller` in the list: ```shell helm list -A ``` - Datadog recommends using Amazon EKS >= 1.16. See [Best Practices for OPW Aggregator Architecture](https://docs.datadoghq.com/observability_pipelines/legacy/architecture/) for production-level requirements. {% /tab %} {% tab title="APT-based Linux" %} There are no provider-specific requirements for APT-based Linux. {% /tab %} {% tab title="RPM-based Linux" %} There are no provider-specific requirements for APT-based Linux. {% /tab %} {% tab title="Terraform (AWS)" %} To run the Worker in your AWS account, you need administrative access to that account and the following information: - The VPC ID your instances will run in. - The subnet IDs your instances will run in. - The AWS region your VPC is located in. {% /tab %} ## Set up Log Archives{% #set-up-log-archives %} When you install the Observability Pipelines Worker later on, the sample configuration provided includes a sink for sending logs to Amazon S3 under a Datadog-rehydratable format. To use this configuration, create an S3 bucket for your archives and set up an IAM policy that allows the Workers to write to the S3 bucket. Then, connect the S3 bucket to Datadog Log Archives. {% callout %} # Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com See [AWS Pricing](https://aws.amazon.com/s3/pricing/) for inter-region data transfer fees and how cloud storage costs may be impacted. {% /callout %} ### Create an S3 bucket and set up an IAM policy{% #create-an-s3-bucket-and-set-up-an-iam-policy %} {% tab title="Docker" %} 1. Navigate to [Amazon S3](https://s3.console.aws.amazon.com/s3/). Create an S3 bucket to send your archives to. Do not make your bucket publicly readable. 1. Create a policy with the following permissions. Make sure to update the bucket name to the name of the S3 bucket you created earlier. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject"], "Resource": "arn:aws:s3:::/*" }, { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::" } ] } ``` Create an IAM user and attach the above policy to it. Create access credentials for the IAM user. Save these credentials as `AWS_ACCESS_KEY` and `AWS_SECRET_ACCESS_KEY`. {% /tab %} {% tab title="AWS EKS" %} 1. Navigate to [Amazon S3](https://s3.console.aws.amazon.com/s3/). Create an S3 bucket to send your archives to. Do not make your bucket publicly readable. 1. Create a policy with the following permissions. Make sure to update the bucket name to the name of the S3 bucket you created earlier. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject"], "Resource": "arn:aws:s3:::/*" }, { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::" } ] } ``` [Create a service account](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) to use the policy you created above. {% /tab %} {% tab title="APT-based Linux" %} 1. Navigate to [Amazon S3](https://s3.console.aws.amazon.com/s3/). Create an S3 bucket to send your archives to. Do not make your bucket publicly readable. 1. Create a policy with the following permissions. Make sure to update the bucket name to the name of the S3 bucket you created earlier. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject"], "Resource": "arn:aws:s3:::/*" }, { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::" } ] } ``` Create an IAM user and attach the above policy to it. Create access credentials for the IAM user. Save these credentials as `AWS_ACCESS_KEY` and `AWS_SECRET_ACCESS_KEY`. {% /tab %} {% tab title="RPM-based Linux" %} 1. Navigate to [Amazon S3](https://s3.console.aws.amazon.com/s3/). Create an S3 bucket to send your archives to. Do not make your bucket publicly readable. 1. Create a policy with the following permissions. Make sure to update the bucket name to the name of the S3 bucket you created earlier. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject"], "Resource": "arn:aws:s3:::/*" }, { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::" } ] } ``` Create an IAM user and attach the policy above to it. Create access credentials for the IAM user. Save these credentials as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. {% /tab %} {% tab title="Terraform (AWS)" %} 1. Navigate to [Amazon S3](https://s3.console.aws.amazon.com/s3/). Create an S3 bucket to send your archives to. Do not make your bucket publicly readable. 1. Create a policy with the following permissions. Make sure to update the bucket name to the name of the S3 bucket you created earlier. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject"], "Resource": "arn:aws:s3:::/*" }, { "Sid": "DatadogUploadAndRehydrateLogArchives", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::" } ] } ``` Attach the policy to the IAM Instance Profile that is created with Terraform, which you can find under the `iam-role-name` output. {% /tab %} ### Connect the S3 bucket to Datadog Log Archives{% #connect-the-s3-bucket-to-datadog-log-archives %} You need to connect the S3 bucket you created earlier to Datadog Log Archives so that you can rehydrate the archives later on. 1. Navigate to Datadog [Log Forwarding](https://app.datadoghq.com/logs/pipelines/log-forwarding). 1. Click **+ New Archive**. 1. Enter a descriptive archive name. 1. Add a query that filters out all logs going through log pipelines so that those logs do not go into this archive. For example, add the query `observability_pipelines_read_only_archive`, assuming that no logs going through the pipeline have that tag added. 1. Select **AWS S3**. 1. Select the AWS Account that your bucket is in. 1. Enter the name of the S3 bucket. 1. Optionally, enter a path. 1. Check the confirmation statement. 1. Optionally, add tags and define the maximum scan size for rehydration. See [Advanced settings](https://docs.datadoghq.com/logs/log_configuration/archives/#advanced-settings) for more information. 1. Click **Save**. See the [Log Archives documentation](https://docs.datadoghq.com/logs/log_configuration/archives) for additional information. ### Install the Observability Pipelines Worker{% #install-the-observability-pipelines-worker %} {% tab title="Docker" %} The Observability Pipelines Worker Docker image is published to Docker Hub [here](https://hub.docker.com/r/datadog/observability-pipelines-worker). 1. Download the [sample pipeline configuration file](https://docs.datadoghq.com/resources/yaml/observability_pipelines/archives/pipeline.yaml). 1. Run the following command to start the Observability Pipelines Worker with Docker: ```shell docker run -i -e DD_API_KEY= \ -e DD_OP_PIPELINE_ID= \ -e DD_SITE= \ -e AWS_ACCESS_KEY_ID= \ -e AWS_SECRET_ACCESS_KEY= \ -e DD_ARCHIVES_BUCKET= \ -e DD_ARCHIVES_SERVICE_ACCOUNT= \ -p 8282:8282 \ -v ./pipeline.yaml:/etc/observability-pipelines-worker/pipeline.yaml:ro \ datadog/observability-pipelines-worker run ``` Replace these placeholders with the following information: - `` with your Datadog API key. - `` with your Observability Pipelines configuration ID. - `` with . - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` with the AWS credentials you created earlier. - `` with the name of the S3 bucket storing the logs. - `` with the [AWS region](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints) of the target service. - `./pipeline.yaml` must be the relative or absolute path to the configuration you downloaded in step 1. {% /tab %} {% tab title="AWS EKS" %} 1. Download the [Helm chart values file](https://docs.datadoghq.com/resources/yaml/observability_pipelines/archives/aws_eks.yaml) for AWS EKS. 1. In the Helm chart, replace these placeholders with the following information: - `datadog.apiKey` with your Datadog API key. - `datadog.pipelineId` with your Observability Pipelines configuration ID. - `site` with . - `${DD_ARCHIVES_SERVICE_ACCOUNT}` in `serviceAccount.name` with the service account name. - `${DD_ARCHIVES_BUCKET}` in `pipelineConfig.sinks.datadog_archives` with the name of the S3 bucket storing the logs. - `${DD_ARCHIVES_SERVICE_ACCOUNT}` in `pipelineConfig.sinks.datadog_archives` with the [AWS region](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints) of the target service. 1. Install it in your cluster with the following commands: ```shell helm repo add datadog https://helm.datadoghq.com ``` ```shell helm repo update ``` ```shell helm upgrade --install \ opw datadog/observability-pipelines-worker \ -f aws_eks.yaml ``` {% /tab %} {% tab title="APT-based Linux" %} 1. Run the following commands to set up APT to download through HTTPS: ``` sudo apt-get update sudo apt-get install apt-transport-https curl gnupg ``` 1. Run the following commands to set up the Datadog `deb` repo on your system and create a Datadog archive keyring: ``` sudo sh -c "echo 'deb [signed-by=/usr/share/keyrings/datadog-archive-keyring.gpg] https://apt.datadoghq.com/ stable observability-pipelines-worker-1' > /etc/apt/sources.list.d/datadog-observability-pipelines-worker.list" sudo touch /usr/share/keyrings/datadog-archive-keyring.gpg sudo chmod a+r /usr/share/keyrings/datadog-archive-keyring.gpg curl https://keys.datadoghq.com/DATADOG_APT_KEY_CURRENT.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch curl https://keys.datadoghq.com/DATADOG_APT_KEY_06462314.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch curl https://keys.datadoghq.com/DATADOG_APT_KEY_F14F620E.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch curl https://keys.datadoghq.com/DATADOG_APT_KEY_C0962C7D.public | sudo gpg --no-default-keyring --keyring /usr/share/keyrings/datadog-archive-keyring.gpg --import --batch ``` 1. Run the following commands to update your local `apt` repo and install the Worker: ``` sudo apt-get update sudo apt-get install observability-pipelines-worker datadog-signing-keys ``` 1. Add your keys and the site () to the Worker's environment variables. Replace `` with the name of the S3 bucket storing the logs and `` with the [AWS region](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints) of the target service. ``` sudo cat <<-EOF > /etc/default/observability-pipelines-worker AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= DD_ARCHIVES_BUCKET= DD_ARCHIVES_SERVICE_ACCOUNT= EOF ``` 1. Download the [sample configuration file](https://docs.datadoghq.com/resources/yaml/observability_pipelines/archives/pipeline.yaml) to `/etc/observability-pipelines-worker/pipeline.yaml` on the host. 1. Start the worker: ``` sudo systemctl restart observability-pipelines-worker ``` {% /tab %} {% tab title="RPM-based Linux" %} 1. Run the following commands to set up the Datadog `rpm` repo on your system: ``` cat < /etc/yum.repos.d/datadog-observability-pipelines-worker.repo [observability-pipelines-worker] name = Observability Pipelines Worker baseurl = https://yum.datadoghq.com/stable/observability-pipelines-worker-1/\$basearch/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://keys.datadoghq.com/DATADOG_RPM_KEY_CURRENT.public https://keys.datadoghq.com/DATADOG_RPM_KEY_4F09D16B.public https://keys.datadoghq.com/DATADOG_RPM_KEY_B01082D3.public https://keys.datadoghq.com/DATADOG_RPM_KEY_FD4BF915.public https://keys.datadoghq.com/DATADOG_RPM_KEY_E09422B3.public EOF ``` **Note:** If you are running RHEL 8.1 or CentOS 8.1, use `repo_gpgcheck=0` instead of `repo_gpgcheck=1` in the configuration above. 1. Update your packages and install the Worker: ``` sudo yum makecache sudo yum install observability-pipelines-worker ``` 1. Add your keys and the site () to the Worker's environment variables. Replace `` with the name of the S3 bucket storing the logs and `` with the [AWS region](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints) of the target service. ``` sudo cat <<-EOF > /etc/default/observability-pipelines-worker AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= DD_ARCHIVES_BUCKET= DD_ARCHIVES_SERVICE_ACCOUNT= EOF ``` 1. Download the [sample configuration file](https://docs.datadoghq.com/resources/yaml/observability_pipelines/archives/pipeline.yaml) to `/etc/observability-pipelines-worker/pipeline.yaml` on the host. 1. Start the worker: ``` sudo systemctl restart observability-pipelines-worker ``` {% /tab %} {% tab title="Terraform (AWS)" %} 1. Download the [sample configuration](https://docs.datadoghq.com/resources/yaml/observability_pipelines/archives/terraform_opw_archives.tf). 1. Set up the Worker module in your existing Terraform using the sample configuration. Make sure to update the values in `vpc-id`, `subnet-ids`, and `region` to match your AWS deployment in the configuration. Also, update the values in `datadog-api-key` and `pipeline-id` to match your pipeline. {% /tab %} ### Load balancing{% #load-balancing %} {% tab title="Docker" %} Production-oriented setup is not included in the Docker instructions. Instead, refer to your company's standards for load balancing in containerized environments. If you are testing on your local machine, configuring a load balancer is unnecessary. {% /tab %} {% tab title="AWS EKS" %} Use the load balancers provided by your cloud provider. The load balancers adjust based on autoscaling events that the default Helm setup is configured for. The load balancers are internal-facing, so they are only accessible inside your network. Use the load balancer URL given to you by Helm when you configure the Datadog Agent. NLBs provisioned by the [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/) are used. See [Capacity Planning and Scaling](https://docs.datadoghq.com/observability_pipelines/legacy/architecture/capacity_planning_scaling/) for load balancer recommendations when scaling the Worker. #### Cross-availability-zone load balancing{% #cross-availability-zone-load-balancing %} The provided Helm configuration tries to simplify load balancing, but you must take into consideration the potential price implications of cross-AZ traffic. Wherever possible, the samples try to avoid creating situations where multiple cross-AZ hops can happen. The sample configurations do not enable the cross-zone load balancing feature available in this controller. To enable it, add the following annotation to the `service` block: ```gdscript3 service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true ``` See [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/service/annotations/#load-balancer-attributes) for more details. {% /tab %} {% tab title="APT-based Linux" %} Given the single-machine nature of the installation, there is no built-in support for load-balancing. Provision your own load balancers using your company's standard. {% /tab %} {% tab title="RPM-based Linux" %} Given the single-machine nature of the installation, there is no built-in support for load-balancing. You need to provision your own load balancers based on your company's standard. {% /tab %} {% tab title="Terraform (AWS)" %} The Terraform module provisions an NLB to point at the instances. The DNS address is returned in the `lb-dns` output in Terraform. {% /tab %} ### Buffering{% #buffering %} Observability Pipelines includes multiple buffering strategies that allow you to increase the resilience of your cluster to downstream faults. The provided sample configurations use disk buffers, the capacities of which are rated for approximately 10 minutes of data at 10Mbps/core for Observability Pipelines deployments. That is often enough time for transient issues to resolve themselves, or for incident responders to decide what needs to be done with the observability data. {% tab title="Docker" %} By default, the Observability Pipelines Worker's data directory is set to `/var/lib/observability-pipelines-worker`. Make sure that your host machine has a sufficient amount of storage capacity allocated to the container's mountpoint. {% /tab %} {% tab title="AWS EKS" %} For AWS, Datadog recommends using the `io2` EBS drive family. Alternatively, the `gp3` drives could also be used. {% /tab %} {% tab title="APT-based Linux" %} By default, the Observability Pipelines Worker's data directory is set to `/var/lib/observability-pipelines-worker` - if you are using the sample configuration, you should ensure that this has at least 288GB of space available for buffering. Where possible, it is recommended to have a separate SSD mounted at that location. {% /tab %} {% tab title="RPM-based Linux" %} By default, the Observability Pipelines Worker's data directory is set to `/var/lib/observability-pipelines-worker` - if you are using the sample configuration, you should ensure that this has at least 288GB of space available for buffering. Where possible, it is recommended to have a separate SSD mounted at that location. {% /tab %} {% tab title="Terraform (AWS)" %} By default, a 288GB EBS drive is allocated to each instance, and the sample configuration above is set to use that for buffering. {% /tab %} ## Connect the Datadog Agent to the Observability Pipelines Worker{% #connect-the-datadog-agent-to-the-observability-pipelines-worker %} To send Datadog Agent logs to the Observability Pipelines Worker, update your agent configuration with the following: ```yaml observability_pipelines_worker: logs: enabled: true url: "http://:8282" ``` `OPW_HOST` is the IP of the load balancer or machine you set up earlier. For single-host Docker-based installs, this is the IP address of the underlying host. For Kubernetes-based installs, you can retrieve it by running the following command and copying the `EXTERNAL-IP`: ```shell kubectl get svc opw-observability-pipelines-worker ``` For Terraform installs, the `lb-dns` output provides the necessary value. At this point, your observability data should be going to the Worker and then sent along to your S3 archive. ## Updating deployment modes{% #updating-deployment-modes %} After deploying a pipeline, you can also switch deployment methods, such as going from a manually managed pipeline to a remote configuration enabled pipeline or vice versa. If you want to switch from a remote configuration deployment to a manually managed deployment: 1. Navigate to Observability Pipelines and select the pipeline. 1. Click the settings cog. 1. In **Deployment Mode**, select **manual** to enable it. 1. Set the `DD_OP_REMOTE_CONFIGURATION_ENABLED` flag to `false` and restart the Worker. Workers that are not restarted with this flag continue to be remote configuration enabled, which means that the Workers are not updated manually through a local configuration file. If you want to switch from manually managed deployment to a remote configuration deployment: 1. Navigate to Observability Pipelines and select the pipeline. 1. Click the settings cog. 1. In **Deployment Mode**, select **Remote Configuration** to enable it. 1. Set the `DD_OP_REMOTE_CONFIGURATION_ENABLED` flag to `true` and restart the Worker. Workers that are not restarted with this flag are not polled for configurations deployed in the UI. 1. Deploy a version in your version history, so that the Workers receive the new version configuration. Click on a version. Click **Edit as Draft** and then click **Deploy**. ## Rehydrate your archives{% #rehydrate-your-archives %} See [Rehydrating from Archives](https://docs.datadoghq.com/logs/log_configuration/rehydrating/) for instructions on how to rehydrate your archive in Datadog so that you can start analyzing and investigating those logs. ## Further reading{% #further-reading %} - [Production deployment design and principles for the Observability Pipelines Worker](https://docs.datadoghq.com/observability_pipelines/legacy/production_deployment_overview/) - [Safe and Secure Local Processing with Observability Pipelines](https://dtdg.co/d22op)