--- title: Install BYOC Logs on AWS EKS description: Learn how to install and configure BYOC Logs on AWS EKS breadcrumbs: Docs > BYOC Logs > Install BYOC Logs > Install BYOC Logs on AWS EKS --- # Install BYOC Logs on AWS EKS {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} This document walks you through the process of configuring your AWS environment and installing BYOC (Bring Your Own Cloud) Logs on AWS EKS. ## Prerequisites{% #prerequisites %} To deploy BYOC Logs on AWS, you must configure: - AWS credentials and authentication. - AWS region selection. - IAM permissions for S3 object storage. - RDS PostgreSQL database (recommended). - EKS cluster. - See the [Cluster Sizing](https://docs.datadoghq.com/byoc-logs/operate/sizing.md) documentation for guidelines on planning your node groups for your expected TB/day. - If you are not using EKS auto-mode, the [EKS Pod Identity Agent](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html) and [EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) are required to use persistent volumes and claims from `searcher.persistentVolume` or `indexer.persistentVolume`. ### AWS credentials{% #aws-credentials %} When starting a node, BYOC Logs uses the default credential provider chain from the [AWS SDK for Rust](https://docs.aws.amazon.com/sdk-for-rust/latest/dg/credentials.html) to find AWS credentials in this order: 1. Environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, with `AWS_SESSION_TOKEN` as an optional addition for temporary credentials. 1. Credential profiles file, typically located at `~/.aws/credentials` or otherwise specified by the `AWS_SHARED_CREDENTIALS_FILE` and `AWS_PROFILE` environment variables if set and not empty. 1. Web Identity Token, loaded when `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set. This is the mechanism used by [EKS IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IRSA). To use IRSA, annotate the BYOC service account with `eks.amazonaws.com/role-arn` and configure an OIDC trust policy on the target IAM role. 1. Amazon ECS container credentials, loaded from the Amazon ECS container if the environment variable `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` is set. 1. Instance profile credentials, used on Amazon EC2 instances and delivered through the Amazon EC2 metadata service. An error is returned if no credentials are found in the chain. ### AWS Region{% #aws-region %} BYOC Logs attempts to find the AWS region from multiple sources, using the following order of precedence: 1. **Environment variables**: Checks `AWS_REGION`, then `AWS_DEFAULT_REGION`. 1. **AWS config file**: Typically located at `~/.aws/config`, or at the path specified by the `AWS_CONFIG_FILE` environment variable (if set and not empty). 1. **EC2 instance metadata**: Uses the region of the currently running Amazon EC2 instance. 1. **Default**: Falls back to `us-east-1` if no other source provides a region. {% alert level="warning" %} The resolved AWS region must match the region of the S3 bucket used for BYOC Logs storage. {% /alert %} ### IAM access for S3{% #iam-access-for-s3 %} Required authorized actions: - `ListBucket` (on the bucket directly) - `GetObject` - `PutObject` - `DeleteObject` - `ListMultipartUploadParts` - `AbortMultipartUpload` 1. Copy the following JSON to a file, replacing `` with your bucket name: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListMultipartUploadParts", "s3:AbortMultipartUpload" ], "Resource": [ "arn:aws:s3:::/*" ] } ] } ``` Create the IAM policy, replacing `` with the name of the file you created in Step 1: ```bash aws iam create-policy \ --policy-name \ --policy-document file:// \ --region ``` Get the policy ARN for role creation: ```bash POLICY_ARN=$(aws iam list-policies --query \ "Policies[?PolicyName==''].Arn" \ --output text) ``` Create the IAM role and service account: ```bash eksctl create iamserviceaccount \ --name byoclogs \ --namespace \ --cluster \ --role-name byoclogs \ --region \ --attach-policy-arn $POLICY_ARN \ --approve ``` This creates an IAM role called `byoclogs` and a Kubernetes service account called `byoclogs` in the `` namespace. If the namespace does not already exist, the command creates it. ### Create an RDS database{% #create-an-rds-database %} Create an RDS instance with the following command, replacing `` with your desired password: ```shell aws rds create-db-instance \ --db-instance-identifier byoclogs-demo-postgres \ --db-instance-class db.t4g.medium \ --engine postgres \ --engine-version 18.3 \ --master-username byoclogs \ --master-user-password '' \ --allocated-storage 20 \ --storage-type gp3 \ --db-subnet-group-name \ --vpc-security-group-ids \ --db-name byoclogs \ --backup-retention-period 0 \ --region \ --no-multi-az ``` You can retrieve RDS information by executing the following shell commands. The below commands retrieve information for the RDS instance created above: ```shell # Get RDS instance details RDS_INFO=$(aws rds describe-db-instances --db-instance-identifier byoclogs-demo-postgres --query 'DBInstances[0].{Status:DBInstanceStatus,Endpoint:Endpoint.Address,Port:Endpoint.Port,Database:DBName}' --output json 2>/dev/null) STATUS=$(echo $RDS_INFO | jq -r '.Status') ENDPOINT=$(echo $RDS_INFO | jq -r '.Endpoint') PORT=$(echo $RDS_INFO | jq -r '.Port') DATABASE=$(echo $RDS_INFO | jq -r '.Database') echo "" echo "Full URI:" # Replace with the master-user-password you set for the instance echo "postgres://byoclogs:@$ENDPOINT:$PORT/$DATABASE" echo "" ``` ## Installation steps{% #installation-steps %} 1. Install the BYOC Logs Helm chart 1. Verify installation ## Install the BYOC Logs Helm chart{% #install-the-byoc-logs-helm-chart %} 1. Add and update the Datadog Helm repository: ```shell helm repo add datadog https://helm.datadoghq.com helm repo update ``` 1. Store your Datadog API key as a Kubernetes secret: ```shell kubectl create secret generic datadog-secret \ -n \ --from-literal api-key="" ``` You can set a default namespace for your current context to avoid having to type `-n ` with every command: ``` kubectl config set-context --current --namespace= ``` 1. Store the PostgreSQL database connection string as a Kubernetes secret: ```shell kubectl create secret generic byoclogs-metastore-uri \ -n \ --from-literal QW_METASTORE_URI="postgres://:@:/" ``` 1. Customize the Helm chart Create a `datadog-values.yaml` file to override the default values with your custom configuration. This is where you define environment-specific settings such as the image tag, AWS account ID, service account, ingress setup, resource requests and limits, and more. Important alert (level: info): The `storageClass` (`sc`) used in the example file below is `gp3`, which is not installed by default and is not the default `sc` for EKS. To create the gp3 storage class, follow the instructions in this [AWS guide](https://docs.aws.amazon.com/eks/latest/userguide/create-storage-class.html). If you do not want to set gp3 as the default (and migrate from gp2), set `storageclass.kubernetes.io/is-default-class: "false"`. Datadog recommends gp3 storage volumes for BYOC Logs to provide the IOPS and throughput flexibility to support higher indexing rates. Any parameters not explicitly overridden in `datadog-values.yaml` fall back to the defaults defined [in the chart's `values.yaml`](https://github.com/DataDog/helm-charts/blob/main/charts/cloudprem/values.yaml). ```shell # Show default values helm show values datadog/cloudprem ``` Here is an example of a `datadog-values.yaml` file with overrides: ```yaml aws: accountId: "123456789012" # Environment variables # Any environment variables defined here are available to all pods in the deployment # Replace the "us-west-1" example below with your AWS region. environment: - name: AWS_REGION value: "us-west-1" # Datadog configuration datadog: # The Datadog [site](https://docs.datadoghq.com/getting_started/site/) to connect to. Defaults to `datadoghq.com`. # site: datadoghq.com # The name of the existing Secret containing the Datadog API key. The secret key name must be `api-key`. apiKeyExistingSecret: datadog-secret # Service account configuration # `serviceAccount.create: false` references the service account created with eksctl. # The chart uses this service account to access S3. serviceAccount: create: false name: byoclogs # eksRoleName is the name of the IAM role to use for the service account. # If serviceAccount.create is set to true, the following annotations will be added to the service account: # - eks.amazonaws.com/role-arn:arn:aws:iam:::role/ # - eks.amazonaws.com/sts-regional-endpoints:"true" eksRoleName: byoclogs extraAnnotations: {} # BYOC Logs node configuration config: # The root URI where index data is stored. This should be an S3 path. # All indexes created in BYOC Logs are stored under this location. default_index_root_uri: s3:///indexes # Internal ingress configuration for access within the VPC # The ingress provisions an Application Load Balancer (ALB) in AWS which is created in private subnets. # # Additional annotations can be added to customize the ALB behavior. ingress: internal: enabled: true name: byoclogs-internal host: byoclogs.example.internal extraAnnotations: alb.ingress.kubernetes.io/load-balancer-name: byoclogs-internal # Metastore configuration # The metastore is responsible for storing and managing index metadata. # It requires a PostgreSQL database connection string to be provided by a Kubernetes secret. # The secret should contain a key named `QW_METASTORE_URI` with a value in the format: # postgresql://:@:/ # # The metastore connection string is mounted into the pods using extraEnvFrom to reference the secret. metastore: extraEnvFrom: - secretRef: name: byoclogs-metastore-uri # Indexer configuration # The indexer is responsible for processing and indexing incoming data. It receives data from various sources (for example, Datadog Agents, log collectors) # and transforms it into searchable files called "splits" stored in S3. # # The indexer is horizontally scalable - you can increase `replicaCount` to handle higher indexing throughput. # The `podSize` parameter sets vCPU, memory, and component-specific settings automatically. # See the sizing guide for available tiers and their configurations. indexer: replicaCount: 2 podSize: xlarge persistentVolume: enabled: true storage: 250Gi storageClass: gp3 # Searcher configuration # The searcher is responsible for executing search queries against the indexed data stored in S3. # It handles search requests from Datadog's query service and returns matching results. # # The searcher is horizontally scalable - you can increase `replicaCount` to handle more concurrent searches. # Resource requirements for searchers are highly workload-dependent and should be determined empirically. # Key factors that impact searcher performance include: # - Query complexity (for example, number of terms, use of wildcards or regex) # - Query concurrency (number of simultaneous searches) # - Amount of data scanned per query # - Data access patterns (cache hit rates) # # Memory is particularly important for searchers as they cache frequently accessed index data in memory. searcher: replicaCount: 2 podSize: xlarge ``` 1. Install or upgrade the Helm chart: ```shell helm upgrade --install datadog/cloudprem \ -n \ -f datadog-values.yaml ``` Important alert (level: info): If a pod remains pending with a warning about insufficient memory or CPU and no available nodes, change `indexer.podSize` to `medium` in `datadog-values.yaml` and run the `helm upgrade --install` command again. ## Verification{% #verification %} ### Check deployment status{% #check-deployment-status %} Verify that all BYOC Logs components are running: ```shell kubectl get pods -n kubectl get ingress -n kubectl get services -n ``` ## Uninstall{% #uninstall %} To uninstall BYOC Logs: ```shell helm uninstall \ -n ``` ## Next step{% #next-step %} **[Set up log ingestion with Datadog Agent](https://docs.datadoghq.com/byoc-logs/ingest/agent.md)** - Configure the Datadog Agent to send logs to BYOC Logs ## Further reading{% #further-reading %} - [Configure BYOC Logs Ingress](https://docs.datadoghq.com/byoc-logs/configure/ingress.md) - [Configure Log Ingestion](https://docs.datadoghq.com/byoc-logs/ingest.md)