Overview

This document walks you through the process of configuring your Azure environment and installing CloudPrem on Azure AKS.

Prerequisites

Before you install CloudPrem on Azure, you must set up a set of supporting infrastructure resources. These components provide the foundational compute, storage, database, and networking services that CloudPrem depends on.

Infrastructure Requirements

Here are the components you must provision:

• Azure Kubernetes Service (AKS): A running AKS cluster sized for your expected CloudPrem workload.
• PostgreSQL Flexible Server: An Azure Database for PostgreSQL instance that CloudPrem will use to store its metadata.
• Blob Storage container: An Azure Storage container to hold CloudPrem logs.
• Client Identity and permissions: An Azure AD application with read/write access to the storage container.
• NGINX Ingress Controller: Installed on the AKS cluster to route external traffic to CloudPrem services.
• Datadog Agent: Deployed on the AKS cluster to collect and send logs to CloudPrem.

Azure Kubernetes Service (AKS)

CloudPrem runs entirely on Kubernetes. You need an AKS cluster with sufficient CPU, memory, and disk space configured for your workload. See the Kubernetes cluster sizing recommendations for guidance.

Deploy the AKS cluster

• Deploy an AKS cluster with the Azure CLI
• Deploy an AKS cluster with Terraform

Verify cluster connectivity and health

To confirm the cluster is reachable and nodes are in the Ready state, run the following command:

kubectl get nodes -o wide

Azure PostgreSQL Flexible Server

CloudPrem stores its metadata and configuration in a PostgreSQL database. Datadog recommends the Azure Database for PostgreSQL Flexible Server. It must be reachable from the AKS cluster, ideally with private networking enabled. See the Postgres sizing recommendations for details.

Create the PostgreSQL database

• Create an Azure Database for PostgreSQL Flexible Server using the Azure CLI
• Create an Azure Database for PostgreSQL Flexible Server using Terraform

Verify database connectivity

Connect to your PostgreSQL database from within the AKS network using the PostgreSQL client, psql. First, start an interactive pod in your Kubernetes cluster using an image that includes psql:

kubectl run psql-client \
  -n <NAMESPACE_NAME> \
  --rm -it \
  --image=bitnami/postgresql:latest \
  --command -- bash

Then, run the following command directly from the shell, replacing the placeholder values with your actual values:

psql "host=<HOST> \
      port=<PORT> \
      dbname=<DATABASE> \
      user=<USERNAME> \
      password=<PASSWORD>"

If successful, you should see a prompt similar to:

psql (15.2)
SSL connection (protocol: TLS...)
Type "help" for help.

<DATABASE>=>

Blob Storage Container

CloudPrem uses Azure Blob Storage to persist logs. Create a dedicated container for this purpose.

Create a Blob Storage container

Use a dedicated container per environment (for example, cloudprem-prod, cloudprem-staging), and assign least-privilege RBAC roles at the container level, rather than at the storage account scope.

• Create an Blob Storage container using the Azure CLI
• Create a Blob Storage container using Terraform

Client Identity and permissions

An Azure AD application must be granted read/write access to the Blob Storage container. Register a dedicated application for CloudPrem and assign the corresponding service principal the Contributor role on the Blob Storage container created above.

Register the application

Register an application in Microsoft Entra ID

Assign Contributor role

Assign an Azure role for access to blob data

NGINX Ingress Controller

Public NGINX Ingress Controller

The public ingress is essential for enabling Datadog’s control plane and query service to manage and query CloudPrem clusters over the public internet. It provides secure access to the CloudPrem gRPC API through the following mechanisms:

• Creates an internet-facing Azure Load Balancer that accepts traffic from Datadog services
• Implements TLS encryption with termination at the ingress controller level
• Uses HTTP/2 (gRPC) for communication between Datadog and CloudPrem clusters
• Requires mutual TLS (mTLS) authentication where Datadog services must present valid client certificates
• Configures the controller in TLS passthrough mode to forward client certificates to CloudPrem pods with the ssl-client-cert header
• Rejects requests that are missing valid client certificates or the certificate header

Use the following nginx-public.yaml Helm values file in order to create the public NGINX Ingress Controller:

controller:
  electionID: public-ingress-controller-leader
  ingressClass: nginx-public
  ingressClassResource:
    name: nginx-public
    enabled: true
    default: false
    controllerValue: k8s.io/public-ingress-nginx
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz

Then, install the controller with Helm using the following command:

helm upgrade --install nginx-public ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace nginx-ingress-public \
  --create-namespace \
  -f nginx-public.yaml

Verify that the controller pod is running:

kubectl get pods -n nginx-ingress-public -l app.kubernetes.io/component=controller

Verify that the service exposes an external IP:

kubectl get svc -n nginx-ingress-public -l app.kubernetes.io/component=controller

Internal NGINX Ingress Controller

The internal ingress enables log ingestion from Datadog Agents and other log collectors within your environment through HTTP. Use the following nginx-internal.yaml Helm values file in order to create the public NGINX Ingress Controller:

controller:
  electionID: internal-ingress-controller-leader
  ingressClass: nginx-internal
  ingressClassResource:
    name: nginx-internal
    enabled: true
    default: false
    controllerValue: k8s.io/internal-ingress-nginx
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: true
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz

Then, install the controller with Helm using the following command:

helm upgrade --install nginx-internal ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace nginx-ingress-internal \
  --create-namespace \
  -f nginx-internal.yaml

Verify that the controller pod is running:

kubectl get pods -n nginx-ingress-internal -l app.kubernetes.io/component=controller

Verify that the service exposes an external IP:

kubectl get svc -n nginx-ingress-internal -l app.kubernetes.io/component=controller

DNS

Optionally, you can add a DNS entry pointing to the IP of the public load balancer, so future IP changes won’t require updating the configuration on the Datadog side.

Installation steps

1. Install the CloudPrem Helm chart
2. Verify installation

Install the CloudPrem Helm chart

1. Add and update the Datadog Helm repository:

   helm repo add datadog https://helm.datadoghq.com
   helm repo update
   

2. Create a Kubernetes namespace for the chart:

   kubectl create namespace <NAMESPACE_NAME>
   

   For example, to create a cloudprem namespace:

   kubectl create namespace cloudprem
   

   Note: You can set a default namespace for your current context to avoid having to type -n <NAMESPACE_NAME> with every command:

   kubectl config set-context --current --namespace=cloudprem
   

3. Store your Datadog API key as a Kubernetes secret:

   kubectl create secret generic datadog-secret \
   -n <NAMESPACE_NAME> \
   --from-literal api-key="<DD_API_KEY>"
   

4. Store the PostgreSQL database connection string as a Kubernetes secret: To retrieve your PostgreSQL connection details, go the Azure Portal, navigate to All resources, then click on your Azure Database for PostgreSQL flexible server instance. Finally, in the Getting started tab, click on the View connection strings link in the Connect card.

   kubectl create secret generic cloudprem-metastore-uri \
     -n <NAMESPACE_NAME> \
     --from-literal QW_METASTORE_URI=postgres://<USERNAME>:<PASSWORD>@<HOST>:<PORT>/<DATABASE>
   

   For example, to store a metastore-uri secret in the cloudprem namespace:

   USERNAME=cloudprem-prod
   PASSWORD=1234567890
   HOST=cloudprem-prod.postgres.database.azure.com
   PORT=5432
   DATABASE=cloudprem_prod
   kubectl create secret generic metastore-uri \
     -n cloudprem \
     --from-literal QW_METASTORE_URI="postgres://$USERNAME:$PASSWORD@$HOST:$PORT/$DATABASE"
   

5. Store the client secret or storage account access key as a Kubernetes secret:

   kubectl create secret generic <SECRET_NAME> \
     -n <NAMESPACE_NAME> \
     --from-literal <SECRET_KEY>=<SECRET_VALUE>
   

6. Customize the Helm chart:

   Create a datadog-values.yaml file to override the default values with your custom configuration. This is where you define environment-specific settings such as the image tag, Azure tenant ID, service account, ingress setup, resource requests and limits, and more.

   Any parameters not explicitly overridden in datadog-values.yaml fall back to the defaults defined in the chart’s values.yaml.

    # Show default values
    helm show values datadog/cloudprem
   

   Here is an example of a datadog-values.yaml file with overrides for Azure:

   datadog-values.yaml

   Copy

   # Datadog configuration
   datadog:
     # The Datadog site (https://docs.datadoghq.com/getting_started/site/) to connect to. Defaults to `datadoghq.com`.
     # site: datadoghq.com
     # The name of the existing Secret containing the Datadog API key. The secret key name must be `api-key`.
     apiKeyExistingSecret: datadog-secret
   
   azure:
     tenantId: <TENANT_ID> # required
     clientId: <CLIENT_ID> # required when using AD App to authenticate with Blob Storage
     clientSecretRef:
       name: <SECRET_NAME>
       key: <SECRET_KEY>
     storageAccount:
       name: <STORAGE_ACCOUNT_NAME> # required
       # If you are using a storage account access key to authenticate with Blob Storage,
       # comment out the `clientSecretRef` section above,
       # and uncomment the `storageAccount` section below:
       # accessKeySecretRef:
         # name: <SECRET_NAME>
         # key: <SECRET_KEY>
   
      # Service account configuration
      # If `serviceAccount.create` is set to `true`, a service account is created with the specified name.
      # Additional annotations can be added using serviceAccount.extraAnnotations.
      serviceAccount:
        create: true
        name: cloudprem
   
   # CloudPrem node configuration
   config:
     # The root URI where index data is stored. This should be an Azure path.
     # All indexes created in CloudPrem are stored under this location.
     default_index_root_uri: azure://<CONTAINER_NAME>/indexes
   
   # Internal ingress configuration
   # The internal ingress NLB is created in private subnets.
   #
   # Additional annotations can be added to customize the ALB behavior.
   ingress:
     # The internal ingress is used by Datadog Agents and other collectors running outside
     # the Kubernetes cluster to send their logs to CloudPrem.
     internal:
       enabled: true
       ingressClassName: nginx-internal
       host: cloudprem.acme.internal
       extraAnnotations: {}
   
   # Metastore configuration
   # The metastore is responsible for storing and managing index metadata.
   # It requires a PostgreSQL database connection string to be provided by a Kubernetes secret.
   # The secret should contain a key named `QW_METASTORE_URI` with a value in the format:
   # postgresql://<username>:<password>@<host>:<port>/<database>
   #
   # The metastore connection string is mounted into the pods using extraEnvFrom to reference the secret.
   metastore:
     extraEnvFrom:
       - secretRef:
           name: cloudprem-metastore-uri
   
   # Indexer configuration
   # The indexer is responsible for processing and indexing incoming data it receives data from various sources (for example, Datadog Agents, log collectors)
   # and transforms it into searchable files called "splits" stored in S3.
   #
   # The indexer is horizontally scalable - you can increase `replicaCount` to handle higher indexing throughput.
   # Resource requests and limits should be tuned based on your indexing workload.
   #
   # The default values are suitable for moderate indexing loads of up to 20 MB/s per indexer pod.
   indexer:
     replicaCount: 2
   
     resources:
       requests:
         cpu: "4"
         memory: "8Gi"
       limits:
         cpu: "4"
         memory: "8Gi"
   
      # Searcher configuration
      # The searcher is responsible for executing search queries against the indexed data stored in S3.
      # It handles search requests from Datadog's query service and returns matching results.
      #
      # The searcher is horizontally scalable - you can increase `replicaCount` to handle more concurrent searches.
      # Resource requirements for searchers are highly workload-dependent and should be determined empirically.
      # Key factors that impact searcher performance include:
      # - Query complexity (for example, number of terms, use of wildcards or regex)
      # - Query concurrency (number of simultaneous searches)
      # - Amount of data scanned per query
      # - Data access patterns (cache hit rates)
      #
      # Memory is particularly important for searchers as they cache frequently accessed index data in memory.
      searcher:
        replicaCount: 2
   
        resources:
          requests:
            cpu: "4"
            memory: "16Gi"
          limits:
            cpu: "4"
            memory: "16Gi"

7. Install or upgrade the Helm chart

   helm upgrade --install <RELEASE_NAME> datadog/cloudprem \
     -n <NAMESPACE_NAME> \
     -f datadog-values.yaml
   

Verification

Check deployment status

Verify that all CloudPrem components are running:

kubectl get pods -n <NAMESPACE_NAME>
kubectl get ingress -n <NAMESPACE_NAME>
kubectl get services -n <NAMESPACE_NAME>

Uninstall

To uninstall CloudPrem, execute the following command:

helm uninstall <RELEASE_NAME>

Next step

Set up log ingestion with Datadog Agent - Configure the Datadog Agent to send logs to CloudPrem