Overview

CloudPrem can offload leaf search operations to AWS Lambda for horizontal scaling. When the local search queue becomes saturated, overflow splits are automatically sent to Lambda functions for processing. This allows CloudPrem to handle traffic spikes without provisioning additional searcher nodes.

Startup validation

When a Lambda configuration is defined, CloudPrem performs a dry run invocation at startup to verify that:

• The Lambda function exists
• The function version matches the running CloudPrem binary
• The invoker has permission to call the function

Prerequisite: IAM permissions

Lambda search offloading requires specific permissions for two separate IAM roles:

• The CloudPrem node role: the role attached to the Kubernetes nodes (or pod identity) running CloudPrem. The role is defined in the serviceAccount section of your values.yaml. This role needs permissions to invoke and deploy the Lambda function.
• The Lambda execution role: the role assumed by the Lambda function itself at runtime. This role needs read access to your index data in S3. Its ARN must be set in the config.searcher.lambda.auto_deploy.execution_role_arn key. For more details, see the Configuration section.

CloudPrem node permissions

The IAM role running CloudPrem needs the following permissions to invoke and deploy the Lambda function:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "lambda:CreateFunction",
        "lambda:GetFunction",
        "lambda:UpdateFunctionCode",
        "lambda:PublishVersion",
        "lambda:ListVersionsByFunction",
        "lambda:DeleteFunction"
      ],
      "Resource": "arn:aws:lambda:<REGION>:<ACCOUNT_ID>:function:cloudprem-*"
    },
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "<LAMBDA_EXECUTION_ROLE_ARN>",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "lambda.amazonaws.com"
        }
      }
    }
  ]
}

Replace <REGION>, <ACCOUNT_ID>, and <LAMBDA_EXECUTION_ROLE_ARN> with values matching your environment.

Lambda execution role

The Lambda function requires its own execution role with read access to your S3 bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::<YOUR_INDEX_BUCKET>/*"
    }
  ]
}

The execution role must also have a trust policy allowing Lambda to assume it:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Optionally, to capture Lambda logs in CloudWatch, add the following permissions to the execution role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:<REGION>:<ACCOUNT_ID>:*"
    }
  ]
}

Configuration

After setting up the IAM permissions, add a lambda section under config.searcher in your Helm chart values file to enable Lambda offloading:

config:
  searcher:
    lambda:
      function_name: cloudprem-search
      offload_threshold: 100
      auto_deploy:
        execution_role_arn: arn:aws:iam::123456789012:role/cloudprem-lambda-role
        invocation_timeout_secs: 15

Replace arn:aws:iam::123456789012:role/cloudprem-lambda-role with the Lambda execution role you have created.

You can then upgrade the Helm chart release:

helm upgrade <RELEASE_NAME> datadog/cloudprem \
  -n <NAMESPACE_NAME> \
  -f datadog-values.yaml

Lambda configuration options

Auto-deploy configuration options

Further reading

Additional helpful documentation, links, and articles:

Configure CloudPremDOCUMENTATION Size your clusterDOCUMENTATION