Language

English Français 日本語 한국어 Español

Datadog Site

Site help
US1 US3 US5 EU AP1 AP2 UK1 US1-FED US2-FED

Get SPA recommendations v2

copy Copy pageCopied

Note: This endpoint is in preview and may change in the future. It is not yet recommended for production use.

POST https://api.ap1.datadoghq.com/api/v2/spa/recommendations-v2/{service}https://api.ap2.datadoghq.com/api/v2/spa/recommendations-v2/{service}https://api.datadoghq.eu/api/v2/spa/recommendations-v2/{service}https://api.ddog-gov.com/api/v2/spa/recommendations-v2/{service}https://api.us2.ddog-gov.com/api/v2/spa/recommendations-v2/{service}https://api.uk1.datadoghq.com/api/v2/spa/recommendations-v2/{service}https://api.datadoghq.com/api/v2/spa/recommendations-v2/{service}https://api.us3.datadoghq.com/api/v2/spa/recommendations-v2/{service}https://api.us5.datadoghq.com/api/v2/spa/recommendations-v2/{service}

Overview

This endpoint is experimental and restricted to Datadog internal use only. Retrieve resource recommendations for a Spark job. The caller (Spark Gateway) provides a service name and the job’s raw arguments. SPA determines which arguments are relevant for the service and returns structured recommendations for driver and executor resources.

Arguments

Path Parameters

Name

Type

Description

service [required]

string

The service name for a Spark job

Request

Body Data (required)

Expand All

Field

Type

Description

data [required]

object

JSON:API resource object for the SPA v2 recommendation request.

attributes [required]

object

Attributes for requesting SPA recommendations by forwarding a Spark job's raw arguments instead of a precomputed shard.

arguments [required]

[string]

Raw, unfiltered Spark job arguments as submitted (for example, --org_id=2). SPA determines which arguments are relevant for the given service.

type [required]

enum

JSON:API resource type for the SPA v2 recommendation request. Allowed enum values: recommendation_v2_request

default: recommendation_v2_request

{
  "data": {
    "attributes": {
      "arguments": [
        ""
      ]
    },
    "type": "recommendation_v2_request"
  }
}

Response

OK

Expand All

Field

Type

Description

data [required]

object

JSON:API resource object for SPA Recommendation. Includes type, optional ID, and resource attributes with structured recommendations.

attributes [required]

object

Attributes of the SPA Recommendation resource. Contains recommendations for both driver and executor components.

confidence_level

double

The confidence level of the recommendation, expressed as a value between 0.0 (low confidence) and 1.0 (high confidence).

driver [required]

object

Resource recommendation for a single Spark component (driver or executor). Contains estimation data used to patch Spark job specs.

estimation [required]

object

Recommended resource values for a Spark driver or executor, derived from recent real usage metrics. Used by SPA to propose more efficient pod sizing.

cpu

object

CPU usage statistics derived from historical Spark job metrics. Provides multiple estimates so users can choose between conservative and cost-saving risk profiles.

max

int64

Maximum CPU usage observed for the job, expressed in millicores. This represents the upper bound of usage.

p75

int64

75th percentile of CPU usage (millicores). Represents a cost-saving configuration while covering most workloads.

p95

int64

95th percentile of CPU usage (millicores). Balances performance and cost, providing a safer margin than p75.

ephemeral_storage

int64

Recommended ephemeral storage allocation (in MiB). Derived from job temporary storage patterns.

heap

int64

Recommended JVM heap size (in MiB).

memory

int64

Recommended total memory allocation (in MiB). Includes both heap and overhead.

overhead

int64

Recommended JVM overhead (in MiB). Computed as total memory - heap.

executor [required]

object

Resource recommendation for a single Spark component (driver or executor). Contains estimation data used to patch Spark job specs.

estimation [required]

object

Recommended resource values for a Spark driver or executor, derived from recent real usage metrics. Used by SPA to propose more efficient pod sizing.

cpu

object

CPU usage statistics derived from historical Spark job metrics. Provides multiple estimates so users can choose between conservative and cost-saving risk profiles.

max

int64

Maximum CPU usage observed for the job, expressed in millicores. This represents the upper bound of usage.

p75

int64

75th percentile of CPU usage (millicores). Represents a cost-saving configuration while covering most workloads.

p95

int64

95th percentile of CPU usage (millicores). Balances performance and cost, providing a safer margin than p75.

ephemeral_storage

int64

Recommended ephemeral storage allocation (in MiB). Derived from job temporary storage patterns.

heap

int64

Recommended JVM heap size (in MiB).

memory

int64

Recommended total memory allocation (in MiB). Includes both heap and overhead.

overhead

int64

Recommended JVM overhead (in MiB). Computed as total memory - heap.

id

string

Resource identifier for the recommendation. Optional in responses.

type [required]

enum

JSON:API resource type for Spark Pod Autosizing recommendations. Identifies the Recommendation resource returned by SPA. Allowed enum values: recommendation

default: recommendation

{
  "data": {
    "attributes": {
      "confidence_level": "number",
      "driver": {
        "estimation": {
          "cpu": {
            "max": "integer",
            "p75": "integer",
            "p95": "integer"
          },
          "ephemeral_storage": "integer",
          "heap": "integer",
          "memory": "integer",
          "overhead": "integer"
        }
      },
      "executor": {
        "estimation": {
          "cpu": {
            "max": "integer",
            "p75": "integer",
            "p95": "integer"
          },
          "ephemeral_storage": "integer",
          "heap": "integer",
          "memory": "integer",
          "overhead": "integer"
        }
      }
    },
    "id": "string",
    "type": "recommendation"
  }
}

JSON:API document containing a single Recommendation resource. Returned by SPA when the Spark Gateway requests recommendations.

Expand All

Field

Type

Description

data [required]

object

JSON:API resource object for SPA Recommendation. Includes type, optional ID, and resource attributes with structured recommendations.

attributes [required]

object

Attributes of the SPA Recommendation resource. Contains recommendations for both driver and executor components.

confidence_level

double

The confidence level of the recommendation, expressed as a value between 0.0 (low confidence) and 1.0 (high confidence).

driver [required]

object

Resource recommendation for a single Spark component (driver or executor). Contains estimation data used to patch Spark job specs.

estimation [required]

object

Recommended resource values for a Spark driver or executor, derived from recent real usage metrics. Used by SPA to propose more efficient pod sizing.

cpu

object

CPU usage statistics derived from historical Spark job metrics. Provides multiple estimates so users can choose between conservative and cost-saving risk profiles.

max

int64

Maximum CPU usage observed for the job, expressed in millicores. This represents the upper bound of usage.

p75

int64

75th percentile of CPU usage (millicores). Represents a cost-saving configuration while covering most workloads.

p95

int64

95th percentile of CPU usage (millicores). Balances performance and cost, providing a safer margin than p75.

ephemeral_storage

int64

Recommended ephemeral storage allocation (in MiB). Derived from job temporary storage patterns.

heap

int64

Recommended JVM heap size (in MiB).

memory

int64

Recommended total memory allocation (in MiB). Includes both heap and overhead.

overhead

int64

Recommended JVM overhead (in MiB). Computed as total memory - heap.

executor [required]

object

Resource recommendation for a single Spark component (driver or executor). Contains estimation data used to patch Spark job specs.

estimation [required]

object

Recommended resource values for a Spark driver or executor, derived from recent real usage metrics. Used by SPA to propose more efficient pod sizing.

cpu

object

CPU usage statistics derived from historical Spark job metrics. Provides multiple estimates so users can choose between conservative and cost-saving risk profiles.

max

int64

Maximum CPU usage observed for the job, expressed in millicores. This represents the upper bound of usage.

p75

int64

75th percentile of CPU usage (millicores). Represents a cost-saving configuration while covering most workloads.

p95

int64

95th percentile of CPU usage (millicores). Balances performance and cost, providing a safer margin than p75.

ephemeral_storage

int64

Recommended ephemeral storage allocation (in MiB). Derived from job temporary storage patterns.

heap

int64

Recommended JVM heap size (in MiB).

memory

int64

Recommended total memory allocation (in MiB). Includes both heap and overhead.

overhead

int64

Recommended JVM overhead (in MiB). Computed as total memory - heap.

id

string

Resource identifier for the recommendation. Optional in responses.

type [required]

enum

JSON:API resource type for Spark Pod Autosizing recommendations. Identifies the Recommendation resource returned by SPA. Allowed enum values: recommendation

default: recommendation

{
  "data": {
    "attributes": {
      "confidence_level": "number",
      "driver": {
        "estimation": {
          "cpu": {
            "max": "integer",
            "p75": "integer",
            "p95": "integer"
          },
          "ephemeral_storage": "integer",
          "heap": "integer",
          "memory": "integer",
          "overhead": "integer"
        }
      },
      "executor": {
        "estimation": {
          "cpu": {
            "max": "integer",
            "p75": "integer",
            "p95": "integer"
          },
          "ephemeral_storage": "integer",
          "heap": "integer",
          "memory": "integer",
          "overhead": "integer"
        }
      }
    },
    "id": "string",
    "type": "recommendation"
  }
}

Bad Request

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Not Authorized

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Not Found

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Too many requests

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example

                  # Path parameters
export service="CHANGE_ME"
# Use a Personal Access Token or Service Access Token
export DD_BEARER_TOKEN="<PERSONAL_ACCESS_TOKEN OR SERVICE_ACCESS_TOKEN>"

# Curl command
curl -X POST "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.us2.ddog-gov.com"https://api.uk1.datadoghq.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/spa/recommendations-v2/${service}" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DD_BEARER_TOKEN}" \
-d @- << EOF
{
  "data": {
    "attributes": {
      "arguments": [
        ""
      ]
    },
    "type": "recommendation_v2_request"
  }
}
EOF