Spa

SPA (Spark Pod Autosizing) API. Provides resource recommendations and cost insights to help optimize Spark job configurations.

Get SPA Recommendations

v2 (latest)

Note: This endpoint is in public beta and may change in the future. It is not yet recommended for production use.

GET https://api.ap1.datadoghq.com/api/v2/spa/recommendations/{service}/{shard}https://api.ap2.datadoghq.com/api/v2/spa/recommendations/{service}/{shard}https://api.datadoghq.eu/api/v2/spa/recommendations/{service}/{shard}https://api.ddog-gov.com/api/v2/spa/recommendations/{service}/{shard}https://api.datadoghq.com/api/v2/spa/recommendations/{service}/{shard}https://api.us3.datadoghq.com/api/v2/spa/recommendations/{service}/{shard}https://api.us5.datadoghq.com/api/v2/spa/recommendations/{service}/{shard}

Overview

Retrieve resource recommendations for a Spark job. The caller (Spark Gateway or DJM UI) provides a service name and shard identifier, and SPA returns structured recommendations for driver and executor resources.

Arguments

Path Parameters

Name

Type

Description

shard [required]

string

The shard tag for a spark job, which differentiates jobs within the same service that have different resource needs

service [required]

string

The service name for a spark job

Response

Expand All

Field

Type

Description

data [required]

object

JSON:API resource object for SPA Recommendation. Includes type, optional ID, and resource attributes with structured recommendations.

attributes [required]

object

Attributes of the SPA Recommendation resource. Contains recommendations for both driver and executor components.

{
  "data": {
    "attributes": {
      "driver": {
        "estimation": {
          "cpu": {
            "max": "integer",
            "p75": "integer",
            "p95": "integer"
          },
          "ephemeral_storage": "integer",
          "heap": "integer",
          "memory": "integer",
          "overhead": "integer"
        }
      },
      "executor": {
        "estimation": {
          "cpu": {
            "max": "integer",
            "p75": "integer",
            "p95": "integer"
          },
          "ephemeral_storage": "integer",
          "heap": "integer",
          "memory": "integer",
          "overhead": "integer"
        }
      }
    },
    "id": "string",
    "type": "recommendation"
  }
}

JSON:API document containing a single Recommendation resource. Returned by SPA when the Spark Gateway requests recommendations.

Expand All

Field

Type

Description

data [required]

object

JSON:API resource object for SPA Recommendation. Includes type, optional ID, and resource attributes with structured recommendations.

attributes [required]

object

Attributes of the SPA Recommendation resource. Contains recommendations for both driver and executor components.

{
  "data": {
    "attributes": {
      "driver": {
        "estimation": {
          "cpu": {
            "max": "integer",
            "p75": "integer",
            "p95": "integer"
          },
          "ephemeral_storage": "integer",
          "heap": "integer",
          "memory": "integer",
          "overhead": "integer"
        }
      },
      "executor": {
        "estimation": {
          "cpu": {
            "max": "integer",
            "p75": "integer",
            "p95": "integer"
          },
          "ephemeral_storage": "integer",
          "heap": "integer",
          "memory": "integer",
          "overhead": "integer"
        }
      }
    },
    "id": "string",
    "type": "recommendation"
  }
}

Bad Request

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Not Authorized

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Too many requests

API error response.

Expand All

Field

Type

Description

errors [required]

[string]

A list of errors.

{
  "errors": [
    "Bad Request"
  ]
}

Code Example

                  # Path parameters
export shard="CHANGE_ME"
export service="CHANGE_ME"
# Curl command
curl -X GET "https://api.ap1.datadoghq.com"https://api.ap2.datadoghq.com"https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com"https://api.us5.datadoghq.com/api/v2/spa/recommendations/${service}/${shard}" \
-H "Accept: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}"

                