Data Jobs Monitoring for Spark on Kubernetes

Data Jobs Monitoring gives visibility into the performance and reliability of Apache Spark applications on Kubernetes.

Setup

Data Jobs Monitoring requires Datadog Agent version 7.55.0 or later, and Java tracer version 1.38.0 or later.

Follow these steps to enable Data Jobs Monitoring for Spark on Kubernetes.

  1. Install the Datadog Agent on your Kubernetes cluster.
  2. Inject Spark instrumentation.

Install the Datadog Agent on your Kubernetes cluster

If you have already installed the Datadog Agent on your Kubernetes cluster, ensure that you have enabled the Datadog Admission Controller. You can then go to the next step, Inject Spark instrumentation.

You can install the Datadog Agent using the Datadog Operator or Helm.

Prerequisites

Installation

  1. Install the Datadog Operator by running the following commands:

    helm repo add datadog https://helm.datadoghq.com
helm install my-datadog-operator datadog/datadog-operator

  2. Create a Kubernetes Secret to store your Datadog API key.

    kubectl create secret generic datadog-secret --from-literal api-key=<DATADOG_API_KEY> --from-literal app-key=<DATADOG_APP_KEY>

  3. Create a file, datadog-agent.yaml, that contains the following configuration:

    kind: DatadogAgent
apiVersion: datadoghq.com/v2alpha1
metadata:
  name: datadog
spec:
  features:
    apm:
      enabled: true
      hostPortConfig:
        enabled: true
        hostPort: 8126
    admissionController:
      enabled: true
      mutateUnlabelled: false
    # (Optional) Uncomment the next three lines to enable logs collection
    # logCollection:
      # enabled: true
      # containerCollectAll: true
  global:
    site: <DATADOG_SITE>
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
      appSecret:
        secretName: datadog-secret
        keyName: app-key
  override:
    nodeAgent:
      image:
        tag: <DATADOG_AGENT_VERSION>

    Replace <DATADOG_SITE> with your Datadog site. Your site is datadoghq.com. (Ensure the correct SITE is selected on the right).

    Replace <DATADOG_AGENT_VERSION> with version 7.55.0 or later.

    Optional: Uncomment the logCollection section to start collecting application logs which will be correlated to Spark job run traces. Once enabled, logs are collected from all discovered containers by default. See the Kubernetes log collection documentation for more details on the setup process.

  4. Deploy the Datadog Agent with the above configuration file:

    kubectl apply -f /path/to/your/datadog-agent.yaml

  1. Create a Kubernetes Secret to store your Datadog API key.

    kubectl create secret generic datadog-secret --from-literal api-key=<DATADOG_API_KEY> --from-literal app-key=<DATADOG_APP_KEY>

  2. Create a file, datadog-values.yaml, that contains the following configuration:

    datadog:
  apiKeyExistingSecret: datadog-secret
  appKeyExistingSecret: datadog-secret
  site: <DATADOG_SITE>
  apm:
    portEnabled: true
    port: 8126
  # (Optional) Uncomment the next three lines to enable logs collection
  # logs:
    # enabled: true
    # containerCollectAll: true

agents:
  image:
    tag: <DATADOG_AGENT_VERSION>

clusterAgent:
  admissionController:
    enabled: true
    muteUnlabelled: false

    Replace <DATADOG_SITE> with your Datadog site. Your site is datadoghq.com. (Ensure the correct SITE is selected on the right).

    Replace <DATADOG_AGENT_VERSION> with version 7.55.0 or later.

    Optional: Uncomment the logs section to start collecting application logs which will be correlated to Spark job run traces. Once enabled, logs are collected from all discovered containers by default. See the Kubernetes log collection documentation for more details on the setup process.

  3. Run the following command:

    helm install <RELEASE_NAME> \
 -f datadog-values.yaml \
 --set targetSystem=<TARGET_SYSTEM> \
 datadog/datadog

    • Replace <RELEASE_NAME> with your release name. For example, datadog-agent.

    • Replace <TARGET_SYSTEM> with the name of your OS. For example, linux or windows.

Inject Spark instrumentation

When you run your Spark job, use the following configurations:

spark.kubernetes.{driver,executor}.label.admission.datadoghq.com/enabled (Required)
true
spark.kubernetes.{driver,executor}.annotation.admission.datadoghq.com/java-lib.version (Required)
latest
spark.{driver,executor}.extraJavaOptions
-Ddd.data.jobs.enabled=true (Required)
true
-Ddd.service (Optional)
Your service name. Because this option sets the job name in Datadog, it is recommended that you use a human-readable name.
-Ddd.env (Optional)
Your environment, such as prod or dev.
-Ddd.version (Optional)
Your version.
-Ddd.tags (Optional)
Other tags you wish to add, in the format <KEY_1>:<VALUE_1>,<KEY_2:VALUE_2>.

Example: spark-submit

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master k8s://<CLUSTER_ENDPOINT> \
  --conf spark.kubernetes.container.image=895885662937.dkr.ecr.us-west-2.amazonaws.com/spark/emr-6.10.0:latest \
  --deploy-mode cluster \
  --conf spark.kubernetes.namespace=<NAMESPACE> \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=<SERVICE_ACCOUNT> \
  --conf spark.kubernetes.authenticate.executor.serviceAccountName=<SERVICE_ACCOUNT> \
  --conf spark.kubernetes.driver.label.admission.datadoghq.com/enabled=true \
  --conf spark.kubernetes.executor.label.admission.datadoghq.com/enabled=true \
  --conf spark.kubernetes.driver.annotation.admission.datadoghq.com/java-lib.version=latest \
  --conf spark.kubernetes.executor.annotation.admission.datadoghq.com/java-lib.version=latest \
  --conf spark.driver.extraJavaOptions="-Ddd.data.jobs.enabled=true -Ddd.service=<JOB_NAME> -Ddd.env=<ENV> -Ddd.version=<VERSION> -Ddd.tags=<KEY_1>:<VALUE_1>,<KEY_2:VALUE_2>" \
  --conf spark.executor.extraJavaOptions="-Ddd.data.jobs.enabled=true -Ddd.service=<JOB_NAME> -Ddd.env=<ENV> -Ddd.version=<VERSION> -Ddd.tags=<KEY_1>:<VALUE_1>,<KEY_2:VALUE_2>" \
  local:///usr/lib/spark/examples/jars/spark-examples.jar 20

Example: AWS start-job-run

aws emr-containers start-job-run \
--virtual-cluster-id <EMR_CLUSTER_ID> \
--name myjob \
--execution-role-arn <EXECUTION_ROLE_ARN> \
--release-label emr-6.10.0-latest \
--job-driver '{
  "sparkSubmitJobDriver": {
    "entryPoint": "s3://BUCKET/spark-examples.jar",
    "sparkSubmitParameters": "--class <MAIN_CLASS> --conf spark.kubernetes.driver.label.admission.datadoghq.com/enabled=true --conf spark.kubernetes.executor.label.admission.datadoghq.com/enabled=true --conf spark.kubernetes.driver.annotation.admission.datadoghq.com/java-lib.version=latest --conf spark.kubernetes.executor.annotation.admission.datadoghq.com/java-lib.version=latest --conf spark.driver.extraJavaOptions=\"-Ddd.data.jobs.enabled=true -Ddd.service=<JOB_NAME> -Ddd.env=<ENV> -Ddd.version=<VERSION> -Ddd.tags=<KEY_1>:<VALUE_1>,<KEY_2:VALUE_2>\"  --conf spark.executor.extraJavaOptions=\"-Ddd.data.jobs.enabled=true -Ddd.service=<JOB_NAME> -Ddd.env=<ENV> -Ddd.version=<VERSION> -Ddd.tags=<KEY_1>:<VALUE_1>,<KEY_2:VALUE_2>\""
  }
}

Validation

In Datadog, view the Data Jobs Monitoring page to see a list of all your data processing jobs.

Advanced Configuration

Tag spans at runtime

You can set tags on Spark spans at runtime. These tags are applied only to spans that start after the tag is added.

// Add tag for all next Spark computations
sparkContext.setLocalProperty("spark.datadog.tags.key", "value")
spark.read.parquet(...)

To remove a runtime tag:

// Remove tag for all next Spark computations
sparkContext.setLocalProperty("spark.datadog.tags.key", null)

Further Reading

Additional helpful documentation, links, and articles:

Data Jobs MonitoringDOCUMENTATION more more