Data Observability: Jobs Monitoring for Spark on Kubernetes

문서 > Data Observability Overview > Data Observability: Jobs Monitoring > Data Observability: Jobs Monitoring for Spark on Kubernetes

이 페이지는 아직 한국어로 제공되지 않습니다. 번역 작업 중입니다.
현재 번역 프로젝트에 대한 질문이나 피드백이 있으신 경우 언제든지 연락주시기 바랍니다.

Data Observability: Jobs Monitoring gives visibility into the performance and reliability of Apache Spark applications on Kubernetes.

Setup

Data Observability: Jobs Monitoring requires Datadog Agent version 7.64.0 or later, and Java tracer version 1.38.0 or later.

Follow these steps to enable Data Observability: Jobs Monitoring for Spark on Kubernetes.

Install the Datadog Agent on your Kubernetes cluster.
(Optional) Collect Spark Data Lineage with OpenLineage.
Enable Single Step Instrumentation.

Install the Datadog Agent on your Kubernetes cluster

If you have already installed the Datadog Agent on your Kubernetes cluster, make sure you’ve enabled the Datadog Admission Controller. You can then go to the next step, Enable Single Step Instrumentation.

You can install the Datadog Agent using the Datadog Operator or Helm.

Prerequisites

Kubernetes cluster version v1.20.X+
Helm
The kubectl CLI

Installation

Install the Datadog Operator by running the following commands:

helm repo add datadog https://helm.datadoghq.com
helm install my-datadog-operator datadog/datadog-operator

Create a Kubernetes Secret to store your Datadog API key.
```
kubectl create secret generic datadog-secret --from-literal api-key=<DATADOG_API_KEY>
```
Replace <DATADOG_API_KEY> with your Datadog API key.

Create a file, datadog-agent.yaml, that contains the following configuration:

kind: DatadogAgent
apiVersion: datadoghq.com/v2alpha1
metadata:
  name: datadog
spec:
  features:
    apm:
      enabled: true
      hostPortConfig:
        enabled: true
        hostPort: 8126
    admissionController:
      enabled: true
      mutateUnlabelled: false
    # (Optional) Uncomment the next three lines to enable logs collection
    # logCollection:
      # enabled: true
      # containerCollectAll: true
  global:
    site: <DATADOG_SITE>
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
  override:
    nodeAgent:
      image:
        tag: <DATADOG_AGENT_VERSION>

Replace <DATADOG_SITE> with your Datadog site. Your site is . (Ensure the correct SITE is selected on the right).

Replace <DATADOG_AGENT_VERSION> with version 7.64.0 or later.

Optional: Uncomment the logCollection section to start collecting application logs which will be correlated to Spark job run traces. Once enabled, logs are collected from all discovered containers by default. See the Kubernetes log collection documentation for more details on the setup process.

Deploy the Datadog Agent with the above configuration file:
```
kubectl apply -f /path/to/your/datadog-agent.yaml
```

Create a Kubernetes Secret to store your Datadog API key.
```
kubectl create secret generic datadog-secret --from-literal api-key=<DATADOG_API_KEY>
```
Replace <DATADOG_API_KEY> with your Datadog API key.
Create a file, datadog-values.yaml, that contains the following configuration:
```
datadog:
  apiKeyExistingSecret: datadog-secret
  site: <DATADOG_SITE>
  apm:
    portEnabled: true
    port: 8126
  # (Optional) Uncomment the next three lines to enable logs collection
  # logs:
    # enabled: true
    # containerCollectAll: true

agents:
  image:
    tag: <DATADOG_AGENT_VERSION>

clusterAgent:
  admissionController:
    enabled: true
    muteUnlabelled: false
```
Replace <DATADOG_SITE> with your Datadog site. Your site is . (Ensure the correct SITE is selected on the right).
Replace <DATADOG_AGENT_VERSION> with version 7.64.0 or later.
Optional: Uncomment the logs section to start collecting application logs which will be correlated to Spark job run traces. Once enabled, logs are collected from all discovered containers by default. See the Kubernetes log collection documentation for more details on the setup process.
Run the following command:
```
helm install <RELEASE_NAME> \
 -f datadog-values.yaml \
 --set targetSystem=<TARGET_SYSTEM> \
 datadog/datadog
```
- Replace <RELEASE_NAME> with your release name. For example, datadog-agent.
- Replace <TARGET_SYSTEM> with the name of your OS. For example, linux or windows.

(Optional) Collect Spark Data Lineage with OpenLineage

Data Observability: Jobs Monitoring can collect Spark lineage through OpenLineage, so you can see the upstream and downstream tables of your jobs and visualize entire Spark pipelines. Setup has two steps: install the OpenLineage Spark provider, then enable the feature in the Java tracer.

OpenLineage lineage collection requires Java tracer version 1.48.0 or later.

Install the OpenLineage Spark provider. Make the openlineage-spark JAR available on the classpath of your Spark driver and executors, following the OpenLineage Spark installation guide. Use OpenLineage version 1.29.0 or later, which is the first version that supports the run-tag facets that Data Observability: Jobs Monitoring relies on.
You do not need to set spark.extraListeners or configure any spark.openlineage.transport.* options. When the feature is enabled and the JAR is present, the Datadog Java tracer registers the OpenLineage listener and routes lineage events through the local Datadog Agent.
Enable OpenLineage collection. Set the DD_DATA_JOBS_OPENLINEAGE_ENABLED environment variable to true on the Spark driver.
If using Single Step Instrumentation, add it to the ddTraceConfigs section of the spark-driver target:
```
ddTraceConfigs:
  - name: DD_DATA_JOBS_ENABLED
    value: "true"
  - name: DD_DATA_JOBS_OPENLINEAGE_ENABLED
    value: "true"
```
Reapply your configuration and restart the targeted pods.

Enable Single Step Instrumentation

Single Step Instrumentation (SSI) injects the Java tracer into your Spark driver and executor pods at startup. It works regardless of whether your Spark driver runs in cluster mode (as a dedicated Kubernetes pod) or client mode (as a process inside your submitter pod; for example, an Airflow scheduler or worker).

Spark automatically sets spark-role: driver on driver pods and spark-role: executor on executor pods. In client mode, replace spark-role: driver with the labels that identify your submitter pod instead. To find those labels, run:

kubectl get pod <SUBMITTER_POD_NAME> -n <NAMESPACE> --show-labels

Requires Datadog Operator version 1.13.0 or later.

Add the features.apm.instrumentation section to your datadog-agent.yaml and apply it:

features:
  apm:
    instrumentation:
      enabled: true
      targets:
        - name: spark-driver
          namespaceSelector:
            matchNames:
              - <NAMESPACE>   # namespace where your Spark jobs run
          podSelector:
            matchLabels:
              spark-role: driver   # replace with your submitter pod labels if running in client mode
          ddTraceVersions:
            java: "latest"
          ddTraceConfigs:
            - name: DD_DATA_JOBS_ENABLED
              value: "true"
        - name: spark-executor
          namespaceSelector:
            matchNames:
              - <NAMESPACE>
          podSelector:
            matchLabels:
              spark-role: executor
          ddTraceVersions:
            java: "latest"
          ddTraceConfigs:
            - name: DD_DATA_JOBS_ENABLED
              value: "true"

kubectl apply -f /path/to/your/datadog-agent.yaml

Add the following to your datadog-values.yaml and apply it:

datadog:
  apm:
    instrumentation:
      enabled: true
      targets:
        - name: spark-driver
          namespaceSelector:
            matchNames:
              - <NAMESPACE>   # namespace where your Spark jobs run
          podSelector:
            matchLabels:
              spark-role: driver   # replace with your submitter pod labels if running in client mode
          ddTraceVersions:
            java: "latest"
          ddTraceConfigs:
            - name: DD_DATA_JOBS_ENABLED
              value: "true"
        - name: spark-executor
          namespaceSelector:
            matchNames:
              - <NAMESPACE>
          podSelector:
            matchLabels:
              spark-role: executor
          ddTraceVersions:
            java: "latest"
          ddTraceConfigs:
            - name: DD_DATA_JOBS_ENABLED
              value: "true"

helm upgrade <RELEASE_NAME> datadog/datadog -f datadog-values.yaml

After applying the configuration, restart the targeted pods. SSI injects the init container into each pod on startup.

Validation

In Datadog, view the Data Observability: Jobs Monitoring page to see a list of all your data processing jobs.

Advanced Configuration

Set service, environment, and version tags

To attach service, environment, and version tags to your job traces, pass the following JVM options in your spark-submit configuration or spark-defaults.conf:

spark.driver.extraJavaOptions=-Ddd.service=<JOB_NAME> -Ddd.env=<ENV> -Ddd.version=<VERSION>
spark.executor.extraJavaOptions=-Ddd.service=<JOB_NAME> -Ddd.env=<ENV> -Ddd.version=<VERSION>

Tag spans at runtime

You can set tags on Spark spans at runtime. These tags are applied only to spans that start after the tag is added.

// Add tag for all next Spark computations
sparkContext.setLocalProperty("spark.datadog.tags.key", "value")
spark.read.parquet(...)

To remove a runtime tag:

// Remove tag for all next Spark computations
sparkContext.setLocalProperty("spark.datadog.tags.key", null)