Overview

This check monitors Kubeflow through the Datadog Agent.

Setup

Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the Autodiscovery Integration Templates for guidance on applying these instructions.

Installation

The Kubeflow check is included in the Datadog Agent package. No additional installation is needed on your server.

Configuration

1. Edit the kubeflow.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory to start collecting your kubeflow performance data. See the sample kubeflow.d/conf.yaml for all available configuration options.

2. Restart the Agent.

Metric collection

Make sure that the Prometheus-formatted metrics are exposed for your kubeflow componenet. For the Agent to start collecting metrics, the kubeflow pods need to be annotated.

Kubeflow has metrics endpoints that can be accessed on port 9090.

To enable metrics exposure in kubeflow through prometheus, you might need to enable the prometheus service monitoring for the component in question.

You can use Kube-Prometheus-Stack or a custom Prometheus installation.

How to install Kube-Prometheus-Stack:

1. Add Helm Repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

2. Install the Chart:

helm install prometheus-stack prometheus-community/kube-prometheus-stack

3. Expose Prometheus service externally:

kubectl port-forward prometheus-stack 9090:9090

Set Up ServiceMonitors for Kubeflow Components:

You need to configure ServiceMonitors for Kubeflow components to expose their Prometheus metrics. If your Kubeflow component exposes Prometheus metrics by default. You’ll just need to configure Prometheus to scrape these metrics.

The ServiceMonitor would look like this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: <kubeflow-component>-monitor
  labels:
    release: prometheus-stack
spec:
  selector:
    matchLabels:
      app: <kubeflow-component-name>
  endpoints:
  - port: http
    path: /metrics

Where <kubeflow-component> is to be replaced by pipelines, kserve or katib and <kubeflow-component-name> is to be replaced by ml-pipeline, kserve or katib.

Note: The listed metrics can only be collected if they are available(depending on the version). Some metrics are generated only when certain actions are performed.

The only parameter required for configuring the kubeflow check is openmetrics_endpoint. This parameter should be set to the location where the Prometheus-formatted metrics are exposed. The default port is 9090. In containerized environments, %%host%% should be used for host autodetection.

apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<POD_NAME>'
  annotations:
    ad.datadoghq.com/controller.checks: |
      {
        "kubeflow": {
          "init_config": {},
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:9090/metrics"
            }
          ]
        }
      }      
    # (...)
spec:
  containers:
    - name: 'controller'
# (...)

Validation

Run the Agent’s status subcommand and look for kubeflow under the Checks section.

Data Collected

Metrics

Events

The Kubeflow integration does not include any events.

Service Checks

kubeflow.openmetrics.health
Returns CRITICAL if the Agent is unable to connect to the Kubeflow OpenMetrics endpoint, otherwise returns OK.
Statuses: ok, critical

Troubleshooting

Need help? Contact Datadog support.