Overview

Datadog monitors every aspect of your Istio environment, so you can:

• Assess the health of Envoy and the Istio control plane with logs.
• Break down the performance of your service mesh with request, bandwidth, and resource consumption metrics.
• Map network communication between containers, pods, and services over the mesh with Cloud Network Monitoring.
• Drill into distributed traces for applications transacting over the mesh with APM.

To learn more about monitoring your Istio environment with Datadog, see the Monitor blog post.

Setup

For general instructions on configuring integrations in containerized environments, see Configure integrations with Autodiscovery on Kubernetes or Configure integrations with Autodiscovery on Docker.

This OpenMetrics-based integration has a latest mode (use_openmetrics: true) and a legacy mode (use_openmetrics: false). To get all the most up-to-date features, Datadog recommends enabling latest mode. For more information, see Latest and Legacy Versioning For OpenMetrics-based Integrations.

If you have multiple instances of Datadog collecting Istio metrics, make sure you are using the same mode for all of them. Otherwise, metrics data may fluctuate on the Datadog site.

Metrics marked as [OpenMetrics V1], [OpenMetrics V2], or [OpenMetrics V1 and V2] are only available using the corresponding mode of the Istio integration. Metrics marked as Istio v1.5+ are collected using Istio version 1.5 or later.

Installation

Istio is included in the Datadog Agent. Install the Datadog Agent on your Istio servers or in your cluster and point it at Istio.

Envoy

If you want to monitor the Envoy proxies in Istio, configure the Envoy integration.

Configuration

Metric collection

To monitor Istio v1.5+ there are two key components matching the Istio architecture for the Prometheus-formatted metrics:

• Data plane: The istio-proxy sidecar containers
• Control plane: The istiod service managing the proxies

These are both run as istio Agent checks, but they have different responsibilities and are configured separately.

Data plane configuration

The default istio.d/auto_conf.yaml file automatically sets up monitoring for each of the istio-proxy sidecar containers. The Agent initializes this check for each sidecar container that it detects automatically. This configuration enables the reporting of istio.mesh.* metrics for the data exposed by each of these sidecar containers.

To customize the data plane portion of the integration, create a custom Istio configuration file istio.yaml. See Configure integrations on Kubernetes or Configure integrations with Autodiscovery on Docker for options in creating this file.

This file must contain:

ad_identifiers:
  - proxyv2
  - proxyv2-rhel8

init_config:

instances:
  - use_openmetrics: true
    send_histograms_buckets: false
    istio_mesh_endpoint: http://%%host%%:15020/stats/prometheus
    tag_by_endpoint: false

Customize this file with any additional configurations. See the sample istio.d/conf.yaml for all available configuration options.

Control plane configuration

To monitor the Istio control plane and report the mixer, galley, pilot, and citadel metrics, you must configure the Agent to monitor the istiod deployment. In Istio v1.5 or later, apply the following pod annotations for the deployment istiod in the istio-system namespace:

ad.datadoghq.com/discovery.checks: |
  {
    "istio": {
      "instances": [
        {
          "istiod_endpoint": "http://%%host%%:15014/metrics",
          "use_openmetrics": "true"
        }
      ]
    }
  }  

This annotation specifies the container discovery to match the default container name of the Istio container in this pod. Replace this annotation ad.datadoghq.com/<CONTAINER_NAME>.checks with the name (.spec.containers[i].name) of your Istio container if yours differs.

The method for applying these annotations varies depending on the Istio deployment strategy (Istioctl, Helm, Operator) used. Consult the Istio documentation for the proper method to apply these pod annotations. See the sample istio.d/conf.yaml for all available configuration options.

Disable sidecar injection for Datadog Agent pods

If you are installing the Datadog Agent in a container, Datadog recommends that you first disable Istio’s sidecar injection.

Istio versions >= 1.10:

Add the sidecar.istio.io/inject: "false" label to the datadog-agent DaemonSet:

# (...)
spec:
  template:
    metadata:
      labels:
        sidecar.istio.io/inject: "false"
    # (...)

This can also be done with the kubectl patch command.

kubectl patch daemonset datadog-agent -p '{"spec":{"template":{"metadata":{"labels":{"sidecar.istio.io/inject":"false"}}}}}'

Istio versions <= 1.9:

Add the sidecar.istio.io/inject: "false" annotation to the datadog-agent DaemonSet:

# (...)
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    # (...)

Using the kubectl patch command:

kubectl patch daemonset datadog-agent -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}}}}}'

Log collection

Available for Agent versions >6.0

First, enable the Datadog Agent to perform log collection in Kubernetes. See Kubernetes Log Collection.

Istio logs

To collect Istio logs from your control plane (istiod), apply the following pod annotations for the deployment istiod in the istio-system namespace:

ad.datadoghq.com/discovery.logs: |
  [
    {
      "source": "istio",
      "service": "<SERVICE_NAME>"
    }
  ]  

This annotation specifies the container discovery to match the default container name of the Istio container in this pod. Replace this annotation ad.datadoghq.com/<CONTAINER_NAME>.logs with the name (.spec.containers[i].name) of your Istio container if yours differs.

Replace <SERVICE_NAME> with your desired Istio service name.

Envoy access logs

To collect Envoy access logs from your data plane (istio-proxy):

1. Enable Envoy access logging within Istio
2. Apply the following annotation to the pod where the istio-proxy container was injected

ad.datadoghq.com/istio-proxy.logs: |
  [
    {
      "source": "envoy",
      "service": "<SERVICE_NAME>"
    }
  ]  

This annotation specifies the container istio-proxy to match the default container name of the injected Istio sidecar container. Replace this annotation ad.datadoghq.com/<CONTAINER_NAME>.logs with the name (.spec.containers[i].name) of your Istio sidecar container if yours differs.

Replace <SERVICE_NAME> with your desired Istio proxy service name.

Validation

Run the Agent’s info subcommand and look for istio under the Checks section.

Data Collected

Metrics

Events

The Istio check does not include any events.

Service Checks

istio.prometheus.health
Returns CRITICAL if the check cannot access the metrics endpoint. Returns OK otherwise.
Statuses: ok, critical

istio.openmetrics.health
Returns CRITICAL if the Agent is unable to connect to the OpenMetrics endpoint, otherwise returns OK.
Statuses: ok, critical

istio.pilot.prometheus.health
Returns CRITICAL if the check cannot access the metrics endpoint. Returns OK otherwise.
Statuses: ok, critical

istio.galley.prometheus.health
Returns CRITICAL if the check cannot access the metrics endpoint. Returns OK otherwise.
Statuses: ok, critical

istio.citadel.prometheus.health
Returns CRITICAL if the check cannot access the metrics endpoint. Returns OK otherwise.
Statuses: ok, critical

Troubleshooting

Invalid chunk length error

If you see the following error on the legacy mode of the Istio integration (Istio integration version 3.13.0 or earlier):

  Error: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)",
  InvalidChunkLength(got length b'', 0 bytes read))

You can use the latest mode of the OpenMetrics-based Istio integration to resolve this error.

You must upgrade to at minimum Agent 7.31.0 and Python 3. See the Configuration section to enable OpenMetrics.

Using the generic OpenMetrics integration in an Istio deployment

If Istio proxy sidecar injection is enabled, monitoring other Prometheus metrics using the OpenMetrics integration with the same metrics endpoint as istio_mesh_endpoint can result in high custom metrics usage and duplicated metric collection.

To ensure that your OpenMetrics configuration does not redundantly collect metrics, either:

1. Use specific metric matching in the metrics configuration option, or
2. If using the wildcard * value for metrics, consider using the following OpenMetrics integration options to exclude metrics already supported by the Istio and Envoy integrations.

OpenMetrics latest mode configuration with generic metric collection

Be sure to exclude Istio and Envoy metrics from your configuration to avoid high custom metrics billing. Use exclude_metrics if openmetrics_endpoint is enabled.

## Every instance is scheduled independent of the others.
#
instances:
  - openmetrics_endpoint: <OPENMETRICS_ENDPOINT>
    metrics:
    - '.*'
    exclude_metrics:
      - istio_*
      - envoy_*

OpenMetrics legacy mode configuration with generic metric collection

Be sure to exclude Istio and Envoy metrics from your configuration to avoid high custom metrics billing. Use ignore_metrics if prometheus_url is enabled.

instances:
  - prometheus_url: <PROMETHEUS_URL>
    metrics:
      - '*'
    ignore_metrics:
      - istio_*
      - envoy_*

Need help? Contact Datadog support.

Further Reading

Additional helpful documentation, links, and articles:

• Monitor your Istio service mesh with Datadog
• Learn how Datadog collects key metrics to monitor Istio
• How to monitor Istio with Datadog