Istio

Supported OS Linux Windows Mac OS

통합 버전7.0.0
이 페이지는 아직 한국어로 제공되지 않으며 번역 작업 중입니다. 번역에 관한 질문이나 의견이 있으시면 언제든지 저희에게 연락해 주십시오.

Overview

Datadog monitors every aspect of your Istio environment, so you can:

To learn more about monitoring your Istio environment with Datadog, see the Monitor blog post.

Setup

For general instructions on configuring integrations in containerized environments, see Configure integrations with Autodiscovery on Kubernetes or Configure integrations with Autodiscovery on Docker.

This OpenMetrics-based integration has a latest mode (use_openmetrics: true) and a legacy mode (use_openmetrics: false). To get all the most up-to-date features, Datadog recommends enabling latest mode. For more information, see Latest and Legacy Versioning For OpenMetrics-based Integrations.

If you have multiple instances of Datadog collecting Istio metrics, make sure you are using the same mode for all of them. Otherwise, metrics data may fluctuate on the Datadog site.

Metrics marked as [OpenMetrics V1], [OpenMetrics V2], or [OpenMetrics V1 and V2] are only available using the corresponding mode of the Istio integration. Metrics marked as Istio v1.5+ are collected using Istio version 1.5 or later.

Installation

Istio is included in the Datadog Agent. Install the Datadog Agent on your Istio servers or in your cluster and point it at Istio.

Envoy

If you want to monitor the Envoy proxies in Istio, configure the Envoy integration.

Configuration

Metric collection

To monitor Istio v1.5+ there are two key components matching the Istio architecture for the Prometheus-formatted metrics:

  • Data plane: The istio-proxy sidecar containers
  • Control plane: The istiod service managing the proxies

These are both run as istio Agent checks, but they have different responsibilities and are configured separately.

Data plane configuration

The default istio.d/auto_conf.yaml file automatically sets up monitoring for each of the istio-proxy sidecar containers. The Agent initializes this check for each sidecar container that it detects automatically. This configuration enables the reporting of istio.mesh.* metrics for the data exposed by each of these sidecar containers.

To customize the data plane portion of the integration, create a custom Istio configuration file istio.yaml. See Configure integrations on Kubernetes or Configure integrations with Autodiscovery on Docker for options in creating this file.

This file must contain:

ad_identifiers:
  - proxyv2
  - proxyv2-rhel8

init_config:

instances:
  - use_openmetrics: true
    send_histograms_buckets: false
    istio_mesh_endpoint: http://%%host%%:15020/stats/prometheus
    tag_by_endpoint: false

Customize this file with any additional configurations. See the sample istio.d/conf.yaml for all available configuration options.

Control plane configuration

To monitor the Istio control plane and report the mixer, galley, pilot, and citadel metrics, you must configure the Agent to monitor the istiod deployment. In Istio v1.5 or later, apply the following pod annotations for the deployment istiod in the istio-system namespace:

ad.datadoghq.com/discovery.checks: |
  {
    "istio": {
      "instances": [
        {
          "istiod_endpoint": "http://%%host%%:15014/metrics",
          "use_openmetrics": "true"
        }
      ]
    }
  }  

This annotation specifies the container discovery to match the default container name of the Istio container in this pod. Replace this annotation ad.datadoghq.com/<CONTAINER_IDENTIFIER>.checks with the name (.spec.containers[i].name) of your Istio container if yours differs.

The method for applying these annotations varies depending on the Istio deployment strategy (Istioctl, Helm, Operator) used. Consult the Istio documentation for the proper method to apply these pod annotations. See the sample istio.d/conf.yaml for all available configuration options.

Disable sidecar injection for Datadog Agent pods

If you are installing the Datadog Agent in a container, Datadog recommends that you first disable Istio’s sidecar injection.

Istio versions >= 1.10:

Add the sidecar.istio.io/inject: "false" label to the datadog-agent DaemonSet:

# (...)
spec:
  template:
    metadata:
      labels:
        sidecar.istio.io/inject: "false"
    # (...)

This can also be done with the kubectl patch command.

kubectl patch daemonset datadog-agent -p '{"spec":{"template":{"metadata":{"labels":{"sidecar.istio.io/inject":"false"}}}}}'

Istio versions <= 1.9:

Add the sidecar.istio.io/inject: "false" annotation to the datadog-agent DaemonSet:

# (...)
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    # (...)

Using the kubectl patch command:

kubectl patch daemonset datadog-agent -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}}}}}'

Log collection

Available for Agent versions >6.0

First, enable the Datadog Agent to perform log collection in Kubernetes. See Kubernetes Log Collection.

Istio logs

To collect Istio logs from your control plane (istiod), apply the following pod annotations for the deployment istiod in the istio-system namespace:

ad.datadoghq.com/discovery.logs: |
  [
    {
      "source": "istio",
      "service": "<SERVICE_NAME>"
    }
  ]  

This annotation specifies the container discovery to match the default container name of the Istio container in this pod. Replace this annotation ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs with the name (.spec.containers[i].name) of your Istio container if yours differs.

Replace <SERVICE_NAME> with your desired Istio service name.

Envoy access logs

To collect Envoy access logs from your data plane (istio-proxy):

  1. Enable Envoy access logging within Istio
  2. Apply the following annotation to the pod where the istio-proxy container was injected
ad.datadoghq.com/istio-proxy.logs: |
  [
    {
      "source": "envoy",
      "service": "<SERVICE_NAME>"
    }
  ]  

This annotation specifies the container istio-proxy to match the default container name of the injected Istio sidecar container. Replace this annotation ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs with the name (.spec.containers[i].name) of your Istio sidecar container if yours differs.

Replace <SERVICE_NAME> with your desired Istio proxy service name.

Validation

Run the Agent’s info subcommand and look for istio under the Checks section.

Data Collected

Metrics

istio.mesh.request.count
(count)
[OpenMetrics V1 and V2] The number of requests. This metric is sent as gauge by default in OpenMetrics V1.
Shown as request
istio.mesh.request.duration.count
(count)
[OpenMetrics V1 and V2] Count of request durations. This metric is as gauge by default in OpenMetrics V1.
Shown as request
istio.mesh.request.duration.sum
(count)
[OpenMetrics V1 and V2] Sum of request durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as millisecond
istio.mesh.request.size.count
(count)
[OpenMetrics V1 and V2] Count of request sizes. This metric is sent as gauge by default in OpenMetrics V1.
Shown as request
istio.mesh.request.size.sum
(count)
[OpenMetrics V1 and V2] Sum of request sizes. This metric is sent as gauge by default in OpenMetrics V1.
Shown as byte
istio.mesh.response.size.count
(count)
[OpenMetrics V1 and V2] Count of response sizes. This metric is sent as gauge by default in OpenMetrics V1.
Shown as response
istio.mesh.response.size.sum
(count)
[OpenMetrics V1 and V2] Sum of response sizes. This metric is sent as gauge by default in OpenMetrics V1.
Shown as byte
istio.mixer.adapter.dispatch_count
(gauge)
[OpenMetrics V1 and V2] Total number of adapter dispatches handled by Mixer
Shown as operation
istio.mixer.adapter.dispatch_duration.count
(count)
[OpenMetrics V1 and V2] Count of durations for adapter dispatches handled by Mixer. This metric is sent as gauge by default in OpenMetrics V1.
Shown as operation
istio.mixer.adapter.dispatch_duration.sum
(count)
[OpenMetrics V1 and V2] Sum of durations for adapter dispatches handled by Mixer. This metric is sent as gauge by default in OpenMetrics V1.
Shown as operation
istio.mixer.adapter.old_dispatch_count
(gauge)
[OpenMetrics V1 and V2] Total number of adapter dispatches handled by Mixer.
Shown as operation
istio.mixer.adapter.old_dispatch_duration.count
(count)
[OpenMetrics V1 and V2] Count of times for adapter dispatches handled by Mixer. This metric is sent as gauge by default in OpenMetrics V1.
Shown as operation
istio.mixer.adapter.old_dispatch_duration.sum
(count)
[OpenMetrics V1 and V2] Sum of times for adapter dispatches handled by Mixer. This metric is sent as gauge by default in OpenMetrics V1.
Shown as operation
istio.mixer.config.resolve_actions.count
(count)
[OpenMetrics V1 and V2] Count of actions resolved by Mixer. This metric is sent as gauge by default in OpenMetrics V1.
Shown as operation
istio.mixer.config.resolve_actions.sum
(count)
[OpenMetrics V1 and V2] Sum of actions resolved by Mixer. This metric is sent as gauge by default in OpenMetrics V1.
Shown as operation
istio.mixer.config.resolve_count
(gauge)
[OpenMetrics V1 and V2] Number of config resolves handled by mixer
Shown as operation
istio.mixer.config.resolve_duration.count
(count)
[OpenMetrics V1 and V2] Seconds per config resolve. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.mixer.config.resolve_duration.sum
(count)
[OpenMetrics V1 and V2] Sum of times for config resolves handled by Mixer. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.mixer.config.resolve_rules.count
(count)
[OpenMetrics V1 and V2] Number of rules resolved by Mixer. This metric is sent as gauge by default in OpenMetrics V1.
Shown as item
istio.mixer.config.resolve_rules.sum
(count)
[OpenMetrics V1 and V2] Sum of rules resolved by Mixer. This metric is sent as gauge by default in OpenMetrics V1.
Shown as item
istio.mixer.go.gc_duration_seconds.count
(count)
[OpenMetrics V1 and V2] Count of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.mixer.go.gc_duration_seconds.quantile
(gauge)
[OpenMetrics V1 and V2] Quantile of the GC invocation durations.
Shown as second
istio.mixer.go.gc_duration_seconds.sum
(count)
[OpenMetrics V1 and V2] Sum of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.mixer.go.goroutines
(gauge)
[OpenMetrics V1 and V2] Number of goroutines that currently exist.
Shown as thread
istio.mixer.go.info
(gauge)
[OpenMetrics V1 and V2] Information about the Go environment.
istio.mixer.go.memstats.alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes allocated and still in use.
Shown as byte
istio.mixer.go.memstats.alloc_bytes_total
(gauge)
[OpenMetrics V1 and V2] Total number of bytes allocated even if freed.
Shown as byte
istio.mixer.go.memstats.buck_hash_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the profiling bucket hash table.
Shown as byte
istio.mixer.go.memstats.frees_total
(gauge)
[OpenMetrics V1 and V2] Total number of frees.
istio.mixer.go.memstats.gc_cpu_fraction
(gauge)
[OpenMetrics V1 and V2] CPU taken up by GC
Shown as percent
istio.mixer.go.memstats.gc_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for garbage collection system metadata.
Shown as byte
istio.mixer.go.memstats.heap_alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Bytes allocated to the heap
Shown as byte
istio.mixer.go.memstats.heap_idle_bytes
(gauge)
[OpenMetrics V1 and V2] Number of idle bytes in the heap
Shown as byte
istio.mixer.go.memstats.heap_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of Bytes in the heap
Shown as byte
istio.mixer.go.memstats.heap_objects
(gauge)
[OpenMetrics V1 and V2] Number of objects in the heap
Shown as object
istio.mixer.go.memstats.heap_released_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes released to the system in the last gc
Shown as byte
istio.mixer.go.memstats.heap_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the heap
Shown as byte
istio.mixer.go.memstats.last_gc_time_seconds
(gauge)
[OpenMetrics V1 and V2] Length of last GC
Shown as second
istio.mixer.go.memstats.lookups_total
(gauge)
[OpenMetrics V1 and V2] Number of lookups
Shown as operation
istio.mixer.go.memstats.mallocs_total
(gauge)
[OpenMetrics V1 and V2] Number of mallocs
Shown as operation
istio.mixer.go.memstats.mcache_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mcache structures.
Shown as byte
istio.mixer.go.memstats.mcache_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mcache structures obtained from system.
Shown as byte
istio.mixer.go.memstats.mspan_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mspan structures.
Shown as byte
istio.mixer.go.memstats.mspan_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mspan structures obtained from system.
Shown as byte
istio.mixer.go.memstats.next_gc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of heap bytes when next garbage collection will take place
Shown as byte
istio.mixer.go.memstats.other_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for other system allocations
Shown as byte
istio.mixer.go.memstats.stack_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by the stack allocator
Shown as byte
istio.mixer.go.memstats.stack_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system for stack allocator
Shown as byte
istio.mixer.go.memstats.sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system
Shown as byte
istio.mixer.go.threads
(gauge)
[OpenMetrics V1 and V2] Number of OS threads created.
Shown as thread
istio.mixer.grpc.server.handled_total
(gauge)
[OpenMetrics V1 and V2] Total number of fully handled requests, with responses
Shown as request
istio.mixer.grpc.server.handling_seconds.count
(count)
[OpenMetrics V1 and V2] Count of response latency (seconds) of gRPC that had been application-level handled by the server. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.mixer.grpc.server.handling_seconds.sum
(count)
[OpenMetrics V1 and V2] Sum of response latency (seconds) of gRPC that had been application-level handled by the server. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.mixer.grpc.server.msg_received_total
(gauge)
[OpenMetrics V1 and V2] Total number of RPC stream messages received on the server.
Shown as message
istio.mixer.grpc.server.msg_sent_total
(gauge)
[OpenMetrics V1 and V2] Total number of messages sent
Shown as message
istio.mixer.grpc.server.started_total
(gauge)
[OpenMetrics V1 and V2] Total number of RPCs started on the server.
istio.mixer.process.cpu_seconds_total
(gauge)
[OpenMetrics V1 and V2] Total user and system CPU time spent in seconds.
Shown as second
istio.mixer.process.max_fds
(gauge)
[OpenMetrics V1 and V2] Maximum number of open file descriptors.
Shown as file
istio.mixer.process.open_fds
(gauge)
[OpenMetrics V1 and V2] Number of open file descriptors.
Shown as file
istio.mixer.process.resident_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Resident memory size in bytes.
Shown as byte
istio.mixer.process.start_time_seconds
(gauge)
[OpenMetrics V1 and V2] Start time of the process since unix epoch in seconds.
Shown as second
istio.mixer.process.virtual_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Virtual memory size in bytes.
Shown as byte
istio.mixer.grpc_io_server.completed_rpcs
(gauge)
[OpenMetrics V1 and V2] Count of RPCs by method and status.
istio.mixer.grpc_io_server.received_bytes_per_rpc
(gauge)
[OpenMetrics V1 and V2] Distribution of received bytes per RPC, by method.
Shown as byte
istio.mixer.grpc_io_server.sent_bytes_per_rpc
(gauge)
[OpenMetrics V1 and V2] Distribution of total sent bytes per RPC, by method.
Shown as byte
istio.mixer.grpc_io_server.server_latency
(gauge)
[OpenMetrics V1 and V2] Distribution of server latency in milliseconds, by method.
istio.mixer.config.attributes_total
(gauge)
[OpenMetrics V1 and V2] The number of known attributes in the current config.
istio.mixer.config.handler_configs_total
(gauge)
[OpenMetrics V1 and V2] The number of known handlers in the current config.
istio.mixer.config.instance_configs_total
(gauge)
[OpenMetrics V1 and V2] The number of known instances in the current config.
istio.mixer.config.rule_configs_total
(gauge)
[OpenMetrics V1 and V2] The number of known rules in the current config.
istio.mixer.dispatcher.destinations_per_request
(gauge)
[OpenMetrics V1 and V2] Number of handlers dispatched per request by Mixer.
istio.mixer.dispatcher.instances_per_request
(gauge)
[OpenMetrics V1 and V2] Number of instances created per request by Mixer.
istio.mixer.handler.daemons_total
(gauge)
[OpenMetrics V1 and V2] The current number of active daemon routines in a given adapter environment.
istio.mixer.handler.new_handlers_total
(gauge)
[OpenMetrics V1 and V2] The number of handlers that were newly created during config transition.
istio.mixer.mcp_sink.reconnections
(gauge)
[OpenMetrics V1 and V2] The number of times the sink has reconnected.
istio.mixer.mcp_sink.request_acks_total
(gauge)
[OpenMetrics V1 and V2] The number of request acks received by the source.
istio.mixer.runtime.dispatches_total
(gauge)
[OpenMetrics V1 and V2] Total number of adapter dispatches handled by Mixer.
Shown as operation
istio.mixer.runtime.dispatch_duration_seconds
(gauge)
[OpenMetrics V1 and V2] Duration in seconds for adapter dispatches handled by Mixer.
Shown as second
istio.pilot.go.gc_duration_seconds.count
(count)
[OpenMetrics V1 and V2] Count of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.pilot.go.gc_duration_seconds.quantile
(gauge)
[OpenMetrics V1 and V2] Quantile of the GC invocation durations.
Shown as second
istio.pilot.go.gc_duration_seconds.sum
(count)
[OpenMetrics V1 and V2] Sum of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.pilot.go.goroutines
(gauge)
[OpenMetrics V1 and V2] Number of goroutines that currently exist.
Shown as thread
istio.pilot.go.info
(gauge)
[OpenMetrics V1 and V2] Information about the Go environment.
istio.pilot.go.memstats.alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes allocated and still in use.
Shown as byte
istio.pilot.go.memstats.alloc_bytes_total
(gauge)
[OpenMetrics V1 and V2] Total number of bytes allocated even if freed.
Shown as byte
istio.pilot.go.memstats.buck_hash_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the profiling bucket hash table.
Shown as byte
istio.pilot.go.memstats.frees_total
(gauge)
[OpenMetrics V1 and V2] Total number of frees.
istio.pilot.go.memstats.gc_cpu_fraction
(gauge)
[OpenMetrics V1 and V2] CPU taken up by GC
Shown as percent
istio.pilot.go.memstats.gc_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for garbage collection system metadata.
Shown as byte
istio.pilot.go.memstats.heap_alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Bytes allocated to the heap
Shown as byte
istio.pilot.go.memstats.heap_idle_bytes
(gauge)
[OpenMetrics V1 and V2] Number of idle bytes in the heap
Shown as byte
istio.pilot.go.memstats.heap_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of Bytes in the heap
Shown as byte
istio.pilot.go.memstats.heap_objects
(gauge)
[OpenMetrics V1 and V2] Number of objects in the heap
Shown as object
istio.pilot.go.memstats.heap_released_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes released to the system in the last gc
Shown as byte
istio.pilot.go.memstats.heap_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the heap
Shown as byte
istio.pilot.go.memstats.last_gc_time_seconds
(gauge)
[OpenMetrics V1 and V2] Length of last GC
Shown as second
istio.pilot.go.memstats.lookups_total
(gauge)
[OpenMetrics V1 and V2] Number of lookups
Shown as operation
istio.pilot.go.memstats.mallocs_total
(gauge)
[OpenMetrics V1 and V2] Number of mallocs
Shown as operation
istio.pilot.go.memstats.mcache_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mcache structures.
Shown as byte
istio.pilot.go.memstats.mcache_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mcache structures obtained from system.
Shown as byte
istio.pilot.go.memstats.mspan_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mspan structures.
Shown as byte
istio.pilot.go.memstats.mspan_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mspan structures obtained from system.
Shown as byte
istio.pilot.go.memstats.next_gc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of heap bytes when next garbage collection will take place
Shown as byte
istio.pilot.go.memstats.other_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for other system allocations
Shown as byte
istio.pilot.go.memstats.stack_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by the stack allocator
Shown as byte
istio.pilot.go.memstats.stack_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system for stack allocator
Shown as byte
istio.pilot.go.memstats.sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system
Shown as byte
istio.pilot.go.threads
(gauge)
[OpenMetrics V1 and V2] Number of OS threads created.
Shown as thread
istio.pilot.process.cpu_seconds_total
(gauge)
[OpenMetrics V1 and V2] Total user and system CPU time spent in seconds.
Shown as second
istio.pilot.process.max_fds
(gauge)
[OpenMetrics V1 and V2] Maximum number of open file descriptors.
Shown as file
istio.pilot.process.open_fds
(gauge)
[OpenMetrics V1 and V2] Number of open file descriptors.
Shown as file
istio.pilot.process.resident_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Resident memory size in bytes.
Shown as byte
istio.pilot.process.start_time_seconds
(gauge)
[OpenMetrics V1 and V2] Start time of the process since unix epoch in seconds.
Shown as second
istio.pilot.process.virtual_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Virtual memory size in bytes.
Shown as byte
istio.pilot.conflict.inbound_listener
(gauge)
[OpenMetrics V1 and V2] Number of conflicting inbound listeners.
istio.pilot.conflict.outbound_listener.http_over_current_tcp
(gauge)
[OpenMetrics V1 and V2] Number of conflicting wildcard http listeners with current wildcard tcp listener.
istio.pilot.conflict.outbound_listener.tcp_over_current_http
(gauge)
[OpenMetrics V1 and V2] Number of conflicting wildcard tcp listeners with current wildcard http listener.
istio.pilot.conflict.outbound_listener.tcp_over_current_tcp
(gauge)
[OpenMetrics V1 and V2] Number of conflicting tcp listeners with current tcp listener.
istio.pilot.destrule_subsets
(gauge)
[OpenMetrics V1 and V2] Duplicate subsets across destination rules for same host.
istio.pilot.duplicate_envoy_clusters
(gauge)
[OpenMetrics V1 and V2] Duplicate envoy clusters caused by service entries with same hostname.
istio.pilot.eds_no_instances
(gauge)
[OpenMetrics V1 and V2] Number of clusters without instances.
istio.pilot.endpoint_not_ready
(gauge)
[OpenMetrics V1 and V2] Endpoint found in unready state.
istio.pilot.invalid_out_listeners
(gauge)
[OpenMetrics V1 and V2] Number of invalid outbound listeners.
istio.pilot.mcp_sink.reconnections
(count)
[OpenMetrics V1 and V2] The number of times the sink has reconnected.
istio.pilot.mcp_sink.recv_failures_total
(count)
[OpenMetrics V1 and V2] The number of recv failures in the source.
istio.pilot.mcp_sink.request_acks_total
(count)
[OpenMetrics V1 and V2] The number of request acks received by the source.
istio.pilot.no_ip
(gauge)
[OpenMetrics V1 and V2] Pods not found in the endpoint table, possibly invalid.
istio.pilot.proxy_convergence_time
(gauge)
[OpenMetrics V1 and V2] Delay between config change and all proxies converging.
Shown as second
istio.pilot.proxy_convergence_time.sum
(count)
[OpenMetrics V1 and V2] Sum of the delay between config change and all proxies converging. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.pilot.proxy_convergence_time.count
(count)
[OpenMetrics V1 and V2] Number of entries of pilot.proxyconvergencetime. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.pilot.rds_expired_nonce
(count)
[OpenMetrics V1 and V2] Total number of RDS messages with an expired nonce.
istio.pilot.services
(gauge)
[OpenMetrics V1 and V2] Total services known to pilot.
istio.pilot.total_xds_internal_errors
(count)
[OpenMetrics V1 and V2] Total number of internal XDS errors in pilot.
istio.pilot.total_xds_rejects
(count)
[OpenMetrics V1 and V2] Total number of XDS responses from pilot rejected by proxy.
istio.pilot.virt_services
(gauge)
[OpenMetrics V1 and V2] Total virtual services known to pilot.
istio.pilot.vservice_dup_domain
(gauge)
[OpenMetrics V1 and V2] Virtual services with dup domains.
istio.pilot.xds
(gauge)
[OpenMetrics V1 and V2] Number of endpoints connected to this pilot using XDS.
istio.pilot.xds.eds_instances
(gauge)
[OpenMetrics V1 and V2] Instances for each cluster, as of last push.
istio.pilot.xds.push.context_errors
(count)
[OpenMetrics V1 and V2] Number of errors (timeouts) initiating push context.
istio.pilot.xds.push.timeout
(count)
[OpenMetrics V1 and V2] Pilot push timeout, will retry.
istio.pilot.xds.push.timeout_failures
(count)
[OpenMetrics V1 and V2] Pilot push timeout failures after repeated attempts.
istio.pilot.xds.pushes
(count)
[OpenMetrics V1 and V2] Pilot build and send errors for lds, rds, cds and eds.
istio.pilot.xds.write_timeout
(count)
[OpenMetrics V1 and V2] Pilot XDS response write timeouts.
istio.pilot.xds.rds_reject
(count)
[OpenMetrics V1 and V2] Pilot rejected RDS.
istio.pilot.xds.lds_reject
(count)
[OpenMetrics V1 and V2] Pilot rejected LDS.
istio.pilot.xds.eds_reject
(count)
[OpenMetrics V1 and V2] Pilot rejected EDS.
istio.pilot.xds.cds_reject
(count)
[OpenMetrics V1 and V2] Pilot rejected CDS.
istio.galley.go.gc_duration_seconds.count
(count)
[OpenMetrics V1 and V2] Count of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.galley.go.gc_duration_seconds.quantile
(gauge)
[OpenMetrics V1 and V2] Quantile of the GC invocation durations.
Shown as second
istio.galley.go.gc_duration_seconds.sum
(count)
[OpenMetrics V1 and V2] Sum of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.galley.go.goroutines
(gauge)
[OpenMetrics V1 and V2] Number of goroutines that currently exist.
Shown as thread
istio.galley.go.info
(gauge)
[OpenMetrics V1 and V2] Information about the Go environment.
istio.galley.go.memstats.alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes allocated and still in use.
Shown as byte
istio.galley.go.memstats.alloc_bytes_total
(gauge)
[OpenMetrics V1 and V2] Total number of bytes allocated even if freed.
Shown as byte
istio.galley.go.memstats.buck_hash_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the profiling bucket hash table.
Shown as byte
istio.galley.go.memstats.frees_total
(gauge)
[OpenMetrics V1 and V2] Total number of frees.
istio.galley.go.memstats.gc_cpu_fraction
(gauge)
[OpenMetrics V1 and V2] CPU taken up by GC
Shown as percent
istio.galley.go.memstats.gc_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for garbage collection system metadata.
Shown as byte
istio.galley.go.memstats.heap_alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Bytes allocated to the heap
Shown as byte
istio.galley.go.memstats.heap_idle_bytes
(gauge)
[OpenMetrics V1 and V2] Number of idle bytes in the heap
Shown as byte
istio.galley.go.memstats.heap_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of Bytes in the heap
Shown as byte
istio.galley.go.memstats.heap_objects
(gauge)
[OpenMetrics V1 and V2] Number of objects in the heap
Shown as object
istio.galley.go.memstats.heap_released_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes released to the system in the last gc
Shown as byte
istio.galley.go.memstats.heap_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the heap
Shown as byte
istio.galley.go.memstats.last_gc_time_seconds
(gauge)
[OpenMetrics V1 and V2] Length of last GC
Shown as second
istio.galley.go.memstats.lookups_total
(gauge)
[OpenMetrics V1 and V2] Number of lookups
Shown as operation
istio.galley.go.memstats.mallocs_total
(gauge)
[OpenMetrics V1 and V2] Number of mallocs
Shown as operation
istio.galley.go.memstats.mcache_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mcache structures.
Shown as byte
istio.galley.go.memstats.mcache_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mcache structures obtained from system.
Shown as byte
istio.galley.go.memstats.mspan_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mspan structures.
Shown as byte
istio.galley.go.memstats.mspan_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mspan structures obtained from system.
Shown as byte
istio.galley.go.memstats.next_gc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of heap bytes when next garbage collection will take place
Shown as byte
istio.galley.go.memstats.other_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for other system allocations
Shown as byte
istio.galley.go.memstats.stack_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by the stack allocator
Shown as byte
istio.galley.go.memstats.stack_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system for stack allocator
Shown as byte
istio.galley.go.memstats.sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system
Shown as byte
istio.galley.go.threads
(gauge)
[OpenMetrics V1 and V2] Number of OS threads created.
Shown as thread
istio.galley.process.cpu_seconds_total
(gauge)
[OpenMetrics V1 and V2] Total user and system CPU time spent in seconds.
Shown as second
istio.galley.process.max_fds
(gauge)
[OpenMetrics V1 and V2] Maximum number of open file descriptors.
Shown as file
istio.galley.process.open_fds
(gauge)
[OpenMetrics V1 and V2] Number of open file descriptors.
Shown as file
istio.galley.process.resident_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Resident memory size in bytes.
Shown as byte
istio.galley.process.start_time_seconds
(gauge)
[OpenMetrics V1 and V2] Start time of the process since unix epoch in seconds.
Shown as second
istio.galley.process.virtual_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Virtual memory size in bytes.
Shown as byte
istio.galley.endpoint_no_pod
(gauge)
[OpenMetrics V1 and V2] Endpoints without an associated pod.
istio.galley.mcp_source.clients_total
(gauge)
[OpenMetrics V1 and V2] The number of streams currently connected.
istio.galley.runtime_processor.event_span_duration_milliseconds
(gauge)
[OpenMetrics V1 and V2] The duration between each incoming event.
Shown as millisecond
istio.galley.runtime_processor.events_processed_total
(gauge)
[OpenMetrics V1 and V2] The number of events that have been processed.
istio.galley.runtime_processor.snapshot_events_total.bucket
(count)
[OpenMetrics V2] The number of events per snapshot per upper_bound.
istio.galley.runtime_processor.snapshot_events_total.count
(count)
[OpenMetrics V1 and V2] The number of events per snapshot.
istio.galley.runtime_processor.snapshot_events_total.sum
(count)
[OpenMetrics V1 and V2] The duration of snapshot events.
istio.galley.runtime_processor.snapshot_lifetime_duration_milliseconds
(gauge)
[OpenMetrics V1 and V2] The duration of each snapshot.
Shown as millisecond
istio.galley.runtime_processor.snapshots_published_total
(count)
[OpenMetrics V1 and V2] The number of snapshots that have been published.
istio.galley.runtime_state_type_instances_total
(gauge)
[OpenMetrics V1 and V2] The number of type instances per type URL.
istio.galley.runtime_strategy.on_change_total
(count)
[OpenMetrics V1 and V2] The number of times the strategy's onChange has been called.
istio.galley.runtime_strategy.timer_max_time_reached_total
(count)
[OpenMetrics V1 and V2] The number of times the max time has been reached.
istio.galley.runtime_strategy.quiesce_reached_total
(count)
[OpenMetrics V1 and V2] The number of times a quiesce has been reached.
istio.galley.runtime_strategy.timer_resets_total
(count)
[OpenMetrics V1 and V2] The number of times the timer has been reset.
istio.galley.source_kube.dynamic_converter_success_total
(count)
[OpenMetrics V1 and V2] The number of times a dynamic kubernetes source successfully converted a resource.
istio.galley.source_kube.event_success_total
(count)
[OpenMetrics V1 and V2] The number of times a kubernetes source successfully handled an event.
istio.galley.validation.cert_key_updates
(count)
[OpenMetrics V1 and V2] Galley validation webhook certificate updates.
istio.galley.validation.config_load
(count)
[OpenMetrics V1 and V2] K8s webhook configuration (re)loads.
istio.galley.validation.config_update
(count)
[OpenMetrics V1 and V2] K8s webhook configuration updates.
istio.galley.validation.passed
(count)
[OpenMetrics V1 and V2] Resource is valid.
istio.citadel.secret_controller.csr_err_count
(count)
[OpenMetrics V1 and V2] The number of errors occurred when creating the CSR.
istio.citadel.secret_controller.secret_deleted_cert_count
(count)
[OpenMetrics V1 and V2] The number of certificates recreated due to secret deletion (service account still exists).
istio.citadel.secret_controller.svc_acc_created_cert_count
(count)
[OpenMetrics V1 and V2] The number of certificates created due to service account creation.
istio.citadel.secret_controller.svc_acc_deleted_cert_count
(count)
[OpenMetrics V1 and V2] The number of certificates deleted due to service account deletion.
istio.citadel.server.authentication_failure_count
(count)
[OpenMetrics V1 and V2] The number of authentication failures.
Shown as error
istio.citadel.server.citadel_root_cert_expiry_timestamp
(gauge)
[OpenMetrics V1 and V2] The unix timestamp, in seconds, when Citadel root cert will expire. We set it to negative in case of internal error.
Shown as second
istio.citadel.server.csr_count
(count)
[OpenMetrics V1 and V2] The number of CSRs received by Citadel server.
istio.citadel.server.csr_parsing_err_count
(count)
[OpenMetrics V1 and V2] The number of errors occurred when parsing the CSR.
Shown as error
istio.citadel.server.id_extraction_err_count
(count)
[OpenMetrics V1 and V2] The number of errors occurred when extracting the ID from CSR.
Shown as error
istio.citadel.server.success_cert_issuance_count
(count)
[OpenMetrics V1 and V2] The number of certificates issuances that have succeeded.
istio.citadel.go.gc_duration_seconds.count
(count)
[OpenMetrics V1 and V2] Count of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.citadel.go.gc_duration_seconds.quantile
(gauge)
[OpenMetrics V1 and V2] Quantile of the GC invocation durations.
Shown as second
istio.citadel.go.gc_duration_seconds.sum
(count)
[OpenMetrics V1 and V2] Sum of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.citadel.go.goroutines
(gauge)
[OpenMetrics V1 and V2] Number of goroutines that currently exist.
Shown as thread
istio.citadel.go.info
(gauge)
[OpenMetrics V1 and V2] Information about the Go environment.
istio.citadel.go.memstats.alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes allocated and still in use.
Shown as byte
istio.citadel.go.memstats.alloc_bytes_total
(count)
[OpenMetrics V1 and V2] Total number of bytes allocated even if freed.
Shown as byte
istio.citadel.go.memstats.buck_hash_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the profiling bucket hash table.
Shown as byte
istio.citadel.go.memstats.frees_total
(count)
[OpenMetrics V1 and V2] Total number of frees.
istio.citadel.go.memstats.gc_cpu_fraction
(gauge)
[OpenMetrics V1 and V2] CPU taken up by GC
Shown as percent
istio.citadel.go.memstats.gc_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for garbage collection system metadata.
Shown as byte
istio.citadel.go.memstats.heap_alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Bytes allocated to the heap
Shown as byte
istio.citadel.go.memstats.heap_idle_bytes
(gauge)
[OpenMetrics V1 and V2] Number of idle bytes in the heap
Shown as byte
istio.citadel.go.memstats.heap_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of Bytes in the heap
Shown as byte
istio.citadel.go.memstats.heap_objects
(gauge)
[OpenMetrics V1 and V2] Number of objects in the heap
Shown as object
istio.citadel.go.memstats.heap_released_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes released to the system in the last gc
Shown as byte
istio.citadel.go.memstats.heap_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the heap
Shown as byte
istio.citadel.go.memstats.last_gc_time_seconds
(gauge)
[OpenMetrics V1 and V2] Length of last GC
Shown as second
istio.citadel.go.memstats.lookups_total
(count)
[OpenMetrics V1 and V2] Number of lookups
Shown as operation
istio.citadel.go.memstats.mallocs_total
(count)
[OpenMetrics V1 and V2] Number of mallocs
Shown as operation
istio.citadel.go.memstats.mcache_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mcache structures.
Shown as byte
istio.citadel.go.memstats.mcache_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mcache structures obtained from system.
Shown as byte
istio.citadel.go.memstats.mspan_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mspan structures.
Shown as byte
istio.citadel.go.memstats.mspan_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mspan structures obtained from system.
Shown as byte
istio.citadel.go.memstats.next_gc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of heap bytes when next garbage collection will take place
Shown as byte
istio.citadel.go.memstats.other_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for other system allocations
Shown as byte
istio.citadel.go.memstats.stack_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by the stack allocator
Shown as byte
istio.citadel.go.memstats.stack_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system for stack allocator
Shown as byte
istio.citadel.go.memstats.sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system
Shown as byte
istio.citadel.go.threads
(gauge)
[OpenMetrics V1 and V2] Number of OS threads created.
Shown as thread
istio.citadel.process.cpu_seconds_total
(gauge)
[OpenMetrics V1 and V2] Total user and system CPU time spent in seconds.
Shown as second
istio.citadel.process.max_fds
(gauge)
[OpenMetrics V1 and V2] Maximum number of open file descriptors.
Shown as file
istio.citadel.process.open_fds
(gauge)
[OpenMetrics V1 and V2] Number of open file descriptors.
Shown as file
istio.citadel.process.resident_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Resident memory size in bytes.
Shown as byte
istio.citadel.process.start_time_seconds
(gauge)
[OpenMetrics V1 and V2] Start time of the process since unix epoch in seconds.
Shown as second
istio.citadel.process.virtual_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Virtual memory size in bytes.
Shown as byte
istio.galley.validation.config_update_error
(count)
[OpenMetrics V1 and V2] K8s webhook configuration update error
Shown as error
istio.citadel.server.cert_chain_expiry_timestamp
(gauge)
[OpenMetrics V1 and V2] The unix timestamp (in seconds) when Citadel cert chain will expire. Negative in case of internal error
Shown as second
istio.citadel.server.root_cert_expiry_timestamp
(gauge)
[OpenMetrics V1 and V2] The unix timestamp (in seconds) when Citadel root cert will expire. Negative in case of internal error
Shown as second
istio.galley.validation.failed
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Count of resource validation failed
istio.pilot.conflict.outbound_listener.http_over_https
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of conflicting HTTP listeners with well known HTTPS ports
istio.pilot.inbound_updates
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of updates received by pilot
istio.pilot.k8s.cfg_events
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Events from k8s config
Shown as event
istio.pilot.k8s.reg_events
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Events from k8s registry
Shown as event
istio.pilot.proxy_queue_time.count
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Count of observation for when proxy is in a push queue before being dequeued. This metric is sent as gauge by default in OpenMetrics V1.
istio.pilot.proxy_queue_time.sum
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Sum of observed values for when proxy is in a push queue before being dequeued. This metric is sent as gauge by default in OpenMetrics V1.
istio.pilot.push.triggers
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of times a push was triggered
Shown as event
istio.pilot.xds.eds_all_locality_endpoints
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Network endpoints for each cluster (across all localities) as of last push. Zero endpoints is an error
istio.pilot.xds.push.time.count
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Count of observation of total time Pilot takes a push. This metric is sent as gauge by default in OpenMetrics V1.
istio.pilot.xds.push.time.sum
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Sum of observed values of total time Pilot takes a push. This metric is sent as gauge by default in OpenMetrics V1.
istio.sidecar_injection.requests_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of sidecar injection requests
Shown as request
istio.sidecar_injection.success_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of successful sidecar injection requests
Shown as request
istio.sidecar_injection.failure_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of failed sidecar injection requests
Shown as request
istio.sidecar_injection.skip_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of skipped sidecar injection requests
Shown as request
istio.mesh.request.duration.milliseconds.sum
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total sum of observed values for duration of requests in ms. This metric is sent as gauge by default in OpenMetrics V1.
Shown as millisecond
istio.mesh.request.duration.milliseconds.count
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total count of observed values for duration of requests. This metric is sent as gauge by default in OpenMetrics V1.
istio.mesh.tcp.connections_closed.total
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Total closed connections
istio.mesh.tcp.connections_opened.total
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Total opened connections
istio.mesh.tcp.received_bytes.total
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Size of total bytes received during request in case of a TCP connection
Shown as byte
istio.mesh.tcp.send_bytes.total
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Size of total bytes sent during response in case of a TCP connection
Shown as byte
istio.mesh.request.count.total
(count)
[OpenMetrics V1 and Istio v1.5+] The number of requests as monotonic count
Shown as request
istio.mesh.request.duration.milliseconds.count.total
(count)
[OpenMetrics V1 and Istio v1.5+] Total count of observed values for duration of requests as monotonic count
istio.mesh.request.duration.milliseconds.sum.total
(count)
[OpenMetrics V1 and Istio v1.5+] Total sum of observed values for duration of requests as monotonic count
istio.mesh.request.size.count.total
(count)
[OpenMetrics V1 and Istio v1.5+] Count of observed request sizes as monotonic count
istio.mesh.request.size.sum.total
(count)
[OpenMetrics V1 and Istio v1.5+] Sum of observed request sizes as monotonic count
istio.mesh.response.size.count.total
(count)
[OpenMetrics V1 and Istio v1.5+] Count of observed response size as monotonic count
istio.mesh.response.size.sum.total
(count)
[OpenMetrics V1 and Istio v1.5+] Sum of observed response size as monotonic count
istio.mesh.tcp.connections_closed.total.total
(count)
[OpenMetrics V1 and Istio v1.5+] Total closed connections as monotonic count
istio.mesh.tcp.connections_opened.total.total
(count)
[OpenMetrics V1 and Istio v1.5+] Total opened connections as monotonic count
istio.mesh.tcp.received_bytes.total.total
(count)
[OpenMetrics V1 and Istio v1.5+] Size of total bytes received during request in case of a TCP connection as monotonic count
Shown as byte
istio.mesh.tcp.send_bytes.total.total
(count)
[OpenMetrics V1 and Istio v1.5+] Size of total bytes sent during response in case of a TCP connection as monotonic count
Shown as byte
istio.mesh.request.duration.count.total
(count)
[OpenMetrics V1 and Istio v1.5+] Count of request durations as monotonic count
Shown as request
istio.mesh.request.duration.sum.total
(count)
[OpenMetrics V1 and Istio v1.5+] Sum of request durations as monotonic count
Shown as millisecond
istio.grpc.server.handled_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of RPCs completed on the server, regardless of success or failure.
istio.grpc.server.handling_seconds.count
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Response latency of gRPC that had been application-level handled by the server. This metric is sent as gauge by default in OpenMetrics V1.
istio.grpc.server.handling_seconds.sum
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Response latency of gRPC that had been application-level handled by the server. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.grpc.server.msg_received_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of RPC stream messages received on the server.
Shown as message
istio.grpc.server.msg_sent_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of gRPC stream messages sent by the server.
Shown as message
istio.grpc.server.started_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of RPCs started on the server.
istio.go.gc_duration_seconds.count
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Count of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.go.gc_duration_seconds.quantile
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Quantile of the GC invocation durations.
Shown as second
istio.go.gc_duration_seconds.sum
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Sum of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.go.goroutines
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of goroutines that currently exist.
Shown as thread
istio.go.info
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Information about the Go environment.
istio.go.memstats.alloc_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes allocated and still in use.
Shown as byte
istio.go.memstats.alloc_bytes_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of bytes allocated even if freed.
Shown as byte
istio.go.memstats.buck_hash_sys_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes used by the profiling bucket hash table.
Shown as byte
istio.go.memstats.frees_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total number of frees.
Shown as byte
istio.go.memstats.gc_cpu_fraction
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] CPU taken up by GC.
Shown as percent
istio.go.memstats.gc_sys_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes used for garbage collection system metadata.
Shown as byte
istio.go.memstats.heap_alloc_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Bytes allocated to the heap.
Shown as byte
istio.go.memstats.heap_idle_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of idle bytes in the heap.
Shown as byte
istio.go.memstats.heap_inuse_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes in the heap.
Shown as byte
istio.go.memstats.heap_objects
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of objects in the heap.
Shown as object
istio.go.memstats.heap_released_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes released to the system in the last gc.
Shown as byte
istio.go.memstats.heap_sys_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes used by the heap.
Shown as byte
istio.go.memstats.last_gc_time_seconds
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Length of last GC.
Shown as second
istio.go.memstats.lookups_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of lookups.
Shown as operation
istio.go.memstats.mallocs_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of mallocs
Shown as operation
istio.go.memstats.mcache_inuse_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes in use by mcache structures.
Shown as byte
istio.go.memstats.mcache_sys_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes used for mcache structures obtained from system.
Shown as byte
istio.go.memstats.mspan_inuse_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes in use by mspan structures.
Shown as byte
istio.go.memstats.mspan_sys_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes used for mspan structures obtained from system.
Shown as byte
istio.go.memstats.next_gc_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of heap bytes when next garbage collection will take place.
Shown as byte
istio.go.memstats.other_sys_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes used for other system allocations.
Shown as byte
istio.go.memstats.stack_inuse_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes in use by the stack allocator.
Shown as byte
istio.go.memstats.stack_sys_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes obtained from system for stack allocator.
Shown as byte
istio.go.memstats.sys_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of bytes obtained from system.
Shown as byte
istio.go.threads
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of OS threads created.
Shown as thread
istio.process.cpu_seconds_total
(count)
[OpenMetrics V1 and V2 and Istio v1.5+] Total user and system CPU time spent.
Shown as second
istio.process.max_fds
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Maximum number of open file descriptors.
Shown as file
istio.process.open_fds
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Number of open file descriptors.
Shown as file
istio.process.resident_memory_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Resident memory size in bytes.
Shown as byte
istio.process.start_time_seconds
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Start time of the process since unix epoch in seconds.
Shown as second
istio.process.virtual_memory_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Amount of virtual memory used.
Shown as byte
istio.process.virtual_memory_max_bytes
(gauge)
[OpenMetrics V1 and V2 and Istio v1.5+] Maximum amount of virtual memory available.
Shown as byte
istio.galley.validation.config_update_error.count
(count)
[OpenMetrics V2 and Istio v1.5+] K8s webhook configuration update error
Shown as error
istio.galley.validation.config_update.count
(count)
[OpenMetrics V2 and Istio v1.5+] k8s webhook configuration updates.
istio.galley.validation.failed.count
(count)
[OpenMetrics V2 and Istio v1.5+] Count of resource validation failed
istio.go.memstats.frees.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total number of frees.
istio.go.memstats.lookups.count
(count)
[OpenMetrics V2 and Istio v1.5+] Number of lookups
Shown as operation
istio.go.memstats.mallocs.count
(count)
[OpenMetrics V2 and Istio v1.5+] Number of mallocs
Shown as byte
istio.grpc.server.handled.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total number of fully handled requests, with responses
Shown as request
istio.grpc.server.msg_received.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total number of RPC stream messages received on the server.
Shown as message
istio.grpc.server.msg_sent.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total number of messages sent
Shown as message
istio.grpc.server.started.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total number of RPCs started on the server.
istio.pilot.inbound_updates.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total number of updates received by pilot
istio.pilot.k8s.cfg_events.count
(count)
[OpenMetrics V2 and Istio v1.5+] Events from k8s config
Shown as event
istio.pilot.k8s.reg_events.count
(count)
[OpenMetrics V2 and Istio v1.5+] Events from k8s registry
Shown as event
istio.pilot.push.triggers.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total number of times a push was triggered
istio.pilot.xds.pushes.count
(count)
[OpenMetrics V2 and Istio v1.5+] Pilot build and send errors for lds, rds, cds and eds.
istio.process.cpu_seconds.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total user and system CPU time spent in seconds.
Shown as second
istio.sidecar_injection.requests.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total number of sidecar injection requests
Shown as request
istio.sidecar_injection.success.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total number of successful sidecar injection requests
Shown as request
istio.mesh.tcp.connections_closed.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total closed connections as monotonic count
istio.mesh.tcp.connections_opened.count
(count)
[OpenMetrics V2 and Istio v1.5+] Total opened connections as monotonic count
istio.mesh.tcp.received_bytes.count
(count)
[OpenMetrics V2 and Istio v1.5+] Size of total bytes received during request in case of a TCP connection as monotonic count
Shown as byte
istio.mesh.tcp.send_bytes.count
(count)
[OpenMetrics V2 and Istio v1.5+] Size of total bytes sent during response in case of a TCP connection as monotonic count
Shown as byte
istio.grpc.server.handling_seconds.bucket
(count)
[OpenMetrics V2 and Istio v1.5+] Bucket of response latency (seconds) of gRPC that had been application-level handled by the server.
Shown as second
istio.pilot.proxy_convergence_time.bucket
(count)
[OpenMetrics V2 and Istio v1.5+] Bucket of delay between config change and all proxies converging.
Shown as second
istio.pilot.proxy_queue_time.bucket
(count)
[OpenMetrics V2 and Istio v1.5+] Bucket of observed values for when proxy is in a push queue before being dequeued
istio.pilot.xds.push.time.bucket
(count)
[OpenMetrics V2 and Istio v1.5+] Bucket of observation of total time Pilot takes a push
istio.mesh.request.duration.milliseconds.bucket
(count)
[OpenMetrics V2 and Istio v1.5+] Bucket of observed values for duration of requests
Shown as millisecond
istio.mesh.response.size.bucket
(count)
[OpenMetrics V2 and Istio v1.5+] Bucket of response sizes
Shown as response
istio.mesh.request.size.bucket
(count)
[OpenMetrics V2 and Istio v1.5+] Bucket of request sizes
Shown as request
istio.mesh.agent.pilot.conflict.outbound_listener.http_over_current_tcp
(gauge)
[OpenMetrics V1 and V2] Number of conflicting wildcard http listeners with current wildcard tcp listener.
istio.mesh.agent.go.memstats.stack_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system for stack allocator
Shown as byte
istio.mesh.agent.conflict.inbound_listener
(gauge)
[OpenMetrics V1 and V2] Number of conflicting inbound listeners.
istio.mesh.agent.go.memstats.sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes obtained from system
Shown as byte
istio.mesh.agent.pilot.xds
(gauge)
[OpenMetrics V1 and V2] Number of endpoints connected to this pilot using XDS.
istio.mesh.agent.go.memstats.alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes allocated and still in use.
Shown as byte
istio.mesh.agent.go.memstats.heap_idle_bytes
(gauge)
[OpenMetrics V1 and V2] Number of idle bytes in the heap.
Shown as byte
istio.mesh.agent.process.resident_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Resident memory size in bytes.
Shown as byte
istio.mesh.agent.conflict.outbound_listener.tcp_over_current_tcp
(gauge)
[OpenMetrics V1 and V2] Number of conflicting tcp listeners with current tcp listener.
istio.mesh.agent.go.memstats.gc_cpu_fraction
(gauge)
[OpenMetrics V1 and V2] CPU taken up by GC.
istio.mesh.agent.go.memstats.heap_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the heap.
Shown as byte
istio.mesh.agent.go.memstats.stack_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by the stack allocator
Shown as byte
istio.mesh.agent.go.memstats.heap_released_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes released to the system in the last gc.
Shown as byte
istio.mesh.agent.go.memstats.mspan_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mspan structures.
Shown as byte
istio.mesh.agent.go.memstats.mallocs.count
(count)
[OpenMetrics V1 and V2] Number of mallocs
Shown as byte
istio.mesh.agent.pilot.endpoint_not_ready
(gauge)
[OpenMetrics V1 and V2] Endpoint found in unready state.
istio.mesh.agent.pilot.no_ip
(gauge)
[OpenMetrics V1 and V2] Pods not found in the endpoint table, possibly invalid.
istio.mesh.agent.num_outgoing_requests.count
(count)
[OpenMetrics V1 and V2] Number of outgoing requests.
istio.mesh.agent.go.memstats.other_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for other system allocations.
Shown as byte
istio.mesh.agent.pilot.xds.config_size_bytes.sum
(count)
[OpenMetrics V1 and V2] Sum of pilot XDS config size.
Shown as byte
istio.mesh.agent.pilot.xds.config_size_bytes.count
(count)
[OpenMetrics V1 and V2] Count of pilot XDS config size samples.
istio.mesh.agent.pilot.xds.config_size_bytes.bucket
(count)
[OpenMetrics V1 and V2] Number of pilot XDS config size.
Shown as byte
istio.mesh.agent.process.open_fds
(gauge)
[OpenMetrics V1 and V2] Number of open file descriptors.
Shown as file
istio.mesh.agent.go.goroutines
(gauge)
[OpenMetrics V1 and V2] Number of goroutines that currently exist.
Shown as thread
istio.mesh.agent.go.threads
(gauge)
[OpenMetrics V1 and V2] Number of OS threads created.
Shown as thread
istio.mesh.agent.go.info
(gauge)
[OpenMetrics V1 and V2] Information about the Go environment.
istio.mesh.agent.go.memstats.frees.count
(count)
[OpenMetrics V1 and V2] Total number of frees.
istio.mesh.agent.go.memstats.mcache_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes in use by mcache structures.
Shown as byte
istio.mesh.agent.process.virtual_memory_bytes
(gauge)
[OpenMetrics V1 and V2] Virtual memory size in bytes.
Shown as byte
istio.mesh.agent.endpoint_no_pod
(gauge)
[OpenMetrics V1 and V2] Endpoints without an associated pod.
istio.mesh.agent.go.gc_duration_seconds.sum
(count)
[OpenMetrics V1 and V2] Sum of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.mesh.agent.go.gc_duration_seconds.count
(count)
[OpenMetrics V1 and V2] Count of the GC invocation durations. This metric is sent as gauge by default in OpenMetrics V1.
Shown as second
istio.mesh.agent.go.gc_duration_seconds.quantile
(gauge)
[OpenMetrics V1 and V2] Quantile of the GC invocation durations.
Shown as second
istio.mesh.agent.process.cpu_seconds.count
(count)
[OpenMetrics V1 and V2] Total user and system CPU time spent in seconds.
Shown as second
istio.mesh.agent.go.memstats.heap_objects
(gauge)
[OpenMetrics V1 and V2] Number of objects in the heap
Shown as object
istio.mesh.agent.pilot.vservice_dup_domain
(gauge)
[OpenMetrics V1 and V2] Virtual services with dup domains.
istio.mesh.agent.process.virtual_memory_max_bytes
(gauge)
[OpenMetrics V1 and V2] Maximum amount of virtual memory available.
Shown as byte
istio.mesh.agent.go.memstats.mcache_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mcache structures obtained from system.
Shown as byte
istio.mesh.agent.scrapes.count
(count)
[OpenMetrics V1 and V2] Number of scrapes.
istio.mesh.agent.pilot.duplicate_envoy_clusters
(gauge)
[OpenMetrics V1 and V2] Duplicate envoy clusters caused by service entries with same hostname.
istio.mesh.agent.go.memstats.buck_hash_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used by the profiling bucket hash table.
Shown as byte
istio.mesh.agent.pilot.xds.push_time.sum
(count)
[OpenMetrics V1 and V2] Sum pilot XDS push time.
istio.mesh.agent.pilot.xds.push_time.count
(count)
[OpenMetrics V1 and V2] Total number of samples of pilot XDS push time.
istio.mesh.agent.pilot.xds.push_time.bucket
(count)
[OpenMetrics V1 and V2] Time of pilot XDS push time.
istio.mesh.agent.wasm_cache_entries
(gauge)
[OpenMetrics V1 and V2] Number of Web Assembly cache entries.
Shown as entry
istio.mesh.agent.pilot.eds_no_instances
(gauge)
[OpenMetrics V1 and V2] Number of clusters without instances.
istio.mesh.agent.go.memstats.heap_alloc_bytes
(gauge)
[OpenMetrics V1 and V2] Bytes allocated to the heap
Shown as byte
istio.mesh.agent.pilot.virt_services
(gauge)
[OpenMetrics V1 and V2] Total virtual services known to pilot.
istio.mesh.agent.go.memstats.next_gc_bytes
(gauge)
[OpenMetrics V1 and V2] Number of heap bytes when next garbage collection will take place
Shown as byte
istio.mesh.agent.startup_duration_seconds
(gauge)
[OpenMetrics V1 and V2] Start up duration time in seconds.
Shown as second
istio.mesh.agent.go.memstats.last_gc_time_seconds
(gauge)
[OpenMetrics V1 and V2] Length of last GC
Shown as second
istio.mesh.agent.pilot.xds.send_time.sum
(count)
[OpenMetrics V1 and V2] Sum of pilot XDS send time.
istio.mesh.agent.pilot.xds.send_time.count
(count)
[OpenMetrics V1 and V2] Count of pilot XDS send time sample.
istio.mesh.agent.pilot.xds.send_time.bucket
(count)
[OpenMetrics V1 and V2] Pilot XDS send time.
istio.mesh.agent.go.memstats.heap_inuse_bytes
(gauge)
[OpenMetrics V1 and V2] Number of Bytes in the heap
Shown as byte
istio.mesh.agent.process.max_fds
(gauge)
[OpenMetrics V1 and V2] Maximum number of open file descriptors.
Shown as file
istio.mesh.agent.go.memstats.gc_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for garbage collection system metadata.
Shown as byte
istio.mesh.agent.pilot.destrule_subsets
(gauge)
[OpenMetrics V1 and V2] Duplicate subsets across destination rules for same host.
istio.mesh.agent.pilot.xds.pushes.count
(count)
[OpenMetrics V1 and V2] Pilot build and send errors for lds, rds, cds and eds.
istio.mesh.agent.process.start_time_seconds
(gauge)
[OpenMetrics V1 and V2] Start time of the process since unix epoch in seconds.
Shown as second
istio.mesh.agent.go.memstats.lookups.count
(count)
[OpenMetrics V1 and V2] Number of lookups
Shown as operation
istio.mesh.agent.outgoing_latency.count
(count)
[OpenMetrics V1 and V2] Total outgoing latency.
istio.mesh.agent.go.memstats.mspan_sys_bytes
(gauge)
[OpenMetrics V1 and V2] Number of bytes used for mspan structures obtained from system.
Shown as byte
istio.mesh.agent.pilot.conflict.outbound_listener.tcp_over_current_http
(gauge)
[OpenMetrics V1 and V2] Number of conflicting wildcard tcp listeners with current wildcard http listener.
istio.galley.source.kube.event.error.count
(count)
[OpenMetrics V2] Number of times a kubernetes source encountered errored while handling an event.
istio.galley.source.kube.dynamic.converter.failure.count
(count)
[OpenMetrics V2] Number of times a dynamnic kubernetes source failed converting a resources
istio.galley.validation.cert.key.update.errors.count
(count)
[OpenMetrics V2] Number of Galley validation webhook certificate updates errors.
istio.galley.validation.http.error.count
(count)
[OpenMetrics V2] Number of resource validation http serve errors.
istio.mcp.clients.count
(count)
[OpenMetrics V2] Number of streams currently connected.
istio.mcp.request.acks.count
(count)
[OpenMetrics V2] Number of request acks received by the source.
istio.mcp.request.nacks.count
(count)
[OpenMetrics V2] Number of request nacks received by the source.
istio.mixer.config.rule.config.errors.count
(count)
[OpenMetrics V2] Number of errors encountered during processing of the rule configuration.
istio.mixer.config.rule.config.match.errors.count
(count)
[OpenMetrics V2] Number of rule conditions that was not parseable.
istio.mixer.config.unsatisfied.action.handlers.count
(count)
[OpenMetrics V2] Number of actions that failed due to handlers being unavailable.
istio.mixer.config.adapter.info.configs.count
(count)
[OpenMetrics V2] Number of known adapters in the current config.
istio.mixer.config.adapter.info.config.errors.count
(count)
[OpenMetrics V2] Number of errors encountered during processing of the adapter info configuration.
istio.mixer.config.handler.validation.errors.count
(count)
[OpenMetrics V2] Number of errors encountered because handler validation returned error.
istio.mixer.config.instance.config.errors.count
(count)
[OpenMetrics V2] Number of errors encountered during processing of the instance configuration.
istio.mixer.handler.handler.build.failures.count
(count)
[OpenMetrics V2] Number of handlers that failed creation during config transition.
istio.galley.istio.networking.virtualservices
(gauge)
istio.galley.istio.networking.destinationrules
(gauge)
istio.galley.istio.networking.gateways
(gauge)
istio.galley.istio.authentication.meshpolicies
(gauge)

Events

The Istio check does not include any events.

Service Checks

istio.prometheus.health
Returns CRITICAL if the check cannot access the metrics endpoint. Returns OK otherwise.
Statuses: ok, critical

istio.openmetrics.health
Returns CRITICAL if the Agent is unable to connect to the OpenMetrics endpoint, otherwise returns OK.
Statuses: ok, critical

istio.pilot.prometheus.health
Returns CRITICAL if the check cannot access the metrics endpoint. Returns OK otherwise.
Statuses: ok, critical

istio.galley.prometheus.health
Returns CRITICAL if the check cannot access the metrics endpoint. Returns OK otherwise.
Statuses: ok, critical

istio.citadel.prometheus.health
Returns CRITICAL if the check cannot access the metrics endpoint. Returns OK otherwise.
Statuses: ok, critical

Troubleshooting

Invalid chunk length error

If you see the following error on the legacy mode of the Istio integration (Istio integration version 3.13.0 or earlier):

  Error: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)",
  InvalidChunkLength(got length b'', 0 bytes read))

You can use the latest mode of the OpenMetrics-based Istio integration to resolve this error.

You must upgrade to at minimum Agent 7.31.0 and Python 3. See the Configuration section to enable OpenMetrics.

Using the generic OpenMetrics integration in an Istio deployment

If Istio proxy sidecar injection is enabled, monitoring other Prometheus metrics using the OpenMetrics integration with the same metrics endpoint as istio_mesh_endpoint can result in high custom metrics usage and duplicated metric collection.

To ensure that your OpenMetrics configuration does not redundantly collect metrics, either:

  1. Use specific metric matching in the metrics configuration option, or
  2. If using the wildcard * value for metrics, consider using the following OpenMetrics integration options to exclude metrics already supported by the Istio and Envoy integrations.

OpenMetrics latest mode configuration with generic metric collection

Be sure to exclude Istio and Envoy metrics from your configuration to avoid high custom metrics billing. Use exclude_metrics if openmetrics_endpoint is enabled.

## Every instance is scheduled independent of the others.
#
instances:
  - openmetrics_endpoint: <OPENMETRICS_ENDPOINT>
    metrics:
    - '.*'
    exclude_metrics:
      - istio_*
      - envoy_*

OpenMetrics legacy mode configuration with generic metric collection

Be sure to exclude Istio and Envoy metrics from your configuration to avoid high custom metrics billing. Use ignore_metrics if prometheus_url is enabled.

instances:
  - prometheus_url: <PROMETHEUS_URL>
    metrics:
      - '*'
    ignore_metrics:
      - istio_*
      - envoy_*

Need help? Contact Datadog support.

Further Reading

Additional helpful documentation, links, and articles: