Milvus

Supported OS Linux Windows Mac OS

Integration version1.2.0

Overview

This check monitors Milvus through the Datadog Agent. It provides insights into your Milvus deployment’s performance by collecting information about the latency and number of executions of individual operations. This integration also allows for monitoring the size and ressource allocation of your deployment.

Setup

Installation

The Milvus check is included in the Datadog Agent package. No additional installation is needed on your server.

Configuration

Host

Follow the instructions below to configure this check for an Agent running on a host. For containerized environments, see the Containerized section.

Edit the milvus.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory to start collecting your Milvus performance data. See the sample milvus.d/conf.yaml for all available configuration options.
Restart the Agent.

Containerized

For containerized environments, see the Autodiscovery Integration Templates for guidance on applying these instructions.

Logs

The Milvus integration can collect logs from the Milvus pods or containers.

Apply this if you want to collect logs from Milvus standalone containers.

Collecting logs is disabled by default in the Datadog Agent. Enable it in your datadog.yaml file:
```
logs_enabled: true
```
Uncomment and edit the logs configuration block in your milvus.d/conf.yaml file. Here’s an example:
```
logs:
  - type: docker
    source: milvus
    service: milvus
```

Apply this if you want to collect logs from a Milvus Kubernetes cluster.

Collecting logs is disabled by default in the Datadog Agent. To enable it, see Kubernetes Log Collection.

Then, set Log Integrations as pod annotations. This can also be configured with a file, a configmap, or a key-value store. For more information, see the configuration section of Kubernetes Log Collection.

Validation

Run the Agent’s status subcommand and look for milvus under the Checks section.

Data Collected

Metrics

milvus.bf_search_cnt.bucket (count)	Histogram bucket for the number of brute-force searches per request
milvus.bf_search_cnt.count (count)	Count aggregation of brute-force searches per request
milvus.bf_search_cnt.sum (count)	Sum aggregation of brute-force searches per request
milvus.bitset_ratio.bucket (count)	Bitset ratio histogram bucket
milvus.bitset_ratio.count (count)	Count of bitset ratio calculations
milvus.bitset_ratio.sum (count)	Sum of bitset ratios
milvus.build_info (gauge)	Build information of milvus
milvus.build_latency.bucket (count)	Index build latency histogram bucket
milvus.build_latency.count (count)	Count of index builds
milvus.build_latency.sum (count)	Sum of index build latencies(s) Shown as second
milvus.cache_hit_cnt.bucket (count)	Histogram bucket for the cache hit count per request
milvus.cache_hit_cnt.count (count)	Count aggregation of cache hit count per request
milvus.cache_hit_cnt.sum (count)	Sum aggregation of cache hit count per request
milvus.cgo.active_future_total (gauge)	Total number of active futures.
milvus.cgo.cgo_duration_seconds.bucket (count)	Histogram bucket of cgo call duration in seconds
milvus.cgo.cgo_duration_seconds.count (count)	Count of cgo calls
milvus.cgo.cgo_duration_seconds.sum (count)	Sum of cgo call duration in seconds. Shown as second
milvus.cgo.cgo_queue_duration_seconds.bucket (count)	Histogram bucket for the duration of cgo calls in queue
milvus.cgo.cgo_queue_duration_seconds.count (count)	Counnt of cgo calls in queue
milvus.cgo.cgo_queue_duration_seconds.sum (count)	Sum of the duration of cgo calls in queue Shown as second
milvus.cgo.running_cgo_call_total (gauge)	Total number of running cgo calls.
milvus.datacoord.channel_checkpoint_unix_seconds (gauge)	Channel checkpoint timestamp in Unix seconds
milvus.datacoord.collection_num (gauge)	Number of collections
milvus.datacoord.consume_datanode_tt_lag_ms (gauge)	Now time minus time travel per physical channel Shown as millisecond
milvus.datacoord.datanode_num (gauge)	Number of data nodes
milvus.datacoord.import_tasks (gauge)	The import tasks grouping by type and state
milvus.datacoord.index.node_num (gauge)	Number of IndexNodes managed by IndexCoord
milvus.datacoord.index.req.count (count)	Number of building index requests
milvus.datacoord.index.task (gauge)	Number of index tasks of each type
milvus.datacoord.segment_num (gauge)	Number of segments
milvus.datacoord.stored.binlog_size (gauge)	Binlog size of healthy segments Shown as byte
milvus.datacoord.stored.index_files_size (gauge)	Index files size of the segments Shown as byte
milvus.datacoord.stored.rows_num (gauge)	Number of stored rows of healthy segment
milvus.datacoord.task_execute_max_latency.bucket (count)	Histogram bucket for the latency of task execute operation
milvus.datacoord.task_execute_max_latency.count (count)	Count of task execute operations
milvus.datacoord.task_execute_max_latency.sum (count)	Sum of the latency of task execute operations Shown as millisecond
milvus.datacoord.watched_dml_chanel_num (gauge)	The num of data manipulation language channel watched by datanode
milvus.datanode.autoflush_buffer_op.count (count)	Count of auto flush buffer operations
milvus.datanode.consume.bytes.count (count)	Count of consumed bytes
milvus.datanode.consume.msg.count (count)	Count of consumed msg
milvus.datanode.consume.tt_lag_ms (gauge)	Now time minus time travel per physical channel Shown as millisecond
milvus.datanode.encode_buffer_latency.bucket (count)	Histogram bucket for the latency of encode buffer data
milvus.datanode.encode_buffer_latency.count (count)	Count of encode buffer data operations
milvus.datanode.encode_buffer_latency.sum (count)	Sum of the latency of encode buffer data operations Shown as millisecond
milvus.datanode.flowgraph_num (gauge)	Number of flowgraphs
milvus.datanode.flush.buffer_op.count (count)	Count of flush buffer operations
milvus.datanode.flush.req.count (count)	Count of flush request
milvus.datanode.flushed_data.rows.count (count)	Num of rows flushed to storage
milvus.datanode.flushed_data.size.count (count)	Byte size of data flushed to storage Shown as byte
milvus.datanode.msg.dispatcher_tt_lag_ms (gauge)	time.Now() sub dispatcher's current consume time Shown as millisecond
milvus.datanode.msg.rows.count (count)	Count of rows consumed from msgStream
milvus.datanode.save_latency.bucket (count)	Histogram bucket for the latency of saving flush data to storage
milvus.datanode.save_latency.count (count)	Count of saving flush data to storage events
milvus.datanode.save_latency.sum (count)	Sum of the latency of saving flush data to storage events Shown as millisecond
milvus.diskann.range_search_iters.bucket (count)	Histogram bucket for DISKANN range search iterations
milvus.diskann.range_search_iters.count (count)	Count of DISKANN range search events
milvus.diskann.range_search_iters.sum (count)	Sum of DISKANN range search iterations
milvus.diskann.search_hops.bucket (count)	Histogram for DISKANN search hops
milvus.diskann.search_hops.count (count)	Count of DISKANN search hop events
milvus.diskann.search_hops.sum (count)	Sum of DISKANN search hops
milvus.diskann_bitset_ratio.bucket (count)	Histogram bucket for the DISKANN bitset ratio for search and range search
milvus.diskann_bitset_ratio.count (count)	Count of the DISKANN bitset ratio operations for search and range search
milvus.diskann_bitset_ratio.sum (count)	Sum of the DISKANN bitset ratio for search and range search
milvus.exec_latency.bucket (count)	Histogram bucket for the execute latency per request
milvus.exec_latency.count (count)	Count aggregation of the execute latency per request
milvus.exec_latency.sum (count)	Sum aggregation of the execute latency per request
milvus.filter.connectivity_ratio.bucket (count)	Histogram bucket for the average connectivity ratio set under filtering per request
milvus.filter.connectivity_ratio.count (count)	Count of the connectivity ratios set under filtering per request
milvus.filter.connectivity_ratio.sum (count)	Sum of the average connectivity ratios set under filtering per request
milvus.filter.mv.activated_fields_cnt.bucket (count)	Histogram bucket of the average materialized view activated fields per request
milvus.filter.mv.activated_fields_cnt.count (count)	Count of the requests activating materialized view fields
milvus.filter.mv.activated_fields_cnt.sum (count)	Sum of the average materialized view activated fields per request
milvus.filter.mv.change_base_cnt.bucket (count)	Histogram bucket for the materialized view change base count per request
milvus.filter.mv.change_base_cnt.count (count)	Count of requests triggering materialized view change base events
milvus.filter.mv.change_base_cnt.sum (count)	Sum of materialized view change base counts per request
milvus.filter.mv.only_cnt.bucket (count)	Histogram bucket for the materialized view only count per request
milvus.filter.mv.only_cnt.count (count)	Count of requests that trigger materialized view only events
milvus.filter.mv.only_cnt.sum (count)	Sum of materialized view only counts
milvus.filter.mv.supplement_ep_bool_cnt.bucket (count)	Histogram bucket for materialized view supplement entry point from bitset boolean count per request
milvus.filter.mv.supplement_ep_bool_cnt.count (count)	Count of requests triggering a materialized view supplement entry point from bitset boolean event
milvus.filter.mv.supplement_ep_bool_cnt.sum (count)	Sum of materialized view supplement entry point from bitset boolean events
milvus.flushed_segment_file_num.bucket (count)	Histogram bucket for the number of files for flushed segment
milvus.flushed_segment_file_num.count (count)	Count of flushed segment events
milvus.flushed_segment_file_num.sum (count)	Sum of the number of files for flushed segments
milvus.go.gc_duration_seconds.count (count)	A summary of the pause duration of garbage collection cycles.
milvus.go.gc_duration_seconds.quantile (gauge)	A summary of the pause duration of garbage collection cycles.
milvus.go.gc_duration_seconds.sum (count)	A summary of the pause duration of garbage collection cycles.
milvus.go.goroutines (gauge)	Number of goroutines that currently exist.
milvus.go.info (gauge)	Information about the Go environment.
milvus.go.memstats.alloc_bytes (gauge)	Number of bytes allocated and still in use.
milvus.go.memstats.alloc_bytes.count (count)	Total number of bytes allocated, even if freed.
milvus.go.memstats.buck_hash_sys_bytes (gauge)	Number of bytes used by the profiling bucket hash table.
milvus.go.memstats.frees.count (count)	Total number of frees.
milvus.go.memstats.gc_sys_bytes (gauge)	Number of bytes used for garbage collection system metadata.
milvus.go.memstats.heap.alloc_bytes (gauge)	Number of heap bytes allocated and still in use.
milvus.go.memstats.heap.idle_bytes (gauge)	Number of heap bytes waiting to be used.
milvus.go.memstats.heap.inuse_bytes (gauge)	Number of heap bytes that are in use.
milvus.go.memstats.heap.objects (gauge)	Number of allocated objects.
milvus.go.memstats.heap.released_bytes (gauge)	Number of heap bytes released to OS.
milvus.go.memstats.heap.sys_bytes (gauge)	Number of heap bytes obtained from system.
milvus.go.memstats.last_gc_time_seconds (gauge)	Number of seconds since 1970 of last garbage collection.
milvus.go.memstats.lookups.count (count)	Total number of pointer lookups.
milvus.go.memstats.mallocs.count (count)	Total number of mallocs.
milvus.go.memstats.mcache.inuse_bytes (gauge)	Number of bytes in use by mcache structures.
milvus.go.memstats.mcache.sys_bytes (gauge)	Number of bytes used for mcache structures obtained from system.
milvus.go.memstats.mspan.inuse_bytes (gauge)	Number of bytes in use by mspan structures.
milvus.go.memstats.mspan.sys_bytes (gauge)	Number of bytes used for mspan structures obtained from system.
milvus.go.memstats.next_gc_bytes (gauge)	Number of heap bytes when next garbage collection will take place.
milvus.go.memstats.other_sys_bytes (gauge)	Number of bytes used for other system allocations.
milvus.go.memstats.stack.inuse_bytes (gauge)	Number of bytes in use by the stack allocator.
milvus.go.memstats.stack.sys_bytes (gauge)	Number of bytes obtained from system for stack allocator.
milvus.go.memstats.sys_bytes (gauge)	Number of bytes obtained from system.
milvus.go.threads (gauge)	Number of OS threads created.
milvus.graph_search_cnt.bucket (count)	Histogram bucket for the number of graph search per request
milvus.graph_search_cnt.count (count)	Count of requests that trigger a graph search
milvus.graph_search_cnt.sum (count)	Sum of graph searches
milvus.hnsw.bitset_ratio.bucket (count)	Histogram bucket of the HNSW bitset ratio for search and range search
milvus.hnsw.bitset_ratio.count (count)	Count aggregation of the HNSW bitset ratio for search and range search
milvus.hnsw.bitset_ratio.sum (count)	Sum aggregation of the HNSW bitset ratio for search and range search
milvus.hnsw.search_hops.bucket (count)	Histogram bucket of the HNSW search hops in layer 0
milvus.hnsw.search_hops.count (count)	Count aggregation of the HNSW search hops in layer 0
milvus.hnsw.search_hops.sum (count)	Sum aggregation of the HNSW search hops in layer 0
milvus.indexnode.build_index_latency.bucket (count)	Histogram bucket of the latency of build index for segment
milvus.indexnode.build_index_latency.count (count)	Count of build index events
milvus.indexnode.build_index_latency.sum (count)	Sum of the latency of build index events Shown as millisecond
milvus.indexnode.encode_index_latency.bucket (count)	Histogram bucket of the latency of encoding the index file
milvus.indexnode.encode_index_latency.count (count)	Count of the index file encoding events
milvus.indexnode.encode_index_latency.sum (count)	Sum of the latency of index file encoding events Shown as millisecond
milvus.indexnode.index.task.count (count)	Number of tasks that index node received
milvus.indexnode.index.task_latency_in_queue.bucket (count)	Histogram bucket for the latency of index task in queue
milvus.indexnode.index.task_latency_in_queue.count (count)	Count of index task in queue
milvus.indexnode.index.task_latency_in_queue.sum (count)	Sum of the latency of index task in queue Shown as millisecond
milvus.indexnode.knowhere_build_index_latency.bucket (count)	Histogram bucket for the latency of building the index by knowhere
milvus.indexnode.knowhere_build_index_latency.count (count)	Count of index builds by knowhere
milvus.indexnode.knowhere_build_index_latency.sum (count)	Sum of the latency of building the index by knowhere Shown as millisecond
milvus.indexnode.save_index_latency.bucket (count)	Histogram bucket for the latency of saving the index file
milvus.indexnode.save_index_latency.count (count)	Count of index file save envents
milvus.indexnode.save_index_latency.sum (count)	Sum of the latency of saving the index file Shown as millisecond
milvus.internal.core_search_latency.bucket (count)	Histogram bucket for the [cpp]latency(us) of search on segment
milvus.internal.core_search_latency.count (count)	Count aggregation of [cpp]latency(us) of search on segment
milvus.internal.core_search_latency.sum (count)	Sum aggregation of [cpp]latency(us) of search on segment Shown as microsecond
milvus.internal.mmap.allocated_space_bytes.bucket (count)	Histogram bucket for the [cpp]mmap allocated space stats
milvus.internal.mmap.allocated_space_bytes.count (count)	Count aggregation of [cpp]mmap allocated space stats
milvus.internal.mmap.allocated_space_bytes.sum (count)	Sum aggregation of [cpp]mmap allocated space stats
milvus.internal.mmap.in_used_space_bytes (gauge)	[cpp]mmap in used space stats
milvus.internal.storage.kv_size.bucket (count)	Histogram bucket for the [cpp]key-value size stats
milvus.internal.storage.kv_size.count (count)	Count aggregation of [cpp]key-value size stats
milvus.internal.storage.kv_size.sum (count)	Sum aggregation of [cpp]key-value size stats
milvus.internal.storage.load_duration.bucket (count)	Histogram bucket for the [cpp]durations of load segment
milvus.internal.storage.load_duration.count (count)	Count aggregation of [cpp]durations of load segment
milvus.internal.storage.load_duration.sum (count)	Sum aggregation of [cpp]durations of load segment
milvus.internal.storage.op.count (count)	[cpp]count of persistent data operation
milvus.internal.storage.request_latency.bucket (count)	Histogram bucket for the [cpp]request latency(ms) on the client side
milvus.internal.storage.request_latency.count (count)	Count aggregation of [cpp]request latency(ms) on the client side
milvus.internal.storage.request_latency.sum (count)	Sum aggregation of [cpp]request latency(ms) on the client side Shown as millisecond
milvus.io_cnt.bucket (count)	Histogram bucket for the IO count per request
milvus.io_cnt.count (count)	Count of requests triggering IO operations
milvus.io_cnt.sum (count)	Sum of IO operation counts
milvus.ivf_search_cnt.bucket (count)	Histogram bucket for the number of inverted file search per request
milvus.ivf_search_cnt.count (count)	Count of requests trigerring an inverted file search
milvus.ivf_search_cnt.sum (count)	Sum of inverted file searches
milvus.load_latency.bucket (count)	Histogram bucket for the index load latency (ms)
milvus.load_latency.count (count)	Count of index load events
milvus.load_latency.sum (count)	Sum of index load latency (ms) Shown as millisecond
milvus.meta.kv_size.bucket (count)	Histogram for the key-value size stats
milvus.meta.kv_size.count (count)	Count aggregation of the key-value size stats
milvus.meta.kv_size.sum (count)	Sum aggregation of the key-value size stats
milvus.meta.op.count (count)	Count of meta operation
milvus.meta.request_latency.bucket (count)	Histogram bucket for the request latency on the client side
milvus.meta.request_latency.count (count)	Count aggregation of the request latency on the client side
milvus.meta.request_latency.sum (count)	Sum aggregation of the request latency on the client side Shown as millisecond
milvus.msg_queue_consumer_num (gauge)	Number of consumers
milvus.msgstream.op.count (count)	Count of stream message operation
milvus.msgstream.request_latency.bucket (count)	Histogram bucket for the request latency on the client side
milvus.msgstream.request_latency.count (count)	Count aggregation of the request latency on the client side
milvus.msgstream.request_latency.sum (count)	Sum aggregation of the request latency on the client side Shown as millisecond
milvus.num_node (gauge)	Number of nodes and coordinates
milvus.process.cpu_seconds.count (count)	Total user and system CPU time spent in seconds.
milvus.process.max_fds (gauge)	Maximum number of open file descriptors.
milvus.process.open_fds (gauge)	Number of open file descriptors.
milvus.process.resident_memory_bytes (gauge)	Resident memory size in bytes.
milvus.process.start_time_seconds (gauge)	Start time of the process since Unix epoch in seconds.
milvus.process.virtual_memory.bytes (gauge)	Virtual memory size in bytes.
milvus.process.virtual_memory.max_bytes (gauge)	Maximum amount of virtual memory available in bytes.
milvus.proxy.apply.pk_latency.bucket (count)	Histogram bucket for the latency of 'apply primary key' events
milvus.proxy.apply.pk_latency.count (count)	Count of 'apply primary key' events
milvus.proxy.apply.pk_latency.sum (count)	Sum of latencies for 'apply primary key' events Shown as millisecond
milvus.proxy.apply.timestamp_latency.bucket (count)	Histogram bucket for the latency of proxy 'apply timestamp' events
milvus.proxy.apply.timestamp_latency.count (count)	Count of proxy 'apply timestamp' events
milvus.proxy.apply.timestamp_latency.sum (count)	Sum of latencies of proxy 'apply timestamp' events Shown as millisecond
milvus.proxy.assign_segmentID_latency.bucket (count)	Histogram bucket for the latency of proxy 'get segmentID from dataCoord' events
milvus.proxy.assign_segmentID_latency.count (count)	Count of proxy 'get segmentID from dataCoord' events
milvus.proxy.assign_segmentID_latency.sum (count)	Sum of latencies for proxy 'get segmentID from dataCoord' events Shown as millisecond
milvus.proxy.cache.hit.count (count)	Count of cache hits/miss
milvus.proxy.cache.update_latency.bucket (count)	Histogram bucket for the latency of proxy 'update cache when cache miss' events
milvus.proxy.cache.update_latency.count (count)	Count of proxy 'update cache when cache miss' events
milvus.proxy.cache.update_latency.sum (count)	Sum of latencies of proxy 'update cache when cache miss' Shown as millisecond
milvus.proxy.delete_vectors.count (count)	Counter of vectors successfully deleted
milvus.proxy.msgstream_obj_num (gauge)	Number of MsgStream objects per physical channel
milvus.proxy.mutation_send_latency.bucket (count)	Histogram bucket for the latency of proxy 'send insert request to MsgStream' events
milvus.proxy.mutation_send_latency.count (count)	Count of proxy 'send insert request to MsgStream' events
milvus.proxy.mutation_send_latency.sum (count)	Sum of latencies of proxy 'send insert request to MsgStream' events Shown as millisecond
milvus.proxy.rate_limit_req.count (count)	Count of operation executed
milvus.proxy.report_value.count (count)	Report value about the request
milvus.proxy.req.count (count)	Count of operation executed
milvus.proxy.req.in_queue_latency.bucket (count)	Histogram bucket for the time in queue of request
milvus.proxy.req.in_queue_latency.count (count)	Count of request in queue
milvus.proxy.req.in_queue_latency.sum (count)	Sum of the time requests spend in queue Shown as millisecond
milvus.proxy.req.latency.bucket (count)	Histogram bucket for the latency of each request
milvus.proxy.req.latency.count (count)	Count of request latency events
milvus.proxy.req.latency.sum (count)	Sum of latencies of requests Shown as millisecond
milvus.proxy.send_bytes.count (count)	Count of bytes sent back to sdk
milvus.proxy.sq.decode_result_latency.bucket (count)	Histogram bucket for the latency of proxy 'decode the search result' events
milvus.proxy.sq.decode_result_latency.count (count)	Count of proxy 'decode the search result' events
milvus.proxy.sq.decode_result_latency.sum (count)	Sum of latencies for proxy 'decode the search result' events Shown as millisecond
milvus.proxy.sq.reduce_result_latency.bucket (count)	Histogram bucket for the latency of proxy 'reduces search result' events
milvus.proxy.sq.reduce_result_latency.count (count)	Count of proxy 'reduces search result' events
milvus.proxy.sq.reduce_result_latency.sum (count)	Sum of latencies for proxy 'reduces search result' events Shown as millisecond
milvus.proxy.sq.wait_result_latency.bucket (count)	Histogram bucket for the latency of proxy 'waits for the result' events
milvus.proxy.sq.wait_result_latency.count (count)	Count of proxy 'waits for the result' events
milvus.proxy.sq.wait_result_latency.sum (count)	Sum of latencies for proxy 'waits for the result' events Shown as millisecond
milvus.proxy.sync_segment_request_length.bucket (count)	Histogram bucket for the length of SegmentIDRequests when assigning segments for insert
milvus.proxy.sync_segment_request_length.count (count)	Count of assigning segments for insert events
milvus.proxy.sync_segment_request_length.sum (count)	Sum of the length of SegmentIDRequests when assigning segments for insert
milvus.proxy.tt_lag_ms (gauge)	Now time minus time travel per physical channel Shown as millisecond
milvus.quant.compute_cnt.bucket (count)	Histogram bucket for the quant compute count per request
milvus.quant.compute_cnt.count (count)	Count of requests triggering quant compute events
milvus.quant.compute_cnt.sum (count)	Sum of quant compute event counts
milvus.querycoord.collection_num (gauge)	Number of collections
milvus.querycoord.current_target_checkpoint_unix_seconds (gauge)	Current target checkpoint timestamp in Unix seconds
milvus.querycoord.load.latency.bucket (count)	Histogram bucket Latency of load the entire collection
milvus.querycoord.load.latency.count (count)	Latency of load the entire collection
milvus.querycoord.load.latency.sum (count)	Latency of load the entire collection Shown as millisecond
milvus.querycoord.load.req.count (count)	Count of load request
milvus.querycoord.partition_num (gauge)	Number of partitions
milvus.querycoord.querynode_num (gauge)	Number of QueryNodes managered by QueryCoord
milvus.querycoord.release.latency.bucket (count)	Histogram bucket for the latency of release request
milvus.querycoord.release.latency.count (count)	Count of release request events
milvus.querycoord.release.latency.sum (count)	Sum of the latencies of release request events Shown as millisecond
milvus.querycoord.release.req.count (count)	Count of release request
milvus.querycoord_task_num (gauge)	the number of tasks in QueryCoord's scheduler
milvus.querynode.apply_bf_latency.bucket (count)	Histogram bucket for the apply brute-force cost in ms
milvus.querynode.apply_bf_latency.count (count)	Count of apply brute-force events
milvus.querynode.apply_bf_latency.sum (count)	Sum of the apply brute-force cost in ms Shown as millisecond
milvus.querynode.collection_num (gauge)	Number of collections loaded
milvus.querynode.consume.bytes_counter.count (count)	Count of consumed bytes
milvus.querynode.consume.msg.count (count)	Count of consumed msg
milvus.querynode.consume.tt_lag_ms (gauge)	Now time minus time travel per physical channel Shown as millisecond
milvus.querynode.disk.cache.evict.bytes.count (count)	Number of bytes evicted from disk cache Shown as byte
milvus.querynode.disk.cache.evict.count (count)	Number of segments evicted from disk cache
milvus.querynode.disk.cache.evict.duration.count (count)	Total time cost of evicting segments from disk cache
milvus.querynode.disk.cache.evict.global_duration.bucket (count)	Histogram bucket for the global duration of evicting segments from disk cache
milvus.querynode.disk.cache.evict.global_duration.count (count)	Count of evicting segments from disk cache events
milvus.querynode.disk.cache.evict.global_duration.sum (count)	Sum of the global duration of evicting segments from disk cache Shown as millisecond
milvus.querynode.disk.cache.load.bytes.count (count)	Number of bytes loaded from disk cache Shown as byte
milvus.querynode.disk.cache.load.count (count)	Number of segments loaded from disk cache
milvus.querynode.disk.cache.load.duration.count (count)	Total time cost of loading segments from disk cache
milvus.querynode.disk.cache.load.global_duration.bucket (count)	Histogram bucket for the global duration of loading segments from disk cache
milvus.querynode.disk.cache.load.global_duration.count (count)	Count of loading segments from disk cache events
milvus.querynode.disk.cache.load.global_duration.sum (count)	Sum of the global duration of loading segments from disk cache Shown as millisecond
milvus.querynode.disk.used_size (gauge)	Disk used size(MB) Shown as mebibyte
milvus.querynode.dml_vchannel_num (gauge)	Number of data manipulation language channels watched
milvus.querynode.entity.num (gauge)	Number of entities which can be searched/queried, clustered by collection, partition and state
milvus.querynode.entity.size (gauge)	Entities' memory size, clustered by collection and state Shown as byte
milvus.querynode.execute_bytes_counter.count (count)	Count of execute bytes
milvus.querynode.flowgraph_num (gauge)	Number of flowgraphs
milvus.querynode.forward_delete_latency.bucket (count)	Histogram bucket for the forward delete cost in ms
milvus.querynode.forward_delete_latency.count (count)	Count of forward delete events
milvus.querynode.forward_delete_latency.sum (count)	Sum of the forward delete cost in ms Shown as millisecond
milvus.querynode.load.index_latency.bucket (count)	Histogram bucket for the latency of load per segment's index, in milliseconds
milvus.querynode.load.index_latency.count (count)	Count of load per segment's index events
milvus.querynode.load.index_latency.sum (count)	Sum of the latencies of load per segment's index events, in milliseconds Shown as millisecond
milvus.querynode.load.segment.concurrency (gauge)	Number of concurrent loading segments in QueryNode
milvus.querynode.load.segment.latency.bucket (count)	Histogram bucket for the latency of load per segment
milvus.querynode.load.segment.latency.count (count)	Count of load per segment events
milvus.querynode.load.segment.latency.sum (count)	Sum of the latencies of load per segment events Shown as millisecond
milvus.querynode.msg_dispatcher_tt_lag_ms (gauge)	time.Now() sub dispatcher's current consume time Shown as millisecond
milvus.querynode.partition_num (gauge)	Number of partitions loaded
milvus.querynode.process_insert_or_delete_latency.bucket (count)	Histogram bucket for the process insert or delete cost in ms
milvus.querynode.process_insert_or_delete_latency.count (count)	Count of process insert or delete events
milvus.querynode.process_insert_or_delete_latency.sum (count)	Sum of process insert or delete cost in ms Shown as millisecond
milvus.querynode.read_task.concurrency (gauge)	Number of concurrent executing read tasks in QueryNode
milvus.querynode.read_task.ready_len (gauge)	Number of ready read tasks in readyQueue
milvus.querynode.read_task.unsolved_len (gauge)	Number of unsolved read tasks in unsolvedQueue
milvus.querynode.search.group.nq.bucket (count)	Histogram bucket for the number of queries of each grouped search task
milvus.querynode.search.group.nq.count (count)	Count of grouped search tasks
milvus.querynode.search.group.nq.sum (count)	Sum of the number of queries of grouped search tasks
milvus.querynode.search.group.size.bucket (count)	Histogram bucket for the number of tasks of each grouped search task
milvus.querynode.search.group.size.count (count)	Count of grouped search tasks
milvus.querynode.search.group.size.sum (count)	Sum of the number of tasks of grouped search tasks
milvus.querynode.search.group.topk.bucket (count)	Histogram bucket for the topK of each grouped search task
milvus.querynode.search.group.topk.count (count)	Count of grouped search tasks
milvus.querynode.search.group.topk.sum (count)	Sum of the topK of grouped search tasks
milvus.querynode.search.nq.bucket (count)	Histogram bucket for the number of queries of each search task
milvus.querynode.search.nq.count (count)	Count of search tasks
milvus.querynode.search.nq.sum (count)	Sum of the number of queries of search tasks
milvus.querynode.search.topk.bucket (count)	Histogram bucket for the top of each search task
milvus.querynode.search.topk.count (count)	Count of search tasks
milvus.querynode.search.topk.sum (count)	Sum of the top of search tasks
milvus.querynode.segment.access.count (count)	Number of segments accessed
milvus.querynode.segment.access.duration.count (count)	Total time cost of accessing segments
milvus.querynode.segment.access.global_duration.bucket (count)	Histogram bucket for the global time cost of accessing segments
milvus.querynode.segment.access.global_duration.count (count)	Count of accessing segments events
milvus.querynode.segment.access.global_duration.sum (count)	Sum of the global time cost of accessing segments Shown as millisecond
milvus.querynode.segment.access.wait_cache.count (count)	Number of segments waiting for loading access
milvus.querynode.segment.access.wait_cache.duration.count (count)	Total time cost of waiting for loading access
milvus.querynode.segment.access.wait_cache.global_duration.bucket (count)	Histogram bucket for the global time cost of waiting for loading access
milvus.querynode.segment.access.wait_cache.global_duration.count (count)	Count of waiting for loading access events
milvus.querynode.segment.access.wait_cache.global_duration.sum (count)	Sum of the global time cost of waiting for loading access events Shown as millisecond
milvus.querynode.segment.latency_per_vector.bucket (count)	Histogram bucket for one vector's search latency per segment
milvus.querynode.segment.latency_per_vector.count (count)	Count aggregation of one vector's search latency per segment
milvus.querynode.segment.latency_per_vector.sum (count)	Sum aggregation of one vector's search latency per segment Shown as millisecond
milvus.querynode.segment.num (gauge)	Number of segments loaded, clustered by its collection, partition, state and # of indexed fields
milvus.querynode.sq.core_latency.bucket (count)	Histogram bucket for the latency of searches or queries in segcore
milvus.querynode.sq.core_latency.count (count)	Count of search or query events in segcore
milvus.querynode.sq.core_latency.sum (count)	Sum of latencies of searches or queries in segcore Shown as millisecond
milvus.querynode.sq.queue.latency.bucket (count)	Histogram bucket for the latency of searches or queries in queue
milvus.querynode.sq.queue.latency.count (count)	Count of search or query events
milvus.querynode.sq.queue.latency.sum (count)	Sum of latencies of searches or queries in queue Shown as millisecond
milvus.querynode.sq.queue.user_latency.bucket (count)	Histgram bucket for the latency per user of search or query in queue
milvus.querynode.sq.queue.user_latency.count (count)	Count of search or query in queue
milvus.querynode.sq.queue.user_latency.sum (count)	Sum of latencies per user of search or query in queue Shown as millisecond
milvus.querynode.sq.reduce_latency.bucket (count)	Histogram bucket for the latencies of reduce search or query result
milvus.querynode.sq.reduce_latency.count (count)	Count of reduce search or query result events
milvus.querynode.sq.reduce_latency.sum (count)	Sum of latencies of reduce search or query result Shown as millisecond
milvus.querynode.sq.req.count (count)	count of search / query request
milvus.querynode.sq.req.latency.bucket (count)	Histogram bucket for the latency of search or query requests
milvus.querynode.sq.req.latency.count (count)	Count of search or query requests
milvus.querynode.sq.req.latency.sum (count)	Sum of latencies of search or query requests Shown as millisecond
milvus.querynode.sq.segment_latency.bucket (count)	Histogram bucket for the latency of search or query per segment
milvus.querynode.sq.segment_latency.count (count)	Count of search or query per segment
milvus.querynode.sq.segment_latency.sum (count)	Sum of latencies of search or query per segment Shown as millisecond
milvus.querynode.sq.wait_tsafe_latency.bucket (count)	Histogram bucket for the time that search or query waits for tsafe
milvus.querynode.sq.wait_tsafe_latency.count (count)	Count of occurrences of search or query waiting for tsafe
milvus.querynode.sq.wait_tsafe_latency.sum (count)	Sum of the time that search or query wait for tsafe Shown as millisecond
milvus.querynode.wait_processing_msg (gauge)	Count of wait processing msg
milvus.querynode.watch_dml_channel_latency.bucket (count)	Histogram bucket for the latencies of watch data manipulation language channel
milvus.querynode.watch_dml_channel_latency.count (count)	Count of watch data manipulation language channel events
milvus.querynode.watch_dml_channel_latency.sum (count)	Sum of the latencies of watch data manipulation language channel events Shown as millisecond
milvus.queue.latency.bucket (count)	Histogram bucket for the queue latency per request
milvus.queue.latency.count (count)	Count of request with queue latency
milvus.queue.latency.sum (count)	Sum of queue latencies
milvus.range_search_latency.bucket (count)	Histogram bucket for range search latency (ms)
milvus.range_search_latency.count (count)	Count of range search operations
milvus.range_search_latency.sum (count)	Sum of range search latencies (ms) Shown as millisecond
milvus.raw_compute_cnt.bucket (count)	Histogram bucket for the raw compute count per request
milvus.raw_compute_cnt.count (count)	Count of request that trigger raw compute operations
milvus.raw_compute_cnt.sum (count)	Sum of raw compute operations
milvus.re_search_cnt.bucket (count)	Histogram bucket for the number of fallback search per request
milvus.re_search_cnt.count (count)	Count of requests triggering fallback search operations
milvus.re_search_cnt.sum (count)	Sum of fallback searches
milvus.rootcoord.collection_num (gauge)	Number of collections
milvus.rootcoord.credential_num (gauge)	Number of credentials
milvus.rootcoord.ddl_req.count (count)	Count of DDL operations
milvus.rootcoord.ddl_req.latency.bucket (count)	Histogram bucket for the latency of each DDL operation
milvus.rootcoord.ddl_req.latency.count (count)	Count of DDL operations
milvus.rootcoord.ddl_req.latency.sum (count)	Sum of the latencies of DDL operations Shown as millisecond
milvus.rootcoord.ddl_req.latency_in_queue.bucket (count)	Histogram bucket for the latencies of each DDL operation in queue
milvus.rootcoord.ddl_req.latency_in_queue.count (count)	Count of DDL operations in queue
milvus.rootcoord.ddl_req.latency_in_queue.sum (count)	Sum of the latencies of DDL operations in queue Shown as millisecond
milvus.rootcoord.dml_channel_num (gauge)	Number of DML channels
milvus.rootcoord.entity_num (gauge)	Number of entities, clustered by collection and their status(loaded/total)
milvus.rootcoord.force_deny_writing_counter.count (count)	The number of times milvus turns into force-deny-writing states
milvus.rootcoord.id_alloc.count (count)	Count of ID allocated
milvus.rootcoord.indexed_entity_num (gauge)	Indexed number of entities, clustered by collection, index name and whether it's a vector index
milvus.rootcoord.msgstream_obj_num (gauge)	Number of message streams
milvus.rootcoord.num_of_roles (gauge)	The number of roles
milvus.rootcoord.partition_num (gauge)	Number of partitions
milvus.rootcoord.produce_tt_lag_ms (gauge)	Now time minus time travel per physical channel Shown as millisecond
milvus.rootcoord.proxy_num (gauge)	Number of proxy nodes managered by rootcoord
milvus.rootcoord.qn_mem_high_water_level (gauge)	Querynode memory high water level
milvus.rootcoord.sync_timetick_latency.bucket (count)	Histogram bucket for the latencies of synchronizing timetick message
milvus.rootcoord.sync_timetick_latency.count (count)	Count of synchronizing timetick message events
milvus.rootcoord.sync_timetick_latency.sum (count)	Sum of latencies of synchronizing timetick message events Shown as millisecond
milvus.rootcoord.timestamp (gauge)	Latest timestamp allocated in memory
milvus.rootcoord.timestamp_saved (gauge)	Timestamp saved in meta storage
milvus.runtime_info (gauge)	Runtime information of Milvus
milvus.search.latency.bucket (count)	Histogram bucket for search latencies (ms)
milvus.search.latency.count (count)	Count of search events
milvus.search.latency.sum (count)	Sum of search latencies (ms) Shown as millisecond
milvus.search.topk.bucket (count)	Histogram bucket for search topk
milvus.search.topk.count (count)	Count aggregation of search topk
milvus.search.topk.sum (count)	Sum aggregation of search topk
milvus.storage.kv_size (gauge)	Key-value size stats
milvus.storage.op.count (count)	Count of persistent data operation
milvus.storage.request_latency (gauge)	Request latency on the client side

Events

The Milvus integration does not include any events.

Service Checks

The Milvus integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.