--- title: Temporal description: Monitor the health and performance of Temporal Cluster. breadcrumbs: Docs > Integrations > Temporal --- # Temporal Supported OS Integration version4.5.0 ## Overview{% #overview %} This check monitors [Temporal](https://temporal.io/) through the Datadog Agent. **Note**: This check can only be installed if you are self-hosting Temporal. **To monitor your Temporal Cloud instance**, follow the [Datadog Temporal Cloud integration documentation](https://docs.datadoghq.com/integrations/temporal_cloud/). **Minimum Agent version:** 7.45.0 ## Setup{% #setup %} Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates](https://docs.datadoghq.com/agent/kubernetes/integrations/) for guidance on applying these instructions. ### Installation{% #installation %} The Temporal check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package. No additional installation is needed on your server. ### Configuration{% #configuration %} {% tab title="Host" %} #### Host{% #host %} ##### Metric collection{% #metric-collection %} 1. Configure your Temporal services to expose metrics via a `prometheus` endpoint by following the [official Temporal documentation](https://docs.temporal.io/references/configuration#prometheus). 1. Edit the `temporal.d/conf.yaml` file located in the `conf.d/` folder at the root of your Agent's configuration directory to start collecting your Temporal performance data. Configure the `openmetrics_endpoint` option to match the `listenAddress` and `handlerPath` options from your Temporal server configuration. ```yaml init_config: instances: - openmetrics_endpoint: / ``` Note that when Temporal services in a cluster are deployed independently, every service exposes its own metrics. As a result, you need to configure the `prometheus` endpoint for every service that you want to monitor and define a separate `instance` on the integration's configuration for each of them. ##### Log collection{% #log-collection %} 1. Collecting logs is disabled by default in the Datadog Agent. Enable it in your `datadog.yaml` file: ```yaml logs_enabled: true ``` 1. Configure your Temporal Cluster to output logs to a file by following the [official documentation](https://docs.temporal.io/references/configuration#log). 1. Uncomment and edit the logs configuration block in your `temporal.d/conf.yaml` file, and set the `path` to point to the file you configured on your Temporal Cluster: ```yaml logs: - type: file path: /var/log/temporal/temporal-server.log source: temporal ``` [Restart the Agent](https://docs.datadoghq.com/agent/guide/agent-commands/#start-stop-and-restart-the-agent). {% /tab %} {% tab title="Containerized" %} #### Containerized{% #containerized %} ##### Metric collection{% #metric-collection %} For containerized environments, refer to [Configure integrations with Autodiscovery on Kubernetes](https://docs.datadoghq.com/containers/kubernetes/integrations/) or [Configure integrations with Autodiscovery on Docker](https://docs.datadoghq.com/containers/docker/integrations/) for instructions on using the parameters below. See the [sample temporal.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/temporal/datadog_checks/temporal/data/conf.yaml.example) for a complete list of configuration options. | Parameter | Value | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `` | `temporal` | | `` | blank or `{}` | | `` | `{"openmetrics_endpoint": "/"}`, where `` and `` are replaced by the `listenAddress` and `handlerPath` from your Temporal server configuration. | Note that when Temporal services in a cluster are deployed independently, every service exposes its own metrics. As a result, you need to configure the `prometheus` endpoint for every service that you want to monitor and define a separate `instance` on the integration's configuration for each of them. **Example** The following Kubernetes annotation is applied to a pod under `metadata`, where `` is the name of your Temporal container (or a [custom identifier](https://docs.datadoghq.com/containers/guide/ad_identifiers/)): ``` ad.datadoghq.com/.checks: | { "temporal": { "init_config": {}, "instances": [{"openmetrics_endpoint": "/"}] } } ``` ##### Log collection{% #log-collection %} Collecting logs is disabled by default in the Datadog Agent. To enable it, see [Docker Log Collection](https://docs.datadoghq.com/containers/docker/log/) or [Kubernetes Log Collection](https://docs.datadoghq.com/agent/kubernetes/log/). Apply the following configuration parameter to `logs`: | Parameter | Value | | -------------- | ----------------------------------------------------------------------------------------- | | `` | `{"source": "temporal", "type": "file", "path": "/var/log/temporal/temporal-server.log"}` | **Example** The following Kubernetes annotation is applied to a pod under `metadata`, where `` is the name of your Temporal container (or a [custom identifier](https://docs.datadoghq.com/containers/guide/ad_identifiers/)): ```gdscript3 ad.datadoghq.com/.logs: | [ { "source": "temporal", "type": "file", "path": "/var/log/temporal/temporal-server.log" } ] ``` {% /tab %} ### Validation{% #validation %} [Run the Agent's status subcommand](https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information) and look for `temporal` under the Checks section. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **temporal.server.accept\_workflow\_update\_message.count**(count) | | **temporal.server.ack\_level.update.count**(count) | | **temporal.server.ack\_level.update.failed.count**(count) | | **temporal.server.acquire\_lock\_failed.count**(count) | | **temporal.server.acquire\_shards.count**(count) | | **temporal.server.acquire\_shards.latency.bucket**(count) | | **temporal.server.acquire\_shards.latency.count**(count) | | **temporal.server.acquire\_shards.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.action.count**(count) | | **temporal.server.activity.eager\_execution.count**(count) | | **temporal.server.activity.end\_to\_end\_latency.bucket**(count) | | **temporal.server.activity.end\_to\_end\_latency.count**(count) | | **temporal.server.activity.end\_to\_end\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.activity\_info.bucket**(count) | | **temporal.server.activity\_info.count**(count) | | **temporal.server.activity\_info.size.bucket**(count) | | **temporal.server.activity\_info.size.count**(count) | | **temporal.server.activity\_info.size.sum**(count) | *Shown as byte* | | **temporal.server.activity\_info.sum**(count) | | **temporal.server.add\_search\_attributes.failures.count**(count) | | **temporal.server.add\_search\_attributes.workflow\_failure**(gauge) | | **temporal.server.add\_search\_attributes.workflow\_failure.count**(count) | | **temporal.server.add\_search\_attributes.workflow\_success.count**(count) | | **temporal.server.approximate\_backlog\_age\_seconds**(gauge) | | **temporal.server.approximate\_backlog\_count**(gauge) | | **temporal.server.archival.task\_invalid\_uri.count**(count) | | **temporal.server.archiver.archive.latency.bucket**(count) | | **temporal.server.archiver.archive.latency.count**(count) | | **temporal.server.archiver.archive.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.archiver.archive.target\_latency.bucket**(count) | | **temporal.server.archiver.archive.target\_latency.count**(count) | | **temporal.server.archiver.archive.target\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.archiver.backlog.size.count**(count) | | **temporal.server.archiver.client.history.inline\_archive.attempt.count**(count) | | **temporal.server.archiver.client.history.inline\_archive.failure.count**(count) | | **temporal.server.archiver.client.history.request.count**(count) | | **temporal.server.archiver.client.send\_signal\_error.count**(count) | | **temporal.server.archiver.client.sent\_signal.count**(count) | | **temporal.server.archiver.client.visibility.inline\_archive\_attempt.count**(count) | | **temporal.server.archiver.client.visibility.inline\_archive\_failure.count**(count) | | **temporal.server.archiver.client.visibility.request.count**(count) | | **temporal.server.archiver.coroutine.started.count**(count) | | **temporal.server.archiver.coroutine.stopped.count**(count) | | **temporal.server.archiver.delete.failed\_all\_retries.count**(count) | | **temporal.server.archiver.delete.success.count**(count) | | **temporal.server.archiver.delete.with\_retries.latency.bucket**(count) | | **temporal.server.archiver.delete.with\_retries.latency.count**(count) | | **temporal.server.archiver.delete.with\_retries.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.archiver.handle\_all\_requests\_latency.bucket**(count) | | **temporal.server.archiver.handle\_all\_requests\_latency.count**(count) | | **temporal.server.archiver.handle\_all\_requests\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.archiver.handle\_history.request.latency.bucket**(count) | | **temporal.server.archiver.handle\_history.request.latency.count**(count) | | **temporal.server.archiver.handle\_history.request.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.archiver.handle\_visibility.failed\_all\_retries.count**(count) | | **temporal.server.archiver.handle\_visibility.request.latency.bucket**(count) | | **temporal.server.archiver.handle\_visibility.request.latency.count**(count) | | **temporal.server.archiver.handle\_visibility.request.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.archiver.handle\_visibility.success.count**(count) | | **temporal.server.archiver.non\_retryable\_error.count**(count) | | **temporal.server.archiver.num\_handled\_requests.count**(count) | | **temporal.server.archiver.num\_pumped\_requests.count**(count) | | **temporal.server.archiver.pump.signal\_channel\_closed.count**(count) | | **temporal.server.archiver.pump.signal\_threshold.count**(count) | | **temporal.server.archiver.pump.timeout.count**(count) | | **temporal.server.archiver.pump.timeout\_without\_signals.count**(count) | | **temporal.server.archiver.pumped\_not\_equal\_handled.count**(count) | | **temporal.server.archiver.started.count**(count) | | **temporal.server.archiver.stopped.count**(count) | | **temporal.server.archiver.upload.failed\_all\_retries.count**(count) | | **temporal.server.archiver.upload.success.count**(count) | | **temporal.server.archiver.upload.with\_retries.latency.bucket**(count) | | **temporal.server.archiver.upload.with\_retries.latency.count**(count) | | **temporal.server.archiver.upload.with\_retries.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.archiver.workflow.started.count**(count) | | **temporal.server.archiver.workflow.stopping.count**(count) | | **temporal.server.asyncmatch.latency.bucket**(count) | Distribution of latencies from creation to delivery for async matched tasks. | | **temporal.server.asyncmatch.latency.count**(count) | Count of latencies from creation to delivery for async matched tasks. | | **temporal.server.asyncmatch.latency.sum**(count) | Sum of time latencies creation to delivery for async matched tasks.*Shown as millisecond* | | **temporal.server.auto\_reset\_point.corruption.count**(count) | | **temporal.server.auto\_reset\_points.exceed\_limit.count**(count) | | **temporal.server.batchable\_task.batch\_count**(gauge) | | **temporal.server.batcher.operation\_errors.count**(count) | | **temporal.server.batcher.processor\_errors.count**(count) | | **temporal.server.batcher.processor\_requests.count**(count) | | **temporal.server.buffer\_replication\_tasks.bucket**(count) | | **temporal.server.buffer\_replication\_tasks.count**(count) | | **temporal.server.buffer\_replication\_tasks.sum**(count) | *Shown as millisecond* | | **temporal.server.buffer\_throttle.count**(count) | | **temporal.server.buffered\_events.bucket**(count) | | **temporal.server.buffered\_events.count**(count) | | **temporal.server.buffered\_events.size.bucket**(count) | | **temporal.server.buffered\_events.size.count**(count) | | **temporal.server.buffered\_events.size.sum**(count) | *Shown as byte* | | **temporal.server.buffered\_events.sum**(count) | | **temporal.server.cache.errors.count**(count) | | **temporal.server.cache.latency.bucket**(count) | | **temporal.server.cache.latency.count**(count) | | **temporal.server.cache.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.cache.miss.count**(count) | | **temporal.server.cache.pinned\_usage**(gauge) | | **temporal.server.cache.requests.count**(count) | | **temporal.server.cache.size**(gauge) | | **temporal.server.cache.ttl.bucket**(count) | | **temporal.server.cache.ttl.count**(count) | | **temporal.server.cache.ttl.sum**(count) | *Shown as millisecond* | | **temporal.server.cache.usage**(gauge) | | **temporal.server.cache\_entry.age\_on\_eviction.bucket**(count) | | **temporal.server.cache\_entry.age\_on\_eviction.count**(count) | | **temporal.server.cache\_entry.age\_on\_eviction.sum**(count) | *Shown as millisecond* | | **temporal.server.cache\_entry.age\_on\_get.bucket**(count) | | **temporal.server.cache\_entry.age\_on\_get.count**(count) | | **temporal.server.cache\_entry.age\_on\_get.sum**(count) | *Shown as millisecond* | | **temporal.server.cancel\_activity\_command.count**(count) | | **temporal.server.cancel\_external\_workflow\_command.count**(count) | | **temporal.server.cancel\_timer\_command.count**(count) | | **temporal.server.cancel\_workflow\_command.count**(count) | | **temporal.server.cassandra.init\_session.latency.bucket**(count) | | **temporal.server.cassandra.init\_session.latency.count**(count) | | **temporal.server.cassandra.init\_session.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.cassandra.session\_refresh.failures.count**(count) | | **temporal.server.catchup.ready\_shard\_count**(gauge) | | **temporal.server.certificates.expired**(gauge) | | **temporal.server.certificates.expiring**(gauge) | | **temporal.server.chasm.total\_size.bucket**(count) | | **temporal.server.chasm.total\_size.count**(count) | | **temporal.server.chasm.total\_size.sum**(count) | *Shown as byte* | | **temporal.server.child\_info.bucket**(count) | | **temporal.server.child\_info.count**(count) | | **temporal.server.child\_info.size.bucket**(count) | | **temporal.server.child\_info.size.count**(count) | | **temporal.server.child\_info.size.sum**(count) | *Shown as byte* | | **temporal.server.child\_info.sum**(count) | | **temporal.server.child\_workflow\_command.count**(count) | | **temporal.server.circuit\_breaker.executable\_blocked.count**(count) | | **temporal.server.client.errors.count**(count) | An indicator for connection issues between different Server roles. | | **temporal.server.client.latency.bucket**(count) | | **temporal.server.client.latency.count**(count) | | **temporal.server.client.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.client.redirection.errors.count**(count) | | **temporal.server.client.redirection.latency.bucket**(count) | | **temporal.server.client.redirection.latency.count**(count) | | **temporal.server.client.redirection.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.client.redirection.requests.count**(count) | | **temporal.server.client.requests.count**(count) | | **temporal.server.closed\_workflow\_buffer\_event\_counter.count**(count) | | **temporal.server.cluster\_metadata.callback.lock\_latency.bucket**(count) | | **temporal.server.cluster\_metadata.callback.lock\_latency.count**(count) | | **temporal.server.cluster\_metadata.callback.lock\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.cluster\_metadata.lock\_latency.bucket**(count) | | **temporal.server.cluster\_metadata.lock\_latency.count**(count) | | **temporal.server.cluster\_metadata.lock\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.command.count**(count) | | **temporal.server.complete\_task\_fail.count**(count) | | **temporal.server.complete\_workflow\_command.count**(count) | | **temporal.server.complete\_workflow\_task\_sticky.disabled.count**(count) | | **temporal.server.complete\_workflow\_task\_sticky.enabled.count**(count) | | **temporal.server.complete\_workflow\_update\_message.count**(count) | | **temporal.server.concurrency\_update\_failure.count**(count) | | **temporal.server.condition\_failed\_errors.count**(count) | | **temporal.server.consistent\_query\_timeout.count**(count) | | **temporal.server.continue\_as\_new\_command.count**(count) | | **temporal.server.count\_executions.failures.count**(count) | | **temporal.server.dd.cluster\_metadata.callback\_lock.latency.bucket**(count) | | **temporal.server.dd.cluster\_metadata.callback\_lock.latency.count**(count) | | **temporal.server.dd.cluster\_metadata.callback\_lock.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.dd.cluster\_metadata\_lock\_latency.bucket**(count) | | **temporal.server.dd.cluster\_metadata\_lock\_latency.count**(count) | | **temporal.server.dd.cluster\_metadata\_lock\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.dd.current\_suspected\_deadlocks**(gauge) | | **temporal.server.dd.namespace\_registry\_lock\_latency.bucket**(count) | | **temporal.server.dd.namespace\_registry\_lock\_latency.count**(count) | | **temporal.server.dd.namespace\_registry\_lock\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.dd.shard\_controller\_lock\_latency.bucket**(count) | | **temporal.server.dd.shard\_controller\_lock\_latency.count**(count) | | **temporal.server.dd.shard\_controller\_lock\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.dd.shard\_io\_semaphore\_latency.bucket**(count) | | **temporal.server.dd.shard\_io\_semaphore\_latency.count**(count) | | **temporal.server.dd.shard\_io\_semaphore\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.dd.shard\_lock\_latency.bucket**(count) | | **temporal.server.dd.shard\_lock\_latency.count**(count) | | **temporal.server.dd.shard\_lock\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.dd.suspected\_deadlocks.count**(count) | | **temporal.server.delete\_execution.failures.count**(count) | | **temporal.server.delete\_execution.not\_found.count**(count) | | **temporal.server.delete\_executions.failure.count**(count) | | **temporal.server.delete\_executions.not\_found.count**(count) | | **temporal.server.delete\_executions.success.count**(count) | | **temporal.server.delete\_namespace.failures.count**(count) | | **temporal.server.delete\_namespace.success.count**(count) | | **temporal.server.delete\_namespace.workflow\_failure.count**(count) | | **temporal.server.delete\_namespace.workflow\_success.count**(count) | | **temporal.server.direct\_query\_dispatch.clear\_stickiness.latency.bucket**(count) | | **temporal.server.direct\_query\_dispatch.clear\_stickiness.latency.count**(count) | | **temporal.server.direct\_query\_dispatch.clear\_stickiness.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.direct\_query\_dispatch.clear\_stickiness.success.count**(count) | | **temporal.server.direct\_query\_dispatch.latency.bucket**(count) | | **temporal.server.direct\_query\_dispatch.latency.count**(count) | | **temporal.server.direct\_query\_dispatch.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.direct\_query\_dispatch.non\_sticky.latency.bucket**(count) | | **temporal.server.direct\_query\_dispatch.non\_sticky.latency.count**(count) | | **temporal.server.direct\_query\_dispatch.non\_sticky.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.direct\_query\_dispatch.non\_sticky.success.count**(count) | | **temporal.server.direct\_query\_dispatch.sticky.latency.bucket**(count) | | **temporal.server.direct\_query\_dispatch.sticky.latency.count**(count) | | **temporal.server.direct\_query\_dispatch.sticky.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.direct\_query\_dispatch.sticky.success.count**(count) | | **temporal.server.direct\_query\_dispatch.timeout\_before\_non\_sticky.count**(count) | | **temporal.server.dlq.message\_count**(gauge) | | **temporal.server.dlq\_writes.count**(count) | | **temporal.server.duplicate\_replication\_events.count**(count) | | **temporal.server.dynamic\_rate\_limit\_multiplier**(gauge) | | **temporal.server.dynamic\_worker\_pool\_scheduler.active\_workers**(gauge) | | **temporal.server.dynamic\_worker\_pool\_scheduler.buffer\_size**(gauge) | | **temporal.server.dynamic\_worker\_pool\_scheduler.dequeued\_tasks.count**(count) | | **temporal.server.dynamic\_worker\_pool\_scheduler.enqueued\_tasks.count**(count) | | **temporal.server.dynamic\_worker\_pool\_scheduler.rejected\_tasks.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.bulk\_request\_took\_latency.bucket**(count) | | **temporal.server.elasticsearch.bulk\_processor.bulk\_request\_took\_latency.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.bulk\_request\_took\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.elasticsearch.bulk\_processor.bulk\_size.bucket**(count) | | **temporal.server.elasticsearch.bulk\_processor.bulk\_size.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.bulk\_size.sum**(count) | | **temporal.server.elasticsearch.bulk\_processor.commit.latency.bucket**(count) | | **temporal.server.elasticsearch.bulk\_processor.commit.latency.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.commit.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.elasticsearch.bulk\_processor.corrupted\_data.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.duplicate\_request.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.errors.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.queued\_requests.bucket**(count) | | **temporal.server.elasticsearch.bulk\_processor.queued\_requests.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.queued\_requests.sum**(count) | | **temporal.server.elasticsearch.bulk\_processor.request.latency.bucket**(count) | | **temporal.server.elasticsearch.bulk\_processor.request.latency.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.request.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.elasticsearch.bulk\_processor.requests.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.wait\_add.latency.bucket**(count) | | **temporal.server.elasticsearch.bulk\_processor.wait\_add.latency.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.wait\_add.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.elasticsearch.bulk\_processor.wait\_start.latency.bucket**(count) | | **temporal.server.elasticsearch.bulk\_processor.wait\_start.latency.count**(count) | | **temporal.server.elasticsearch.bulk\_processor.wait\_start.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.elasticsearch.bulk\_processor\_retries.count**(count) | | **temporal.server.elasticsearch.custom\_order\_by\_clause\_counter.count**(count) | | **temporal.server.elasticsearch.document.generate\_failures\_counter.count**(count) | | **temporal.server.elasticsearch.document.parse\_failures\_counter.count**(count) | | **temporal.server.empty\_completion\_commands.count**(count) | | **temporal.server.empty\_replication\_events.count**(count) | | **temporal.server.encounter\_not\_found\_workflow\_count.count**(count) | | **temporal.server.encounter\_pass\_retention\_workflow\_count.count**(count) | | **temporal.server.encounter\_zombie\_workflow\_count.count**(count) | | **temporal.server.event.blob\_size.bucket**(count) | | **temporal.server.event.blob\_size.count**(count) | | **temporal.server.event.blob\_size.sum**(count) | *Shown as byte* | | **temporal.server.event.reapply\_skipped.count**(count) | | **temporal.server.execution\_info.size.bucket**(count) | | **temporal.server.execution\_info.size.count**(count) | | **temporal.server.execution\_info.size.sum**(count) | *Shown as byte* | | **temporal.server.execution\_state.size.bucket**(count) | | **temporal.server.execution\_state.size.count**(count) | | **temporal.server.execution\_state.size.sum**(count) | *Shown as byte* | | **temporal.server.executions\_outstanding**(gauge) | | **temporal.server.executor.deferred.count**(count) | | **temporal.server.executor.done.count**(count) | | **temporal.server.executor.dropped.count**(count) | | **temporal.server.executor.err.count**(count) | | **temporal.server.fail\_workflow\_command.count**(count) | | **temporal.server.failed\_workflow\_tasks.count**(count) | | **temporal.server.finalizer.items\_completed.count**(count) | | **temporal.server.finalizer.items\_unfinished.count**(count) | | **temporal.server.finalizer.latency.bucket**(count) | | **temporal.server.finalizer.latency.count**(count) | | **temporal.server.finalizer.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.force\_loaded\_task\_queue\_partition\_unnecessarily\_count.count**(count) | | **temporal.server.force\_loaded\_task\_queue\_partitions\_count.count**(count) | | **temporal.server.forward\_poll.calls.count**(count) | | **temporal.server.forward\_poll.errors.count**(count) | | **temporal.server.forward\_poll.latency.bucket**(count) | | **temporal.server.forward\_poll.latency.count**(count) | | **temporal.server.forward\_poll.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.forward\_query.calls.count**(count) | | **temporal.server.forward\_query.errors.count**(count) | | **temporal.server.forward\_query.latency.bucket**(count) | | **temporal.server.forward\_query.latency.count**(count) | | **temporal.server.forward\_query.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.forward\_task.calls.count**(count) | | **temporal.server.forward\_task.errors.count**(count) | | **temporal.server.forward\_task.latency.bucket**(count) | | **temporal.server.forward\_task.latency.count**(count) | | **temporal.server.forward\_task.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.forwarded.count**(count) | | **temporal.server.forwarded\_per\_tl.count**(count) | | **temporal.server.generate\_replication\_tasks\_latency.bucket**(count) | | **temporal.server.generate\_replication\_tasks\_latency.count**(count) | | **temporal.server.generate\_replication\_tasks\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.get\_dlq\_replication\_messages.bucket**(count) | | **temporal.server.get\_dlq\_replication\_messages.count**(count) | | **temporal.server.get\_dlq\_replication\_messages.sum**(count) | *Shown as millisecond* | | **temporal.server.get\_engine\_for\_shard.errors.count**(count) | | **temporal.server.get\_engine\_for\_shard.latency.bucket**(count) | | **temporal.server.get\_engine\_for\_shard.latency.count**(count) | | **temporal.server.get\_engine\_for\_shard.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.get\_replication\_messages\_for\_shard.bucket**(count) | | **temporal.server.get\_replication\_messages\_for\_shard.count**(count) | | **temporal.server.get\_replication\_messages\_for\_shard.sum**(count) | *Shown as millisecond* | | **temporal.server.gomaxprocs**(gauge) | | **temporal.server.handover.ready\_shard\_count**(gauge) | | **temporal.server.handover.wait\_latency.bucket**(count) | | **temporal.server.handover.wait\_latency.count**(count) | | **temporal.server.handover.wait\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.heartbeat.timeout.count**(count) | | **temporal.server.history.archiver.archive.non\_retryable\_error.count**(count) | | **temporal.server.history.archiver.archive.success.count**(count) | | **temporal.server.history.archiver.archive.transient\_error.count**(count) | | **temporal.server.history.archiver.blob\_exists.count**(count) | | **temporal.server.history.archiver.blob\_integrity\_check\_failed.count**(count) | | **temporal.server.history.archiver.blob\_size.bucket**(count) | | **temporal.server.history.archiver.blob\_size.count**(count) | | **temporal.server.history.archiver.blob\_size.sum**(count) | *Shown as byte* | | **temporal.server.history.archiver.deterministic\_construction\_check\_failed.count**(count) | | **temporal.server.history.archiver.duplicate\_archivals.count**(count) | | **temporal.server.history.archiver.history\_mutated.count**(count) | | **temporal.server.history.archiver.history\_size.bucket**(count) | | **temporal.server.history.archiver.history\_size.count**(count) | | **temporal.server.history.archiver.history\_size.sum**(count) | *Shown as byte* | | **temporal.server.history.archiver.running\_blob\_integrity\_check.count**(count) | | **temporal.server.history.archiver.running\_deterministic\_construction\_check.count**(count) | | **temporal.server.history.archiver.total\_upload\_size.bucket**(count) | | **temporal.server.history.archiver.total\_upload\_size.count**(count) | | **temporal.server.history.archiver.total\_upload\_size.sum**(count) | *Shown as byte* | | **temporal.server.history.bucket**(count) | | **temporal.server.history.conflicts.count**(count) | | **temporal.server.history.count**(count) | | **temporal.server.history.event\_notification.fail\_delivery.count**(count) | | **temporal.server.history.event\_notification.fanout\_latency.bucket**(count) | | **temporal.server.history.event\_notification.fanout\_latency.count**(count) | | **temporal.server.history.event\_notification.fanout\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.history.event\_notification.inflight\_message**(gauge) | | **temporal.server.history.event\_notification.queueing\_latency.bucket**(count) | | **temporal.server.history.event\_notification.queueing\_latency.count**(count) | | **temporal.server.history.event\_notification.queueing\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.history.size.bucket**(count) | | **temporal.server.history.size.count**(count) | | **temporal.server.history.size.sum**(count) | *Shown as byte* | | **temporal.server.history.sum**(count) | | **temporal.server.history.workflow\_execution\_cache\_latency.bucket**(count) | | **temporal.server.history.workflow\_execution\_cache\_latency.count**(count) | | **temporal.server.history.workflow\_execution\_cache\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.history\_workflow\_execution\_cache.lock\_hold\_duration.bucket**(count) | | **temporal.server.history\_workflow\_execution\_cache.lock\_hold\_duration.count**(count) | | **temporal.server.history\_workflow\_execution\_cache.lock\_hold\_duration.sum**(count) | *Shown as millisecond* | | **temporal.server.host.health**(gauge) | | **temporal.server.host\_rps\_limit**(gauge) | | **temporal.server.http.service\_requests.count**(count) | | **temporal.server.inordered\_buffered\_events.count**(count) | | **temporal.server.invalid\_state\_transition\_workflow\_update\_message.count**(count) | | **temporal.server.invalid\_task\_queue\_name.count**(count) | | **temporal.server.invalid\_task\_queue\_partition.count**(count) | | **temporal.server.last\_processed\_message\_id**(gauge) | | **temporal.server.last\_retrieved\_message\_id**(gauge) | | **temporal.server.lease.failures.count**(count) | | **temporal.server.lease.requests.count**(count) | | **temporal.server.list\_executions.failures.count**(count) | | **temporal.server.loaded\_physical\_task\_queue\_count**(gauge) | | **temporal.server.loaded\_task\_queue\_count**(gauge) | | **temporal.server.loaded\_task\_queue\_family\_count**(gauge) | | **temporal.server.loaded\_task\_queue\_partition\_count**(gauge) | | **temporal.server.local\_to\_local.matches.count**(count) | | **temporal.server.local\_to\_remote.matches.count**(count) | | **temporal.server.lock.failures.count**(count) | | **temporal.server.lock.latency.bucket**(count) | | **temporal.server.lock.latency.count**(count) | | **temporal.server.lock.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.lock.requests.count**(count) | | **temporal.server.membership\_changed.count**(count) | | **temporal.server.memo\_size.bucket**(count) | | **temporal.server.memo\_size.count**(count) | | **temporal.server.memo\_size.sum**(count) | *Shown as byte* | | **temporal.server.memory.allocated**(gauge) | | **temporal.server.memory.frees**(gauge) | | **temporal.server.memory.gc\_pause\_ms.bucket**(count) | | **temporal.server.memory.gc\_pause\_ms.count**(count) | | **temporal.server.memory.gc\_pause\_ms.sum**(count) | *Shown as millisecond* | | **temporal.server.memory.heap**(gauge) | | **temporal.server.memory.heap\_objects**(gauge) | | **temporal.server.memory.heapidle**(gauge) | | **temporal.server.memory.heapinuse**(gauge) | | **temporal.server.memory.mallocs**(gauge) | | **temporal.server.memory.num\_gc.bucket**(count) | | **temporal.server.memory.num\_gc.count**(count) | | **temporal.server.memory.num\_gc.sum**(count) | *Shown as byte* | | **temporal.server.memory.num\_gc\_last**(gauge) | | **temporal.server.memory.pause\_total\_ns\_last**(gauge) | | **temporal.server.memory.stack**(gauge) | | **temporal.server.modify\_workflow\_properties\_command.count**(count) | | **temporal.server.multiple\_completion\_commands.count**(count) | | **temporal.server.mutable\_state.size.bucket**(count) | | **temporal.server.mutable\_state.size.count**(count) | | **temporal.server.mutable\_state.size.sum**(count) | *Shown as byte* | | **temporal.server.mutable\_state\_checksum.invalidated.count**(count) | | **temporal.server.mutable\_state\_checksum.mismatch.count**(count) | | **temporal.server.mutable\_state\_dirty.count**(count) | | **temporal.server.namespace\_cache.callbacks\_latency.bucket**(count) | | **temporal.server.namespace\_cache.callbacks\_latency.count**(count) | | **temporal.server.namespace\_cache.callbacks\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.namespace\_cache.prepare\_callbacks\_latency.bucket**(count) | | **temporal.server.namespace\_cache.prepare\_callbacks\_latency.count**(count) | | **temporal.server.namespace\_cache.prepare\_callbacks\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.namespace\_dlq.ack\_level**(gauge) | | **temporal.server.namespace\_dlq.max\_level**(gauge) | | **temporal.server.namespace\_host\_rps\_limit**(gauge) | | **temporal.server.namespace\_registry.callback\_lock\_latency.bucket**(count) | | **temporal.server.namespace\_registry.callback\_lock\_latency.count**(count) | | **temporal.server.namespace\_registry.callback\_lock\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.namespace\_registry.lock\_latency.bucket**(count) | | **temporal.server.namespace\_registry.lock\_latency.count**(count) | | **temporal.server.namespace\_registry.lock\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.namespace\_replication.dlq\_enqueue\_requests.count**(count) | | **temporal.server.namespace\_replication.task\_ack\_level**(gauge) | | **temporal.server.new\_timer\_notifications.count**(count) | | **temporal.server.nexus.completion\_latency.bucket**(count) | | **temporal.server.nexus.completion\_latency.count**(count) | | **temporal.server.nexus.completion\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.nexus.completion\_request\_preprocess\_errors.count**(count) | | **temporal.server.nexus.completion\_requests.count**(count) | | **temporal.server.nexus.latency.bucket**(count) | | **temporal.server.nexus.latency.count**(count) | | **temporal.server.nexus.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.nexus.request\_preprocess\_errors.count**(count) | | **temporal.server.nexus.requests.count**(count) | | **temporal.server.no\_poller\_tasks.count**(count) | Tasks added to a task queue that has no poller. | | **temporal.server.non\_retryable\_tasks.count**(count) | | **temporal.server.num\_goroutines**(gauge) | | **temporal.server.numshards**(gauge) | | **temporal.server.operation.count**(count) | | **temporal.server.out\_of\_order\_buffered\_events.count**(count) | | **temporal.server.parent\_close\_policy\_processor.errors.count**(count) | | **temporal.server.parent\_close\_policy\_processor.requests.count**(count) | | **temporal.server.paused\_activities.count**(count) | | **temporal.server.pending\_tasks.bucket**(count) | | **temporal.server.pending\_tasks.count**(count) | | **temporal.server.pending\_tasks.sum**(count) | | **temporal.server.persisted\_mutable\_state\_size.bucket**(count) | | **temporal.server.persisted\_mutable\_state\_size.count**(count) | | **temporal.server.persisted\_mutable\_state\_size.sum**(count) | *Shown as byte* | | **temporal.server.persistence.error\_with\_type.count**(count) | Number of persistence errors, tagged by error type. | | **temporal.server.persistence.errors.bad\_request.count**(count) | | **temporal.server.persistence.errors.busy.count**(count) | | **temporal.server.persistence.errors.condition\_failed.count**(count) | | **temporal.server.persistence.errors.count**(count) | Number of persistence errors. | | **temporal.server.persistence.errors.current\_workflow\_condition\_failed.count**(count) | | **temporal.server.persistence.errors.entity\_not\_exists.count**(count) | | **temporal.server.persistence.errors.namespace\_already\_exists.count**(count) | | **temporal.server.persistence.errors.resource\_exhausted.count**(count) | | **temporal.server.persistence.errors.shard\_exists.count**(count) | | **temporal.server.persistence.errors.shard\_ownership\_lost.count**(count) | | **temporal.server.persistence.errors.timeout.count**(count) | | **temporal.server.persistence.errors.workflow\_condition\_failed.count**(count) | | **temporal.server.persistence.latency.bucket**(count) | Distribution of latencies on persistence operations. | | **temporal.server.persistence.latency.count**(count) | Count of latencies on persistence operations. | | **temporal.server.persistence.latency.sum**(count) | Sum of latencies on persistence operations.*Shown as millisecond* | | **temporal.server.persistence.requests.count**(count) | Number of persistence requests. | | **temporal.server.persistence.session\_refresh\_attempts.count**(count) | | **temporal.server.persistence.session\_refresh\_failures.count**(count) | | **temporal.server.persistence.shard\_rps.bucket**(count) | | **temporal.server.persistence.shard\_rps.count**(count) | | **temporal.server.persistence.shard\_rps.sum**(count) | | **temporal.server.poll.success.count**(count) | Number of tasks successfully matched by the poller. | | **temporal.server.poll.success\_sync.count**(count) | Number of tasks successfully matched synchronously. | | **temporal.server.poll.timeouts.count**(count) | Number of times a poller timed out due to no tasks being available. | | **temporal.server.poll\_latency.bucket**(count) | | **temporal.server.poll\_latency.count**(count) | | **temporal.server.poll\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.protocol\_message\_command.count**(count) | | **temporal.server.query\_before\_first\_workflow\_task.count**(count) | | **temporal.server.query\_buffer\_exceeded.count**(count) | | **temporal.server.query\_registry\_invalid\_state.count**(count) | | **temporal.server.queue.action\_errors.count**(count) | | **temporal.server.queue.actions.count**(count) | | **temporal.server.queue.latency\_schedule.bucket**(count) | | **temporal.server.queue.latency\_schedule.count**(count) | | **temporal.server.queue.latency\_schedule.sum**(count) | *Shown as millisecond* | | **temporal.server.queue.reader.bucket**(count) | | **temporal.server.queue.reader.count**(count) | | **temporal.server.queue.reader.sum**(count) | | **temporal.server.queue.slice.bucket**(count) | | **temporal.server.queue.slice.count**(count) | | **temporal.server.queue.slice.sum**(count) | | **temporal.server.rate\_limited\_task.runnable\_wait\_time.bucket**(count) | | **temporal.server.rate\_limited\_task.runnable\_wait\_time.count**(count) | | **temporal.server.rate\_limited\_task.runnable\_wait\_time.sum**(count) | *Shown as millisecond* | | **temporal.server.rate\_limiter.failures.count**(count) | | **temporal.server.reachability\_exit\_point\_count.count**(count) | | **temporal.server.read\_namespace.errors.count**(count) | | **temporal.server.read\_namespace.failures.count**(count) | | **temporal.server.reclaim\_resources.delete\_executions.failure.count**(count) | | **temporal.server.reclaim\_resources.delete\_executions.success.count**(count) | | **temporal.server.reclaim\_resources.namespace\_delete.failure.count**(count) | | **temporal.server.reclaim\_resources.namespace\_delete.success.count**(count) | | **temporal.server.record\_marker\_command.count**(count) | | **temporal.server.reject\_workflow\_update\_message.count**(count) | | **temporal.server.remote\_to\_local.matches.count**(count) | | **temporal.server.remote\_to\_remote.matches.count**(count) | | **temporal.server.remove\_engine\_for\_shard.latency.bucket**(count) | | **temporal.server.remove\_engine\_for\_shard.latency.count**(count) | | **temporal.server.remove\_engine\_for\_shard.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.rename\_namespace.failures.count**(count) | | **temporal.server.rename\_namespace.success.count**(count) | | **temporal.server.replication.dlq.ack\_level**(gauge) | | **temporal.server.replication.dlq.enqueue\_failed.count**(count) | | **temporal.server.replication.dlq.max\_level**(gauge) | | **temporal.server.replication.dlq\_non\_empty.count**(count) | | **temporal.server.replication.duplicated\_task.count**(count) | | **temporal.server.replication.latency.bucket**(count) | | **temporal.server.replication.latency.count**(count) | | **temporal.server.replication.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.outlier\_namespace.count**(count) | | **temporal.server.replication.service\_error.count**(count) | | **temporal.server.replication.stream\_error.count**(count) | | **temporal.server.replication.stream\_panic.count**(count) | | **temporal.server.replication.stream\_stuck.count**(count) | | **temporal.server.replication.task\_cleanup.count**(count) | | **temporal.server.replication.task\_cleanup.failed.count**(count) | | **temporal.server.replication.task\_generation\_latency.bucket**(count) | | **temporal.server.replication.task\_generation\_latency.count**(count) | | **temporal.server.replication.task\_generation\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.task\_load\_latency.bucket**(count) | | **temporal.server.replication.task\_load\_latency.count**(count) | | **temporal.server.replication.task\_load\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.task\_load\_size.bucket**(count) | | **temporal.server.replication.task\_load\_size.count**(count) | | **temporal.server.replication.task\_load\_size.sum**(count) | | **temporal.server.replication.task\_processing\_latency.bucket**(count) | | **temporal.server.replication.task\_processing\_latency.count**(count) | | **temporal.server.replication.task\_processing\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.task\_send\_attempt.bucket**(count) | | **temporal.server.replication.task\_send\_attempt.count**(count) | | **temporal.server.replication.task\_send\_attempt.sum**(count) | | **temporal.server.replication.task\_send\_backlog.bucket**(count) | | **temporal.server.replication.task\_send\_backlog.count**(count) | | **temporal.server.replication.task\_send\_backlog.sum**(count) | | **temporal.server.replication.task\_send\_error.count**(count) | | **temporal.server.replication.task\_send\_latency.bucket**(count) | | **temporal.server.replication.task\_send\_latency.count**(count) | | **temporal.server.replication.task\_send\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.task\_transmission\_latency.bucket**(count) | | **temporal.server.replication.task\_transmission\_latency.count**(count) | | **temporal.server.replication.task\_transmission\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.tasks.applied.count**(count) | | **temporal.server.replication.tasks.applied\_latency.bucket**(count) | | **temporal.server.replication.tasks.applied\_latency.count**(count) | | **temporal.server.replication.tasks.applied\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.tasks.failed.count**(count) | | **temporal.server.replication.tasks.fetched.bucket**(count) | | **temporal.server.replication.tasks.fetched.count**(count) | | **temporal.server.replication.tasks.fetched.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.tasks.lag.bucket**(count) | | **temporal.server.replication.tasks.lag.count**(count) | | **temporal.server.replication.tasks.lag.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.tasks.returned.bucket**(count) | | **temporal.server.replication.tasks.returned.count**(count) | | **temporal.server.replication.tasks.returned.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.tasks\_back\_fill.count**(count) | | **temporal.server.replication.tasks\_back\_fill\_latency.bucket**(count) | | **temporal.server.replication.tasks\_back\_fill\_latency.count**(count) | | **temporal.server.replication.tasks\_back\_fill\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.replication.tasks\_recv.count**(count) | | **temporal.server.replication.tasks\_recv\_backlog.bucket**(count) | | **temporal.server.replication.tasks\_recv\_backlog.count**(count) | | **temporal.server.replication.tasks\_recv\_backlog.sum**(count) | | **temporal.server.replication.tasks\_send.count**(count) | | **temporal.server.replication.tasks\_skipped.count**(count) | | **temporal.server.replication\_events\_size.bucket**(count) | | **temporal.server.replication\_events\_size.count**(count) | | **temporal.server.replication\_events\_size.sum**(count) | *Shown as millisecond* | | **temporal.server.replicator.dlq\_enqueue\_fails.count**(count) | | **temporal.server.replicator.errors.count**(count) | | **temporal.server.replicator.latency.bucket**(count) | | **temporal.server.replicator.latency.count**(count) | | **temporal.server.replicator.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.replicator.messages.count**(count) | | **temporal.server.request\_cancel\_info.bucket**(count) | | **temporal.server.request\_cancel\_info.count**(count) | | **temporal.server.request\_cancel\_info.size.bucket**(count) | | **temporal.server.request\_cancel\_info.size.count**(count) | | **temporal.server.request\_cancel\_info.size.sum**(count) | *Shown as byte* | | **temporal.server.request\_cancel\_info.sum**(count) | | **temporal.server.request\_workflow\_update\_message.count**(count) | | **temporal.server.respond\_nexus\_failed.count**(count) | | **temporal.server.respond\_query\_failed.count**(count) | | **temporal.server.respond\_workflow\_update\_message.count**(count) | | **temporal.server.restarts.count**(count) | | **temporal.server.scan\_duration.bucket**(count) | | **temporal.server.scan\_duration.count**(count) | | **temporal.server.scan\_duration.sum**(count) | *Shown as millisecond* | | **temporal.server.scavenger.errors.count**(count) | | **temporal.server.scavenger.skips.count**(count) | | **temporal.server.scavenger.success.count**(count) | | **temporal.server.scavenger.validation.failures.count**(count) | | **temporal.server.scavenger.validation.requests.count**(count) | | **temporal.server.scavenger.validation.skips.count**(count) | | **temporal.server.schedule.action.errors.count**(count) | | **temporal.server.schedule.action.success.count**(count) | | **temporal.server.schedule.buffer\_overruns.count**(count) | | **temporal.server.schedule.cancel\_workflow.errors.count**(count) | | **temporal.server.schedule.missed\_catchup\_window.count**(count) | | **temporal.server.schedule.rate\_limited.count**(count) | | **temporal.server.schedule.terminate\_workflow.errors.count**(count) | | **temporal.server.schedule\_action.attempt.count**(count) | | **temporal.server.schedule\_action.dropped.count**(count) | | **temporal.server.schedule\_action\_delay.bucket**(count) | | **temporal.server.schedule\_action\_delay.count**(count) | | **temporal.server.schedule\_action\_delay.sum**(count) | *Shown as millisecond* | | **temporal.server.schedule\_activity\_command.count**(count) | | **temporal.server.schedule\_to\_close.timeout.count**(count) | | **temporal.server.schedule\_to\_start.timeout.count**(count) | | **temporal.server.search\_attributes\_size.bucket**(count) | | **temporal.server.search\_attributes\_size.count**(count) | | **temporal.server.search\_attributes\_size.sum**(count) | *Shown as byte* | | **temporal.server.semaphore.failures.count**(count) | | **temporal.server.semaphore.latency.bucket**(count) | | **temporal.server.semaphore.latency.count**(count) | | **temporal.server.semaphore.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.semaphore.requests.count**(count) | | **temporal.server.service.authorization\_latency.bucket**(count) | | **temporal.server.service.authorization\_latency.count**(count) | | **temporal.server.service.authorization\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.service.error\_with\_type.count**(count) | Errors encountered by the service, tagged by type. | | **temporal.server.service.errors.authorize\_failed.count**(count) | | **temporal.server.service.errors.bad\_binary.count**(count) | | **temporal.server.service.errors.cancellation\_already\_requested.count**(count) | | **temporal.server.service.errors.client\_version\_not\_supported.count**(count) | | **temporal.server.service.errors.context\_cancelled.count**(count) | | **temporal.server.service.errors.context\_timeout.count**(count) | | **temporal.server.service.errors.count**(count) | | **temporal.server.service.errors.critical.count**(count) | | **temporal.server.service.errors.entity\_not\_found.count**(count) | | **temporal.server.service.errors.execution\_already\_started.count**(count) | | **temporal.server.service.errors.incomplete\_history.count**(count) | | **temporal.server.service.errors.invalid\_argument.count**(count) | | **temporal.server.service.errors.namespace\_already\_exists.count**(count) | | **temporal.server.service.errors.namespace\_not\_active.count**(count) | | **temporal.server.service.errors.nondeterministic.count**(count) | | **temporal.server.service.errors.query\_failed.count**(count) | | **temporal.server.service.errors.resource\_exhausted.count**(count) | | **temporal.server.service.errors.retry\_task.count**(count) | | **temporal.server.service.errors.shard\_ownership\_lost.count**(count) | | **temporal.server.service.errors.task\_already\_started.count**(count) | | **temporal.server.service.errors.unauthorized.count**(count) | | **temporal.server.service.latency.bucket**(count) | Distribution of latencies for all client request operations. | | **temporal.server.service.latency.count**(count) | Count of latencies for all client request operations. | | **temporal.server.service.latency.nouserlatency.bucket**(count) | | **temporal.server.service.latency.nouserlatency.count**(count) | | **temporal.server.service.latency.nouserlatency.sum**(count) | *Shown as millisecond* | | **temporal.server.service.latency.sum**(count) | Sum of latencies for all client requests operations.*Shown as millisecond* | | **temporal.server.service.latency.userlatency.bucket**(count) | | **temporal.server.service.latency.userlatency.count**(count) | | **temporal.server.service.latency.userlatency.sum**(count) | *Shown as millisecond* | | **temporal.server.service.panics.count**(count) | | **temporal.server.service.pending\_requests**(gauge) | | **temporal.server.service.requests.count**(count) | Service requests received per Task Queue. | | **temporal.server.shard.linger\_success.bucket**(count) | | **temporal.server.shard.linger\_success.count**(count) | | **temporal.server.shard.linger\_success.sum**(count) | *Shown as millisecond* | | **temporal.server.shard.linger\_timeouts.count**(count) | | **temporal.server.shard.lock\_latency.bucket**(count) | | **temporal.server.shard.lock\_latency.count**(count) | | **temporal.server.shard.lock\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.shard\_closed.count**(count) | | **temporal.server.shard\_controller.lock\_latency.bucket**(count) | | **temporal.server.shard\_controller.lock\_latency.count**(count) | | **temporal.server.shard\_controller.lock\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.shardinfo.immediate\_queue.lag.bucket**(count) | | **temporal.server.shardinfo.immediate\_queue.lag.count**(count) | | **temporal.server.shardinfo.immediate\_queue.lag.sum**(count) | | **temporal.server.shardinfo.replication.lag.bucket**(count) | | **temporal.server.shardinfo.replication.lag.count**(count) | | **temporal.server.shardinfo.replication.lag.sum**(count) | | **temporal.server.shardinfo.replication.pending\_task.bucket**(count) | | **temporal.server.shardinfo.replication.pending\_task.count**(count) | | **temporal.server.shardinfo.replication.pending\_task.sum**(count) | | **temporal.server.shardinfo.scheduled\_queue.lag.bucket**(count) | | **temporal.server.shardinfo.scheduled\_queue.lag.count**(count) | | **temporal.server.shardinfo.scheduled\_queue.lag.sum**(count) | *Shown as millisecond* | | **temporal.server.shardinfo.timer.active.pending\_task.bucket**(count) | | **temporal.server.shardinfo.timer.active.pending\_task.count**(count) | | **temporal.server.shardinfo.timer.active.pending\_task.sum**(count) | | **temporal.server.shardinfo.timer.lag.bucket**(count) | | **temporal.server.shardinfo.timer.lag.count**(count) | | **temporal.server.shardinfo.timer.lag.sum**(count) | *Shown as millisecond* | | **temporal.server.shardinfo.timer.standby.pending\_task.bucket**(count) | | **temporal.server.shardinfo.timer.standby.pending\_task.count**(count) | | **temporal.server.shardinfo.timer.standby.pending\_task.sum**(count) | | **temporal.server.shardinfo.timer\_diff.bucket**(count) | | **temporal.server.shardinfo.timer\_diff.count**(count) | | **temporal.server.shardinfo.timer\_diff.sum**(count) | *Shown as millisecond* | | **temporal.server.shardinfo.timer\_failover\_in\_progress.bucket**(count) | | **temporal.server.shardinfo.timer\_failover\_in\_progress.count**(count) | | **temporal.server.shardinfo.timer\_failover\_in\_progress.sum**(count) | | **temporal.server.shardinfo.timer\_failover\_latency.bucket**(count) | | **temporal.server.shardinfo.timer\_failover\_latency.count**(count) | | **temporal.server.shardinfo.timer\_failover\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.shardinfo.transfer.active.pending\_task.bucket**(count) | | **temporal.server.shardinfo.transfer.active.pending\_task.count**(count) | | **temporal.server.shardinfo.transfer.active.pending\_task.sum**(count) | | **temporal.server.shardinfo.transfer.diff.bucket**(count) | | **temporal.server.shardinfo.transfer.diff.count**(count) | | **temporal.server.shardinfo.transfer.diff.sum**(count) | | **temporal.server.shardinfo.transfer.failover\_in\_progress.bucket**(count) | | **temporal.server.shardinfo.transfer.failover\_in\_progress.count**(count) | | **temporal.server.shardinfo.transfer.failover\_in\_progress.sum**(count) | | **temporal.server.shardinfo.transfer.failover\_latency.bucket**(count) | | **temporal.server.shardinfo.transfer.failover\_latency.count**(count) | | **temporal.server.shardinfo.transfer.failover\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.shardinfo.transfer.lag.bucket**(count) | | **temporal.server.shardinfo.transfer.lag.count**(count) | | **temporal.server.shardinfo.transfer.lag.sum**(count) | | **temporal.server.shardinfo.transfer.standby.pending\_task.bucket**(count) | | **temporal.server.shardinfo.transfer.standby.pending\_task.count**(count) | | **temporal.server.shardinfo.transfer.standby.pending\_task.sum**(count) | | **temporal.server.shardinfo.visibility.lag.bucket**(count) | | **temporal.server.shardinfo.visibility.lag.count**(count) | | **temporal.server.shardinfo.visibility.lag.sum**(count) | | **temporal.server.shardinfo.visibility.pending\_task.bucket**(count) | | **temporal.server.shardinfo.visibility.pending\_task.count**(count) | | **temporal.server.shardinfo.visibility.pending\_task.sum**(count) | | **temporal.server.sharditem.acquisition\_latency.bucket**(count) | | **temporal.server.sharditem.acquisition\_latency.count**(count) | | **temporal.server.sharditem.acquisition\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.sharditem.created.count**(count) | | **temporal.server.sharditem.removed.count**(count) | | **temporal.server.signal\_external\_workflow\_command.count**(count) | | **temporal.server.signal\_info.bucket**(count) | | **temporal.server.signal\_info.count**(count) | | **temporal.server.signal\_info.size.bucket**(count) | | **temporal.server.signal\_info.size.count**(count) | | **temporal.server.signal\_info.size.sum**(count) | *Shown as byte* | | **temporal.server.signal\_info.sum**(count) | | **temporal.server.signal\_request\_id\_count.bucket**(count) | | **temporal.server.signal\_request\_id\_count.count**(count) | | **temporal.server.signal\_request\_id\_count.sum**(count) | | **temporal.server.signal\_request\_id\_size.bucket**(count) | | **temporal.server.signal\_request\_id\_size.count**(count) | | **temporal.server.signal\_request\_id\_size.sum**(count) | *Shown as byte* | | **temporal.server.signal\_with\_start\_skip\_delay\_count.count**(count) | | **temporal.server.speculative\_workflow\_task.commits.count**(count) | | **temporal.server.speculative\_workflow\_task.rollbacks.count**(count) | | **temporal.server.stale\_mutable\_state.count**(count) | | **temporal.server.stale\_replication\_events.count**(count) | | **temporal.server.start\_deployment.transition\_count.count**(count) | | **temporal.server.start\_timer\_command.count**(count) | | **temporal.server.start\_to\_close.timeout.count**(count) | | **temporal.server.start\_workflow\_request\_deduped.count**(count) | | **temporal.server.started.count**(count) | | **temporal.server.state\_machine\_timer.processing\_failures.count**(count) | | **temporal.server.state\_machine\_timer.skips.count**(count) | | **temporal.server.state\_transition.bucket**(count) | | **temporal.server.state\_transition.count**(count) | | **temporal.server.state\_transition.sum**(count) | | **temporal.server.stopped.count**(count) | | **temporal.server.sync\_throttle.count**(count) | | **temporal.server.syncmatch.latency.bucket**(count) | | **temporal.server.syncmatch.latency.count**(count) | | **temporal.server.syncmatch.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.syncshard.remote.count**(count) | | **temporal.server.syncshard.remote.failed.count**(count) | | **temporal.server.task.attempt.bucket**(count) | Distribution of the number of attempts on each task execution. | | **temporal.server.task.attempt.count**(count) | Count of the number of attempts on each task execution. | | **temporal.server.task.attempt.sum**(count) | Sum of the number of attempts on each task execution. | | **temporal.server.task.batch\_complete\_counter.count**(count) | | **temporal.server.task.bucket**(count) | | **temporal.server.task.count**(count) | | **temporal.server.task.deleted**(gauge) | | **temporal.server.task.dependency\_task\_not\_completed.count**(count) | | **temporal.server.task.dispatch\_latency.bucket**(count) | | **temporal.server.task.dispatch\_latency.count**(count) | | **temporal.server.task.dispatch\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.task.dlq\_failures.count**(count) | | **temporal.server.task.dlq\_latency.bucket**(count) | | **temporal.server.task.dlq\_latency.count**(count) | | **temporal.server.task.dlq\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.task.errors.corruption.count**(count) | | **temporal.server.task.errors.count**(count) | Number of task process errors. | | **temporal.server.task.errors.discarded.count**(count) | | **temporal.server.task.errors.internal.count**(count) | | **temporal.server.task.errors.limit\_exceeded\_counter.count**(count) | | **temporal.server.task.errors.namespace\_handover.count**(count) | | **temporal.server.task.errors.not\_active\_counter.count**(count) | | **temporal.server.task.errors.standby\_retry\_counter.count**(count) | | **temporal.server.task.errors.throttled.count**(count) | | **temporal.server.task.errors.version\_mismatch.count**(count) | | **temporal.server.task.errors.workflow\_busy.count**(count) | | **temporal.server.task.lag\_per\_tl**(gauge) | | **temporal.server.task.latency.bucket**(count) | Distribution of in-memory latencies across multiple attempts. | | **temporal.server.task.latency.count**(count) | Count of in-memory latencies across multiple attempts. | | **temporal.server.task.latency.load.bucket**(count) | Distribution of durations from task generation to task loading. | | **temporal.server.task.latency.load.count**(count) | Count of durations from task generation to task loading. | | **temporal.server.task.latency.load.sum**(count) | Sum of durations from task generation to task loading.*Shown as millisecond* | | **temporal.server.task.latency.processing.bucket**(count) | Distribution of task processing latencies per attempt. | | **temporal.server.task.latency.processing.count**(count) | Count of task processing latencies per attempt. | | **temporal.server.task.latency.processing.sum**(count) | Sum of task processing latencies per attempt.*Shown as millisecond* | | **temporal.server.task.latency.queue.bucket**(count) | Distribution of durations from when a task is fired to when the task is done. | | **temporal.server.task.latency.queue.count**(count) | Count of durations from when a task is fired to when the task is done. | | **temporal.server.task.latency.queue.sum**(count) | Sum of durations from when a task is fired to when the task is done.*Shown as millisecond* | | **temporal.server.task.latency.schedule.bucket**(count) | Distribution of durations from task submission to processing. | | **temporal.server.task.latency.schedule.count**(count) | Count of durations from task submission to processing. | | **temporal.server.task.latency.schedule.sum**(count) | Sum of durations from task submission to processing.*Shown as millisecond* | | **temporal.server.task.latency.sum**(count) | Sum of in-memory latencies across multiple attempts.*Shown as millisecond* | | **temporal.server.task.latency.user.bucket**(count) | | **temporal.server.task.latency.user.count**(count) | | **temporal.server.task.latency.user.sum**(count) | *Shown as millisecond* | | **temporal.server.task.processed**(gauge) | | **temporal.server.task.requests.count**(count) | Number of task process requests. | | **temporal.server.task.schedule\_to\_start\_latency.bucket**(count) | | **temporal.server.task.schedule\_to\_start\_latency.count**(count) | | **temporal.server.task.schedule\_to\_start\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.task.skipped.count**(count) | | **temporal.server.task.sum**(count) | | **temporal.server.task.terminal\_failures.count**(count) | | **temporal.server.task.throttled\_counter.count**(count) | | **temporal.server.task.write.latency.bucket**(count) | | **temporal.server.task.write.latency.count**(count) | | **temporal.server.task.write.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.task.write.throttle.count**(count) | | **temporal.server.task\_queue.started.count**(count) | | **temporal.server.task\_queue.stopped.count**(count) | | **temporal.server.task\_rescheduler.pending\_tasks.bucket**(count) | | **temporal.server.task\_rescheduler.pending\_tasks.count**(count) | | **temporal.server.task\_rescheduler.pending\_tasks.sum**(count) | | **temporal.server.task\_rewrites.count**(count) | | **temporal.server.task\_scheduler.throttled.count**(count) | | **temporal.server.taskqueue.deleted**(gauge) | | **temporal.server.taskqueue.outstanding**(gauge) | | **temporal.server.taskqueue.processed**(gauge) | | **temporal.server.tasks.per\_shardinfo\_update.bucket**(count) | | **temporal.server.tasks.per\_shardinfo\_update.count**(count) | | **temporal.server.tasks.per\_shardinfo\_update.sum**(count) | | **temporal.server.tasks\_expired.count**(count) | | **temporal.server.time\_between\_shardinfo\_update.bucket**(count) | | **temporal.server.time\_between\_shardinfo\_update.count**(count) | | **temporal.server.time\_between\_shardinfo\_update.sum**(count) | *Shown as millisecond* | | **temporal.server.timer\_info.bucket**(count) | | **temporal.server.timer\_info.count**(count) | | **temporal.server.timer\_info.size.bucket**(count) | | **temporal.server.timer\_info.size.count**(count) | | **temporal.server.timer\_info.size.sum**(count) | *Shown as byte* | | **temporal.server.timer\_info.sum**(count) | | **temporal.server.total\_activity\_count.bucket**(count) | | **temporal.server.total\_activity\_count.count**(count) | | **temporal.server.total\_activity\_count.sum**(count) | | **temporal.server.total\_child\_execution\_count.bucket**(count) | | **temporal.server.total\_child\_execution\_count.count**(count) | | **temporal.server.total\_child\_execution\_count.sum**(count) | | **temporal.server.total\_request\_cancel\_external\_count.bucket**(count) | | **temporal.server.total\_request\_cancel\_external\_count.count**(count) | | **temporal.server.total\_request\_cancel\_external\_count.sum**(count) | | **temporal.server.total\_signal\_count.bucket**(count) | | **temporal.server.total\_signal\_count.count**(count) | | **temporal.server.total\_signal\_count.sum**(count) | | **temporal.server.total\_signal\_external\_count.bucket**(count) | | **temporal.server.total\_signal\_external\_count.count**(count) | | **temporal.server.total\_signal\_external\_count.sum**(count) | | **temporal.server.total\_user\_timer\_count.bucket**(count) | | **temporal.server.total\_user\_timer\_count.count**(count) | | **temporal.server.total\_user\_timer\_count.sum**(count) | | **temporal.server.transfer\_task.missing\_event\_counter.count**(count) | | **temporal.server.unbuffer\_replication\_tasks.bucket**(count) | | **temporal.server.unbuffer\_replication\_tasks.count**(count) | | **temporal.server.unbuffer\_replication\_tasks.sum**(count) | *Shown as millisecond* | | **temporal.server.unknown\_build.polls.count**(count) | | **temporal.server.unknown\_build.tasks.count**(count) | | **temporal.server.update\_namespace.failures.count**(count) | | **temporal.server.upsert\_workflow\_search\_attributes\_command.count**(count) | | **temporal.server.utf8\_validation\_errors.count**(count) | | **temporal.server.verify.describe\_mutable\_state\_latency.bucket**(count) | | **temporal.server.verify.describe\_mutable\_state\_latency.count**(count) | | **temporal.server.verify.describe\_mutable\_state\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.verify.replication\_task.failed.count**(count) | | **temporal.server.verify.replication\_task.not\_found.count**(count) | | **temporal.server.verify.replication\_task.success.count**(count) | | **temporal.server.verify.replication\_tasks.latency.bucket**(count) | | **temporal.server.verify.replication\_tasks.latency.count**(count) | | **temporal.server.verify.replication\_tasks.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.version\_check.failed.count**(count) | | **temporal.server.version\_check.latency.bucket**(count) | | **temporal.server.version\_check.latency.count**(count) | | **temporal.server.version\_check.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.version\_check.request\_failed.count**(count) | | **temporal.server.version\_check.success.count**(count) | | **temporal.server.visibility.archiver.archive.non\_retryable\_error.count**(count) | | **temporal.server.visibility.archiver.archive.success.count**(count) | | **temporal.server.visibility.archiver.archive.transient\_error.count**(count) | | **temporal.server.visibility.persistence.error\_with\_type.count**(count) | | **temporal.server.visibility.persistence.errors.count**(count) | | **temporal.server.visibility.persistence.latency.bucket**(count) | | **temporal.server.visibility.persistence.latency.count**(count) | | **temporal.server.visibility.persistence.latency.sum**(count) | *Shown as millisecond* | | **temporal.server.visibility.persistence.requests.count**(count) | | **temporal.server.visibility.persistence.resource\_exhausted.count**(count) | | **temporal.server.wf\_too\_many\_pending.activities.count**(count) | | **temporal.server.wf\_too\_many\_pending.cancel\_requests.count**(count) | | **temporal.server.wf\_too\_many\_pending.child\_workflows.count**(count) | | **temporal.server.wf\_too\_many\_pending.external\_workflow\_signals.count**(count) | | **temporal.server.worker\_deployment\_created.count**(count) | | **temporal.server.worker\_deployment\_version\_created.count**(count) | | **temporal.server.worker\_deployment\_version\_created\_managed\_by\_controller.count**(count) | | **temporal.server.worker\_deployment\_version\_visibility\_query\_count.count**(count) | | **temporal.server.worker\_deployment\_versioning\_override\_count.count**(count) | | **temporal.server.worker\_not\_supports\_consistent\_query.count**(count) | | **temporal.server.workflow.cancel.count**(count) | | **temporal.server.workflow.cleanup.archive.count**(count) | | **temporal.server.workflow.cleanup.delete.count**(count) | | **temporal.server.workflow.cleanup.delete\_history\_inline.count**(count) | | **temporal.server.workflow.cleanup.nop.count**(count) | | **temporal.server.workflow.context\_cleared.count**(count) | | **temporal.server.workflow.continued\_as\_new.count**(count) | | **temporal.server.workflow.cron\_backoff.timer.count**(count) | | **temporal.server.workflow.eager\_execution.count**(count) | | **temporal.server.workflow.eager\_execution.denied.count**(count) | | **temporal.server.workflow.failed.count**(count) | | **temporal.server.workflow.retry\_backoff.timer.count**(count) | | **temporal.server.workflow.run\_timeout\_overrides.count**(count) | | **temporal.server.workflow.success.count**(count) | | **temporal.server.workflow.task.attempt.bucket**(count) | | **temporal.server.workflow.task.attempt.count**(count) | | **temporal.server.workflow.task.attempt.sum**(count) | | **temporal.server.workflow.task.heartbeat\_timeout.count**(count) | | **temporal.server.workflow.task.query\_latency.bucket**(count) | | **temporal.server.workflow.task.query\_latency.count**(count) | | **temporal.server.workflow.task.query\_latency.sum**(count) | *Shown as millisecond* | | **temporal.server.workflow.task.timeout\_overrides.count**(count) | | **temporal.server.workflow.terminate.count**(count) | | **temporal.server.workflow.timeout.count**(count) | | **temporal.server.workflow\_backoff\_timer.count**(count) | | **temporal.server.workflow\_delayed\_start.backoff\_timer.count**(count) | | **temporal.server.workflow\_query.failure\_count.count**(count) | | **temporal.server.workflow\_query.success\_count.count**(count) | | **temporal.server.workflow\_query.timeout\_count.count**(count) | | **temporal.server.workflow\_reset\_count.count**(count) | | **temporal.server.workflow\_tasks.completed.count**(count) | | **temporal.server.workflow\_update.aborted.count**(count) | | **temporal.server.workflow\_update.client\_timeout.count**(count) | | **temporal.server.workflow\_update.continue\_as\_new\_suggestions.count**(count) | | **temporal.server.workflow\_update.normal\_workflow\_task.count**(count) | | **temporal.server.workflow\_update.registry\_size.bucket**(count) | | **temporal.server.workflow\_update.registry\_size.count**(count) | | **temporal.server.workflow\_update.registry\_size.sum**(count) | *Shown as byte* | | **temporal.server.workflow\_update.registry\_size\_limited.count**(count) | | **temporal.server.workflow\_update.request\_rate\_limited.count**(count) | | **temporal.server.workflow\_update.request\_too\_many.count**(count) | | **temporal.server.workflow\_update.sent\_to\_worker.count**(count) | | **temporal.server.workflow\_update.sent\_to\_worker\_again.count**(count) | | **temporal.server.workflow\_update.server\_timeout.count**(count) | | **temporal.server.workflow\_update.speculative\_workflow\_task.count**(count) | | **temporal.server.workflow\_update.wait\_stage\_accepted.count**(count) | | **temporal.server.workflow\_update.wait\_stage\_completed.count**(count) | ### Events{% #events %} The Temporal integration does not include any events. ### Service Checks{% #service-checks %} **temporal.server.openmetrics.health** Returns `CRITICAL` if the check cannot access the Prometheus metrics endpoint of the Temporal instance. *Statuses: ok, critical* ### Logs{% #logs %} The Temporal integration can collect logs from the Temporal Cluster and forward them to Datadog. ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/). ## Further Reading{% #further-reading %} - [Monitor the health of your Temporal Server with Datadog](https://www.datadoghq.com/blog/temporal-server-integration/)