--- title: GitLab Runners description: Track all the metrics from your GitLab runners with Datadog. breadcrumbs: Docs > Integrations > GitLab Runners --- # GitLab Runners Supported OS Integration version7.5.0 ## Overview{% #overview %} Integration that allows to: - Visualize and monitor metrics collected with GitLab Runners through Prometheus - Validate that the GitLab Runner can connect to GitLab For more information about the GitLab Runner and its integration with Prometheus, see the [GitLab Runner documentation](https://docs.gitlab.com/runner/monitoring/). **Minimum Agent version:** 6.0.0 ## Setup{% #setup %} Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates](https://docs.datadoghq.com/agent/kubernetes/integrations.md) for guidance on applying these instructions. ### Installation{% #installation %} The GitLab Runner check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package, so you don't need to install anything else on your GitLab servers. ### Configuration{% #configuration %} Edit the `gitlab_runner.d/conf.yaml` file, in the `conf.d/` folder at the root of your [Agent's configuration directory](https://docs.datadoghq.com/agent/guide/agent-configuration-files.md#agent-configuration-directory), to point to the Runner's Prometheus metrics endpoint and to the GitLab master to have a service check. See the [sample gitlab_runner.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/gitlab_runner/datadog_checks/gitlab_runner/data/conf.yaml.example) for all available configuration options. The `allowed_metrics` item in the `init_config` section allows you to specify the metrics that should be extracted. Some metrics should be reported as `rate`, for example: `ci_runner_errors`. ### Validation{% #validation %} [Run the Agent's `status` subcommand](https://docs.datadoghq.com/agent/guide/agent-commands.md#agent-status-and-information) and look for `gitlab_runner` under the Checks section. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **gitlab\_runner.ci\_docker\_machines\_provider\_machine\_creation\_duration\_seconds.count**(gauge) | The count of Docker machine creation time. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_docker\_machines\_provider\_machine\_creation\_duration\_seconds.sum**(gauge) | The sum of Docker machine creation time. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_docker\_machines\_provider\_machine\_creation\_duration\_seconds\_bucket**(gauge) | A histogram of Docker machine creation time. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_docker\_machines\_provider\_machine\_creation\_duration\_seconds\_count**(gauge) | The count of Docker machine creation time. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_docker\_machines\_provider\_machine\_creation\_duration\_seconds\_sum**(gauge) | The sum of Docker machine creation time. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_docker\_machines\_provider\_machine\_states**(gauge) | The current number of CI machines per state in this provider. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_runner\_builds**(gauge) | The current number of running builds. Applies to GitLab Runner < 1.11.0 | | **gitlab\_runner.ci\_runner\_errors**(count) | The number of caught errors. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_runner\_version\_info**(gauge) | A metric with a constant '1' value labeled by different build stats fields. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_ssh\_docker\_machines\_provider\_machine\_creation\_duration\_seconds.count**(gauge) | The count of SSH Docker machine creation time. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_ssh\_docker\_machines\_provider\_machine\_creation\_duration\_seconds.sum**(gauge) | The sum of SSH Docker machine creation time. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_ssh\_docker\_machines\_provider\_machine\_creation\_duration\_seconds\_bucket**(gauge) | A histogram of SSH Docker machine creation time. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_ssh\_docker\_machines\_provider\_machine\_creation\_duration\_seconds\_count**(gauge) | The count of SSH Docker machine creation time. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_ssh\_docker\_machines\_provider\_machine\_creation\_duration\_seconds\_sum**(gauge) | The sum of SSH Docker machine creation time. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.ci\_ssh\_docker\_machines\_provider\_machine\_states**(gauge) | The current number of SSH machines per state in this ssh provider. Applies to GitLab Runner < 1.11.0*Shown as request* | | **gitlab\_runner.gitlab\_runner\_api\_request\_duration\_seconds.count**(gauge) | The count of API request duration observations. | | **gitlab\_runner.gitlab\_runner\_api\_request\_duration\_seconds.sum**(gauge) | The sum of API request durations in seconds.*Shown as second* | | **gitlab\_runner.gitlab\_runner\_autoscaling\_machine\_creation\_duration\_seconds**(gauge) | A histogram of Docker machine creation time. Applies to GitLab Runner 1.11.0+*Shown as request* | | **gitlab\_runner.gitlab\_runner\_autoscaling\_machine\_states**(gauge) | The current number of machines per state in this provider. Applies to GitLab Runner 1.11.0+*Shown as request* | | **gitlab\_runner.gitlab\_runner\_errors\_total**(count) | The number of caught errors. Applies to GitLab Runner 1.11.0+*Shown as request* | | **gitlab\_runner.gitlab\_runner\_job\_queue\_duration\_seconds.count**(gauge) | Histogram bucket and count of job queue duration observations. | | **gitlab\_runner.gitlab\_runner\_job\_queue\_duration\_seconds.sum**(gauge) | Sum of job queue duration in seconds.*Shown as second* | | **gitlab\_runner.gitlab\_runner\_jobs**(gauge) | The current number of running builds. Applies to GitLab Runner 1.11.0+ | | **gitlab\_runner.gitlab\_runner\_jobs\_total**(count) | The total number of jobs executed. | | **gitlab\_runner.gitlab\_runner\_version\_info**(gauge) | A metric with a constant '1' value labeled by different build stats fields. Applies to GitLab Runner 1.11.0+*Shown as request* | | **gitlab\_runner.go\_gc\_duration\_seconds**(gauge) | A summary of the GC invocation durations*Shown as request* | | **gitlab\_runner.go\_gc\_duration\_seconds.count**(gauge) | The count of the GC invocation durations*Shown as request* | | **gitlab\_runner.go\_gc\_duration\_seconds.quantile**(gauge) | A summary quantile of the GC invocation durations*Shown as request* | | **gitlab\_runner.go\_gc\_duration\_seconds.sum**(gauge) | The sum of the GC invocation durations*Shown as request* | | **gitlab\_runner.go\_gc\_duration\_seconds\_count**(gauge) | The count of the GC invocation durations*Shown as request* | | **gitlab\_runner.go\_gc\_duration\_seconds\_sum**(gauge) | The sum of the GC invocation durations*Shown as request* | | **gitlab\_runner.go\_goroutines**(gauge) | The number of goroutines that currently exist*Shown as request* | | **gitlab\_runner.go\_memstats\_alloc\_bytes**(gauge) | The number of bytes allocated and still in use*Shown as byte* | | **gitlab\_runner.go\_memstats\_alloc\_bytes\_total**(count) | The total number of bytes allocated*Shown as byte* | | **gitlab\_runner.go\_memstats\_buck\_hash\_sys\_bytes**(gauge) | The number of bytes used by the profiling bucket hash table*Shown as byte* | | **gitlab\_runner.go\_memstats\_frees\_total**(count) | The total number of frees*Shown as request* | | **gitlab\_runner.go\_memstats\_gc\_sys\_bytes**(gauge) | The number of bytes used for garbage collection system metadata*Shown as byte* | | **gitlab\_runner.go\_memstats\_heap\_alloc\_bytes**(gauge) | The number of heap bytes allocated and still in use*Shown as byte* | | **gitlab\_runner.go\_memstats\_heap\_idle\_bytes**(gauge) | The number of heap bytes waiting to be used*Shown as byte* | | **gitlab\_runner.go\_memstats\_heap\_inuse\_bytes**(gauge) | The number of heap bytes that are in use*Shown as byte* | | **gitlab\_runner.go\_memstats\_heap\_objects**(gauge) | The number of allocated objects*Shown as request* | | **gitlab\_runner.go\_memstats\_heap\_released\_bytes\_total**(count) | The total number of heap bytes released to OS*Shown as byte* | | **gitlab\_runner.go\_memstats\_heap\_sys\_bytes**(gauge) | The number of heap bytes obtained from system*Shown as byte* | | **gitlab\_runner.go\_memstats\_last\_gc\_time\_seconds**(gauge) | The number of seconds since 1970 of last garbage collection*Shown as request* | | **gitlab\_runner.go\_memstats\_lookups\_total**(count) | The total number of pointer lookups*Shown as request* | | **gitlab\_runner.go\_memstats\_mallocs\_total**(count) | The total number of mallocs*Shown as request* | | **gitlab\_runner.go\_memstats\_mcache\_inuse\_bytes**(gauge) | The number of bytes in use by mcache structures*Shown as byte* | | **gitlab\_runner.go\_memstats\_mcache\_sys\_bytes**(gauge) | The number of bytes used for mcache structures obtained from system*Shown as byte* | | **gitlab\_runner.go\_memstats\_mspan\_inuse\_bytes**(gauge) | The number of bytes in use by mspan structures*Shown as byte* | | **gitlab\_runner.go\_memstats\_mspan\_sys\_bytes**(gauge) | The number of bytes used for mspan structures obtained from system*Shown as byte* | | **gitlab\_runner.go\_memstats\_next\_gc\_bytes**(gauge) | The number of heap bytes when next garbage collection will take place*Shown as byte* | | **gitlab\_runner.go\_memstats\_other\_sys\_bytes**(gauge) | The number of bytes used for other system allocations*Shown as byte* | | **gitlab\_runner.go\_memstats\_stack\_inuse\_bytes**(gauge) | The number of bytes in use by the stack allocator*Shown as byte* | | **gitlab\_runner.go\_memstats\_stack\_sys\_bytes**(gauge) | The number of bytes obtained from system for stack allocator*Shown as byte* | | **gitlab\_runner.go\_memstats\_sys\_bytes**(gauge) | The number of bytes obtained by system. Sum of all system allocations*Shown as byte* | | **gitlab\_runner.process\_cpu\_seconds\_total**(count) | The total user and system CPU time spent in seconds*Shown as request* | | **gitlab\_runner.process\_max\_fds**(gauge) | The maximum number of open file descriptors*Shown as request* | | **gitlab\_runner.process\_open\_fds**(gauge) | The number of open file descriptors*Shown as request* | | **gitlab\_runner.process\_resident\_memory\_bytes**(gauge) | The resident memory size in bytes*Shown as byte* | | **gitlab\_runner.process\_start\_time\_seconds**(gauge) | The start time of the process since unix epoch in seconds*Shown as request* | | **gitlab\_runner.process\_virtual\_memory\_bytes**(gauge) | The virtual memory size in bytes*Shown as byte* | ### Log collection{% #log-collection %} 1. In your `gitlab_runner` [configuration file](https://docs.gitlab.com/runner/configuration/advanced-configuration.html), change the log format to `json` (*Available for GitLab Runner versions >=11.4.0* ): ```toml log_format = "json" ``` 1. Collecting logs is disabled by default in the Datadog Agent, you need to enable it in `datadog.yaml`: ```yaml logs_enabled: true ``` 1. Add the `dd-agent` user to the `systemd-journal` group by running: ```text usermod -a -G systemd-journal dd-agent ``` 1. Add this configuration block to your `gitlab_runner.d/conf.yaml` file to start collecting your GitLab Runner Logs: ```yaml logs: - type: journald source: gitlab-runner ``` See the [sample gitlab_runner.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/gitlab_runner/datadog_checks/gitlab_runner/data/conf.yaml.example) for all available configuration options. 1. [Restart the Agent](https://docs.datadoghq.com/agent/guide/agent-commands.md#start-stop-and-restart-the-agent). ### Events{% #events %} The GitLab Runner check does not include any events. ### Service Checks{% #service-checks %} The GitLab Runner check provides a service check to confirm that the Runner can talk to the GitLab master and another one to ensure that the local Prometheus endpoint is available. ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).