GitLab Runners

Docs > Integrations > GitLab Runners

Supported OS Linux Mac OS Windows

Integration version6.1.0

GitLab Runner Integration

Overview

Integration that allows to:

Visualize and monitor metrics collected with GitLab Runners through Prometheus
Validate that the GitLab Runner can connect to GitLab

For more information about the GitLab Runner and its integration with Prometheus, see the GitLab Runner documentation.

Setup

Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the Autodiscovery Integration Templates for guidance on applying these instructions.

Installation

The GitLab Runner check is included in the Datadog Agent package, so you don’t need to install anything else on your GitLab servers.

Configuration

Edit the gitlab_runner.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory, to point to the Runner’s Prometheus metrics endpoint and to the GitLab master to have a service check. See the sample gitlab_runner.d/conf.yaml for all available configuration options.

The allowed_metrics item in the init_config section allows you to specify the metrics that should be extracted. Some metrics should be reported as rate, for example: ci_runner_errors.

Validation

Run the Agent’s status subcommand and look for gitlab_runner under the Checks section.

Data Collected

Metrics


gitlab_runner.ci_docker_machines_provider_machine_creation_duration_seconds_bucket (gauge)	A histogram of Docker machine creation time. Applies to GitLab Runner < 1.11.0 Shown as request
gitlab_runner.ci_docker_machines_provider_machine_creation_duration_seconds_count (gauge)	The count of Docker machine creation time. Applies to GitLab Runner < 1.11.0 Shown as request
gitlab_runner.ci_docker_machines_provider_machine_creation_duration_seconds_sum (gauge)	The sum of Docker machine creation time. Applies to GitLab Runner < 1.11.0 Shown as request
gitlab_runner.ci_docker_machines_provider_machine_states (gauge)	The current number of CI machines per state in this provider. Applies to GitLab Runner < 1.11.0 Shown as request
gitlab_runner.ci_runner_builds (gauge)	The current number of running builds. Applies to GitLab Runner < 1.11.0
gitlab_runner.ci_runner_errors (count)	The number of caught errors. Applies to GitLab Runner < 1.11.0 Shown as request
gitlab_runner.ci_runner_version_info (gauge)	A metric with a constant ‘1’ value labeled by different build stats fields. Applies to GitLab Runner < 1.11.0 Shown as request
gitlab_runner.ci_ssh_docker_machines_provider_machine_creation_duration_seconds_bucket (gauge)	A histogram of SSH Docker machine creation time. Applies to GitLab Runner < 1.11.0 Shown as request
gitlab_runner.ci_ssh_docker_machines_provider_machine_creation_duration_seconds_count (gauge)	The count of SSH Docker machine creation time. Applies to GitLab Runner < 1.11.0 Shown as request
gitlab_runner.ci_ssh_docker_machines_provider_machine_creation_duration_seconds_sum (gauge)	The sum of SSH Docker machine creation time. Applies to GitLab Runner < 1.11.0 Shown as request
gitlab_runner.ci_ssh_docker_machines_provider_machine_states (gauge)	The current number of SSH machines per state in this ssh provider. Applies to GitLab Runner < 1.11.0 Shown as request
gitlab_runner.gitlab_runner_autoscaling_machine_creation_duration_seconds (gauge)	A histogram of Docker machine creation time. Applies to GitLab Runner 1.11.0+ Shown as request
gitlab_runner.gitlab_runner_autoscaling_machine_states (gauge)	The current number of machines per state in this provider. Applies to GitLab Runner 1.11.0+ Shown as request
gitlab_runner.gitlab_runner_errors_total (count)	The number of caught errors. Applies to GitLab Runner 1.11.0+ Shown as request
gitlab_runner.gitlab_runner_jobs (gauge)	The current number of running builds. Applies to GitLab Runner 1.11.0+
gitlab_runner.gitlab_runner_jobs_total (count)	The total number of jobs executed.
gitlab_runner.gitlab_runner_version_info (gauge)	A metric with a constant ‘1’ value labeled by different build stats fields. Applies to GitLab Runner 1.11.0+ Shown as request
gitlab_runner.go_gc_duration_seconds (gauge)	A summary of the GC invocation durations Shown as request
gitlab_runner.go_gc_duration_seconds_count (gauge)	The count of the GC invocation durations Shown as request
gitlab_runner.go_gc_duration_seconds_sum (gauge)	The sum of the GC invocation durations Shown as request
gitlab_runner.go_goroutines (gauge)	The number of goroutines that currently exist Shown as request
gitlab_runner.go_memstats_alloc_bytes (gauge)	The number of bytes allocated and still in use Shown as byte
gitlab_runner.go_memstats_alloc_bytes_total (count)	The total number of bytes allocated Shown as byte
gitlab_runner.go_memstats_buck_hash_sys_bytes (gauge)	The number of bytes used by the profiling bucket hash table Shown as byte
gitlab_runner.go_memstats_frees_total (count)	The total number of frees Shown as request
gitlab_runner.go_memstats_gc_sys_bytes (gauge)	The number of bytes used for garbage collection system metadata Shown as byte
gitlab_runner.go_memstats_heap_alloc_bytes (gauge)	The number of heap bytes allocated and still in use Shown as byte
gitlab_runner.go_memstats_heap_idle_bytes (gauge)	The number of heap bytes waiting to be used Shown as byte
gitlab_runner.go_memstats_heap_inuse_bytes (gauge)	The number of heap bytes that are in use Shown as byte
gitlab_runner.go_memstats_heap_objects (gauge)	The number of allocated objects Shown as request
gitlab_runner.go_memstats_heap_released_bytes_total (count)	The total number of heap bytes released to OS Shown as byte
gitlab_runner.go_memstats_heap_sys_bytes (gauge)	The number of heap bytes obtained from system Shown as byte
gitlab_runner.go_memstats_last_gc_time_seconds (gauge)	The number of seconds since 1970 of last garbage collection Shown as request
gitlab_runner.go_memstats_lookups_total (count)	The total number of pointer lookups Shown as request
gitlab_runner.go_memstats_mallocs_total (count)	The total number of mallocs Shown as request
gitlab_runner.go_memstats_mcache_inuse_bytes (gauge)	The number of bytes in use by mcache structures Shown as byte
gitlab_runner.go_memstats_mcache_sys_bytes (gauge)	The number of bytes used for mcache structures obtained from system Shown as byte
gitlab_runner.go_memstats_mspan_inuse_bytes (gauge)	The number of bytes in use by mspan structures Shown as byte
gitlab_runner.go_memstats_mspan_sys_bytes (gauge)	The number of bytes used for mspan structures obtained from system Shown as byte
gitlab_runner.go_memstats_next_gc_bytes (gauge)	The number of heap bytes when next garbage collection will take place Shown as byte
gitlab_runner.go_memstats_other_sys_bytes (gauge)	The number of bytes used for other system allocations Shown as byte
gitlab_runner.go_memstats_stack_inuse_bytes (gauge)	The number of bytes in use by the stack allocator Shown as byte
gitlab_runner.go_memstats_stack_sys_bytes (gauge)	The number of bytes obtained from system for stack allocator Shown as byte
gitlab_runner.go_memstats_sys_bytes (gauge)	The number of bytes obtained by system. Sum of all system allocations Shown as byte
gitlab_runner.process_cpu_seconds_total (count)	The total user and system CPU time spent in seconds Shown as request
gitlab_runner.process_max_fds (gauge)	The maximum number of open file descriptors Shown as request
gitlab_runner.process_open_fds (gauge)	The number of open file descriptors Shown as request
gitlab_runner.process_resident_memory_bytes (gauge)	The resident memory size in bytes Shown as byte
gitlab_runner.process_start_time_seconds (gauge)	The start time of the process since unix epoch in seconds Shown as request
gitlab_runner.process_virtual_memory_bytes (gauge)	The virtual memory size in bytes Shown as byte

Log collection

In your gitlab_runner configuration file, change the log format to json (Available for GitLab Runner versions >=11.4.0 ):
```
log_format = "json"
```
Collecting logs is disabled by default in the Datadog Agent, you need to enable it in datadog.yaml:
```
logs_enabled: true
```
Add the dd-agent user to the systemd-journal group by running:
```
usermod -a -G systemd-journal dd-agent
```
Add this configuration block to your gitlab_runner.d/conf.yaml file to start collecting your GitLab Runner Logs:
```
logs:
  - type: journald
    source: gitlab-runner
```
See the sample gitlab_runner.d/conf.yaml for all available configuration options.
Restart the Agent.

Events

The GitLab Runner check does not include any events.

Service Checks

The GitLab Runner check provides a service check to confirm that the Runner can talk to the GitLab master and another one to ensure that the local Prometheus endpoint is available.

Troubleshooting

Need help? Contact Datadog support.