Overview
Setting up Network Path involves configuring your environment to monitor and trace the network routes between your services and endpoints. This helps identify bottlenecks, latency issues, and potential points of failure in your network infrastructure. Network Path allows you to manually configure individual network paths or automatically discover them, depending on your needs.
Note: If your network configuration restricts outbound traffic, follow the setup instructions on the Agent proxy configuration documentation.
Setup
Monitor individual paths
You can monitor specific network paths by defining them in the Agent configuration file located at /etc/datadog-agent/conf.d/network_path.d/conf.yaml
.
To get started, copy the example configuration, remove the .example
extension, and update it with your desired settings, or use one of the environment-specific configurations below. For performance optimization, see increase the number of workers.
Agent v7.59+
is required.
Enable the system-probe
traceroute module in /etc/datadog-agent/system-probe.yaml
by adding the following:
traceroute:
enabled: true
Enable network_path
to monitor new destinations from this Agent by creating or editing the /etc/datadog-agent/conf.d/network_path.d/conf.yaml
file:
init_config:
min_collection_interval: 60 # in seconds, default 60 seconds
instances:
# configure the endpoints you want to monitor, one check instance per endpoint
# warning: Do not set the port when using UDP. Setting the port when using UDP can cause traceroute calls to fail and falsely report an unreachable destination.
- hostname: api.datadoghq.eu # endpoint hostname or IP
protocol: TCP
port: 443
tags:
- "tag_key:tag_value"
- "tag_key2:tag_value2"
min_collection_interval: 120 # set min_collection_interval at the instance level
## optional configs:
# max_ttl: 30 # max traderoute TTL, default is 30
# timeout: 1000 # timeout in milliseconds per hop, default is 1s
# more endpoints
- hostname: 1.1.1.1 # endpoint hostname or IP
protocol: UDP
tags:
- "tag_key:tag_value"
- "tag_key2:tag_value2"
Restart the Agent after making these configuration changes to start seeing network paths.
Agent v7.61+
is required.
Note: Windows only supports TCP traceroutes.
Enable the system-probe
traceroute module in %ProgramData%\Datadog\system-probe.yaml
by adding the following:
traceroute:
enabled: true
Enable network_path
to monitor new destinations from this Agent by creating or editing the %ProgramData%\Datadog\conf.d\network_path.d\conf.yaml
file:
init_config:
min_collection_interval: 60 # in seconds, default 60 seconds
instances:
# configure the endpoints you want to monitor, one check instance per endpoint
# warning: Do not set the port when using UDP. Setting the port when using UDP can cause traceroute calls to fail and falsely report an unreachable destination.
- hostname: api.datadoghq.eu # endpoint hostname or IP
protocol: TCP
port: 443
tags:
- "tag_key:tag_value"
- "tag_key2:tag_value2"
min_collection_interval: 120 # set min_collection_interval at the instance level
## optional configs:
# max_ttl: 30 # max traderoute TTL, default is 30
# timeout: 1000 # timeout in milliseconds per hop, default is 1s
# more endpoints
- hostname: 1.1.1.1 # endpoint hostname or IP
protocol: TCP
tags:
- "tag_key:tag_value"
- "tag_key2:tag_value2"
Restart the Agent after making these configuration changes to start seeing network paths.
Limitations:
In Windows environments, the Agent uses UDP by default to monitor individual paths. If the protocol is not specified in the configuration, the Agent attempts a UDP traceroute, and any errors are logged. To work around this, ensure the protocol is set to TCP. For example:
init_config:
min_collection_interval: 60 # in seconds, default 60 seconds
instances:
- hostname: api.datadoghq.eu # endpoint hostname or IP
protocol: TCP
port: 443 # optional port number, default is 80
In Windows Client OS environments, raw packets are not supported. To work around this, set protocol: TCP
and tcp_method: syn_socket
. Agent v7.67+
and Windows version 2004 (10.0; Build 19041) or later are required. For example:
init_config:
min_collection_interval: 60 # in seconds, default 60 seconds
instances:
- hostname: api.datadoghq.eu # endpoint hostname or IP
protocol: TCP
port: 443 # optional port number, default is 80
tcp_method: syn_socket
Agent v7.59+
is required.
To enable Network Path with Kubernetes using Helm, add the following to your values.yaml
file.
datadog:
traceroute:
enabled: true
confd:
network_path.yaml: |-
init_config:
min_collection_interval: 60 # in seconds, default 60 seconds
instances:
# configure the endpoints you want to monitor, one check instance per endpoint
# warning: Do not set the port when using UDP. Setting the port when using UDP can cause traceroute calls to fail and falsely report an unreachable destination.
- hostname: api.datadoghq.eu # endpoint hostname or IP
protocol: TCP
port: 443
tags:
- "tag_key:tag_value"
- "tag_key2:tag_value2"
min_collection_interval: 120 # set min_collection_interval at the instance level
## optional configs:
# max_ttl: 30 # max traderoute TTL, default is 30
# timeout: 1000 # timeout in milliseconds per hop, default is 1s
# more endpoints
- hostname: 1.1.1.1 # endpoint hostname or IP
protocol: UDP
tags:
- "tag_key:tag_value"
- "tag_key2:tag_value2"
Datadog Autodiscovery allows you to enable Network Path on a per-service basis through Kubernetes annotations.
Enable the traceroute module in the Datadog values.yaml
file, which the Network Path integration depends on.
datadog:
traceroute:
enabled: true
After the module is enabled, Datadog automatically detects Network Path annotations added to your Kubernetes pod. For more information, see Kubernetes and Integrations.
apiVersion: v1
kind: Pod
# (...)
metadata:
name: '<POD_NAME>'
annotations:
ad.datadoghq.com/<CONTAINER_NAME>.checks: |
{
"network_path": {
"init_config": {
"min_collection_interval": 300
},
"instances": [
{
"protocol": "TCP",
"port": 443,
"source_service": "<CONTAINER_NAME>",
"tags": [
"tag_key:tag_value",
"tag_key2:tag_value2"
],
"hostname": "api.datadoghq.eu"
},
{
"protocol": "UDP",
"source_service": "<CONTAINER_NAME>",
"tags": [
"tag_key:tag_value",
"tag_key2:tag_value2"
],
"hostname": "1.1.1.1"
},
]
}
}
# (...)
spec:
containers:
- name: '<CONTAINER_NAME>'
# (...)
If you define pods indirectly (with deployments, ReplicaSets, or ReplicationControllers), add pod annotations under spec.template.metadata
.
Increase the number of workers
Network Path monitoring for individual paths runs as an Agent Integration. The number of concurrent workers is controlled by the check_runners
setting in the datadog.yaml
file.
To increase the number of workers, add the following configuration to your datadog.yaml
file:
## @param check_runners - integer - optional - default: 4
## @env DD_CHECK_RUNNERS - integer - optional - default: 4
## The `check_runners` refers to the number of concurrent check runners available for check instance execution.
## The scheduler attempts to spread the instances over the collection interval and will _at most_ be
## running the number of check runners instances concurrently.
##
## The level of concurrency has effects on the Agent's: RSS memory, CPU load, resource contention overhead, etc.
#
check_runners: <NUMBER_OF_WORKERS>
Network traffic paths (experimental)
Prerequisites: CNM must be enabled.
Note: Network traffic paths is experimental and is not yet stable. Do not deploy network traffic paths widely in a production environment.
Configure network traffic paths to allow the Agent to automatically discover and monitor network paths based on actual network traffic, eliminating the need to manually configure individual endpoints. See exclude CIDR ranges to filter specific network ranges.
Enabling Network Path to automatically detect paths can generate a significant number of logs, particularly when monitoring network paths across a large number of hosts.
Agent v7.59+
is required.
Enable the system-probe
traceroute module in /etc/datadog-agent/system-probe.yaml
by adding the following:
traceroute:
enabled: true
Enable network_path
to monitor CNM connections by creating or editing the /etc/datadog-agent/datadog.yaml
file:
network_path:
connections_monitoring:
enabled: true
# collector:
# workers: <NUMBER OF WORKERS> # default 4
For full configuration details, reference the example config, or use the following:
network_path:
connections_monitoring:
## @param enabled - bool - required - default:false
## Enable network path collection
#
enabled: true
collector:
## @param workers - int - optional - default:4
## Number of workers that can collect paths in parallel
## Recommendation: leave at default
#
# workers: <NUMBER OF WORKERS> # default 4
#@env DD_NETWORK_PATH_COLLECTOR_PATHTEST_INTERVAL - integer - optional - default: 10m
# The `pathtest_interval` refers to the traceroute run interval for monitored connections.
# pathtest_interval: 10m
# @param pathtest_ttl - integer - optional - default: 35m
# @env DD_NETWORK_PATH_COLLECTOR_PATHTEST_TTL - integer - optional - default: 35m
# The `pathtest_ttl` refers to the duration (time-to-live) a connection will be monitored when it's not seen anymore.
# The TTL is reset each time the connection is seen again.
# pathtest_ttl: 35m
Restart the Agent after making these configuration changes to start seeing network paths.
Agent v7.61+
is required.
Enable the system-probe
traceroute module in %ProgramData%\Datadog\system-probe.yaml
by adding the following:
traceroute:
enabled: true
Enable network_path
to monitor CNM connections by creating or editing the %ProgramData%\Datadog\datadog.yaml
file:
network_path:
connections_monitoring:
enabled: true
# collector:
# workers: <NUMBER OF WORKERS> # default 4
For full configuration details, reference the example config, or use the following:
network_path:
connections_monitoring:
## @param enabled - bool - required - default:false
## Enable network path collection
#
enabled: true
collector:
## @param workers - int - optional - default:4
## Number of workers that can collect paths in parallel
## Recommendation: leave at default
#
# workers: <NUMBER OF WORKERS> # default 4
#@env DD_NETWORK_PATH_COLLECTOR_PATHTEST_INTERVAL - integer - optional - default: 10m
# The `pathtest_interval` refers to the traceroute run interval for monitored connections.
# pathtest_interval: 10m
# @param pathtest_ttl - integer - optional - default: 35m
# @env DD_NETWORK_PATH_COLLECTOR_PATHTEST_TTL - integer - optional - default: 35m
# The `pathtest_ttl` refers to the duration (time-to-live) a connection will be monitored when it's not seen anymore.
# The TTL is reset each time the connection is seen again.
# pathtest_ttl: 35m
Restart the Agent after making these configuration changes to start seeing network paths.
Agent v7.59+
is required.
To enable Network Path with Kubernetes using Helm, add the following to your values.yaml
file.
Note: Helm chart v3.124.0+ is required. For more information, reference the Datadog Helm Chart documentation and the documentation for Kubernetes and Integrations.
datadog:
networkPath:
connectionsMonitoring:
enabled: true
## Set to true to enable the Traceroute Module of the System Probe
traceroute:
enabled: true
## @param collector - custom object - optional
## Configuration related to Network Path Collector.
#
collector:
## @param workers - integer - optional - default: 4
## @env DD_WORKERS - integer - optional - default: 4
## The `workers` refers to the number of concurrent workers available for network path execution.
#
# workers: 4
#@env DD_NETWORK_PATH_COLLECTOR_PATHTEST_INTERVAL - integer - optional - default: 10m
# The `pathtest_interval` refers to the traceroute run interval for monitored connections.
# pathtest_interval: 10m
# @param pathtest_ttl - integer - optional - default: 35m
# @env DD_NETWORK_PATH_COLLECTOR_PATHTEST_TTL - integer - optional - default: 35m
# The `pathtest_ttl` refers to the duration (time-to-live) a connection will be monitored when it's not seen anymore.
# The TTL is reset each time the connection is seen again.
# pathtest_ttl: 35m
Exclude CIDR ranges
Classless Inter-Domain Routing (CIDR) ranges define blocks of IP addresses using network prefixes. You may want to exclude certain CIDR ranges from network traffic paths to:
- Reduce monitoring overhead for internal networks
- Focus on external traffic patterns
- Exclude known infrastructure ranges that don’t require monitoring
To exclude specific CIDR ranges from network traffic paths, configure the following in your /etc/datadog-agent/datadog.yaml
file:
network_path:
connections_monitoring:
enabled: true # enable network path collection
collector:
source_excludes:
"10.0.0.0/8":
- "*" # all ports
dest_excludes:
"10.0.0.0/8":
- "*" # all ports
"8.8.8.8":
- "53" # dns
- "33434-33464" # traceroute range
Troubleshooting
Use the following guidelines to troubleshoot issues with Network Path. If you need additional help, contact Datadog Support.
No Network Path data in the UI
If no data appears in the Network Path UI, the feature may not be fully enabled. Network Path requires the following:
The traceroute module must be enabled in your system-probe.yaml
file:
traceroute:
enabled: true
At least one Network Path feature must be active, such as:
Error: status code: 404
If you encounter an error like the following:
Error: failed to trace path: traceroute request failed: Probe Path <path>, url: <url>, status code: 404
- This indicates that the traceroute module is not enabled. Ensure the traceroute module is enabled in your
system-probe.yaml
file.
Further Reading
Additional helpful documentation, links, and articles: