Run Synthetic Tests from Private Locations

Overview

Private locations allow you to monitor internal-facing applications or any private endpoints that aren’t accessible from the public internet. They can also be used to:

  • Create custom Synthetic locations in areas that are mission-critical to your business.
  • Verify application performance in your internal CI environment before you release new features to production with Continuous Testing and CI/CD.
  • Compare application performance from both inside and outside your internal network.
Architecture diagram of how a private location works in Synthetic Monitoring

Private locations come as Docker containers or Windows services that you can install inside of your private network. After you create and install a private location, you can assign Synthetic tests to it, like with any managed location.

Your private location worker pulls your test configurations from Datadog’s servers using HTTPS, executes the test on a schedule or on-demand, and returns the test results to Datadog’s servers. You can then visualize your private locations test results in a completely identical manner to how you would visualize tests running from managed locations:

Assign a Synthetic test to a private location

Prerequisites

To use private locations for Continuous Testing tests, you need v1.27.0 or later.

Private locations are Docker containers that you can install anywhere inside your private network. You can access the private location worker image on Google Container Registry. It can run on a Linux-based OS or Windows OS if the Docker engine is available on your host and can run in Linux containers mode.

Private locations are Windows services that you can install anywhere inside your private network using an MSI file. Run this file from the virtual or physical machine that you would like to install the private location on.

This machine’s requirements are listed in the table below. PowerShell scripting must be enabled on the machine where you are installing the private location worker.

SystemRequirements
OSWindows Server 2016, Windows Server 2019, or Windows 10.
RAM4GB minimum. 8GB recommended.
CPUIntel or AMD processor with 64-bit support. 2.8 GHz or faster processor recommended.

You must install .NET version 4.7.2 or later on your computer before using the MSI installer.

Datadog private locations endpoints

To pull test configurations and push test results, the private location worker needs access to the following Datadog API endpoints.

PortEndpointDescription
443intake.synthetics.datadoghq.comUsed by the private location to pull test configurations and push test results to Datadog using an in-house protocol based on AWS Signature Version 4 protocol.
443intake-v2.synthetics.datadoghq.com for versions >=0.2.0 and <=1.4.0Used by the private location to push browser test artifacts such as screenshots, errors, and resources.

PortEndpointDescription
443intake.synthetics.datadoghq.euUsed by the private location to pull test configurations and push test results to Datadog using an in-house protocol based on AWS Signature Version 4 protocol.

Note: These domains are pointing to a set of static IP addresses. These addresses can be found at https://ip-ranges.datadoghq.eu.

PortEndpointDescription
443intake.synthetics.us3.datadoghq.comUsed by the private location to pull test configurations and push test results to Datadog using an in-house protocol based on AWS Signature Version 4 protocol.

PortEndpointDescription
443intake.synthetics.ap1.datadoghq.comUsed by the private location to pull test configurations and push test results to Datadog using an in-house protocol based on AWS Signature Version 4 protocol.

PortEndpointDescription
443intake.synthetics.us5.datadoghq.comUsed by the private location to pull test configurations and push test results to Datadog using an in-house protocol based on AWS Signature Version 4 protocol.

PortEndpointDescription
443intake.synthetics.ddog-gov.comUsed by the private location to pull test configurations and push test results to Datadog using an in-house protocol based on AWS Signature Version 4 protocol. For versions 1.32.0 and later, these requests are Federal Information Processing Standards (FIPS) compliant.

Set up your private location

Only users with the Synthetics Private Locations Write role can create private locations. For more information, see Permissions.

Create your private location

Navigate to Synthetic Monitoring > Settings > Private Locations and click Add Private Location.

Create a private location

Fill out your private location details:

  1. Specify your private location’s Name and Description.
  2. Add any Tags you would like to associate with your private location.
  3. Choose one of your existing API Keys. Selecting an API key allows communication between your private location and Datadog. If you don’t have an existing API key, click Generate API key to create one on the dedicated page. Only Name and API key fields are mandatory.
  4. Set access for your private location and click Save Location and Generate Configuration File. Datadog creates your private location and generates the associated configuration file.
Add details to private location

Configure your private location

Configure your private location by customizing the generated configuration file. When you add initial configuration parameters such as proxies and blocked reserved IPs in Step 3, your generated configuration file updates automatically in Step 4.

You can access advanced options to adjust the configuration based on your internal network setup. For more information about the help command, see Configuration.

Proxy configuration

If the traffic between your private location and Datadog has to go through a proxy, specify your proxy URL as http://<YOUR_USER>:<YOUR_PWD>@<YOUR_IP>:<YOUR_PORT> to add the associated proxyDatadog parameter to your generated configuration file.

Add a proxy to your private location configuration file

Blocking reserved IPs

By default, Synthetic users can create Synthetic tests on endpoints using any IP. If you want to prevent users from creating tests on sensitive internal IPs in your network, toggle the Block reserved IPs button to block a default set of reserved IP ranges (IPv4 address registry and IPv6 address registry) and set the associated enableDefaultBlockedIpRanges parameter to true in your generated configuration file.

If some of the endpoints you are willing to test are located within one or several of the blocked reserved IP ranges, you can add their IPs and/or CIDRs to the allowed lists to add the associated allowedIPRanges parameters to your generated configuration file.

Configure reserved IPs

View your configuration file

After adding the appropriate options to your private location configuration file, you can copy and paste this file into your working directory. The configuration file contains secrets for private location authentication, test configuration decryption, and test result encryption.

Configure reserved IPs

Datadog does not store your secrets, so store them locally before clicking View Installation Instructions.

Note: You need to be able to reference these secrets again if you decide to add more workers or install workers on another host.

Install your private location

You can use DATADOG_API_KEY, DATADOG_ACCESS_KEY, DATADOG_SECRET_ACCESS_KEY, DATADOG_PUBLIC_KEY_PEM and DATADOG_PRIVATE_KEY environment variables in your task definition.

Launch your private location on:

Run this command to boot your private location worker by mounting your configuration file to the container. Ensure that your <MY_WORKER_CONFIG_FILE_NAME>.json file is in /etc/docker, not the root home folder:

docker run -d --restart unless-stopped -v $PWD/<MY_WORKER_CONFIG_FILE_NAME>.json:/etc/datadog/synthetics-check-runner.json datadog/synthetics-private-location-worker:latest

Note: If you have blocked reserved IPs, add the NET_ADMIN Linux capabilities to your private location container.

This command starts a Docker container and makes your private location ready to run tests. Datadog recommends running the container in detached mode with proper restart policy.

Root certificates

You can upload custom root certificates to your private locations to have your API and browser tests perform the SSL handshake using your own .pem files.

When spinning up your private location containers, mount the relevant certificate .pem files to /etc/datadog/certs in the same way you mount your private location configuration file. These certificates are considered trusted CA and are used at test runtime.

For more information about private locations parameters for admins, see Configuration.

  1. Create a docker-compose.yml file with:

    version: "3"
    services:
        synthetics-private-location-worker:
            image: datadog/synthetics-private-location-worker:latest
            volumes:
                - PATH_TO_PRIVATE_LOCATION_CONFIG_FILE:/etc/datadog/synthetics-check-runner.json
    

    Note: If you have blocked reserved IPs, add the NET_ADMIN Linux capabilities to your private location container.

  2. Start your container with:

    docker-compose -f docker-compose.yml up
    

Root certificates

You can upload custom root certificates to your private locations to have your API and browser tests perform the SSL handshake using your own .pem files.

When spinning up your private location containers, mount the relevant certificate .pem files to /etc/datadog/certs in the same way you mount your private location configuration file. These certificates are considered trusted CA and are used at test runtime.

For more information about private locations parameters for admins, see Configuration.

The Podman configuration is very similar to Docker, however, you must set NET_RAW as an additional capability to support ICMP tests.

  1. Run sysctl -w "net.ipv4.ping_group_range = 0 2147483647" from the host where the container runs.

  2. Run this command to boot your private location worker by mounting your configuration file to the container. Ensure that your <MY_WORKER_CONFIG_FILE_NAME>.json file is accessible to mount to the container:

    podman run --cap-add=NET_RAW --rm -it -v $PWD/<MY_WORKER_CONFIG_FILE_NAME>.json:/etc/datadog/synthetics-check-runner.json gcr.io/datadoghq/synthetics-private-location-worker:latest
    

    If you have configured blocked reserved IP addresses, add the NET_ADMIN Linux capabilities to your private location container.

This command starts a Podman container and makes your private location ready to run tests. Datadog recommends running the container in detached mode with proper restart policy.

To deploy the private locations worker in a secure manner, set up and mount a Kubernetes Secret resource in the container under /etc/datadog/synthetics-check-runner.json.

  1. Create a Kubernetes Secret with the previously created JSON file by executing the following:

    kubectl create secret generic private-location-worker-config --from-file=<MY_WORKER_CONFIG_FILE_NAME>.json
    
  2. Use deployments to describe the desired state associated with your private locations. Create the following private-location-worker-deployment.yaml file:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: datadog-private-location-worker
      namespace: default
    spec:
      selector:
        matchLabels:
          app: private-location
      template:
        metadata:
          name: datadog-private-location-worker
          labels:
            app: private-location
        spec:
          containers:
          - name: datadog-private-location-worker
            image: datadog/synthetics-private-location-worker
            volumeMounts:
            - mountPath: /etc/datadog/synthetics-check-runner.json
              name: worker-config
              subPath: <MY_WORKER_CONFIG_FILE_NAME>
          volumes:
          - name: worker-config
            secret:
              secretName: private-location-worker-config
    

    Note: If you have blocked reserved IPs, add the NET_ADMIN Linux capabilities to your private location container.

  3. Apply the configuration:

    kubectl apply -f private-location-worker-deployment.yaml
    

For OpenShift, run the private location with the anyuid SCC. This is required for your browser test to run.

You can set environment variables in your configuration parameters that point to secrets you have already configured. To create environment variables with secrets, see the Kubernetes documentation.

Alternatively:

  1. Add the Datadog Synthetics Private Location to your Helm repositories:

    helm repo add datadog https://helm.datadoghq.com
    helm repo update
    
  2. Install the chart with the release name <RELEASE_NAME> by using the previously created JSON file:

    helm install <RELEASE_NAME> datadog/synthetics-private-location --set-file configFile=<MY_WORKER_CONFIG_FILE_NAME>.json
    

Note: If you have blocked reserved IPs, add the NET_ADMIN Linux capabilities to your private location container.

Create a new EC2 task definition that matches the following. Replace each parameter with the corresponding value found in your previously generated private location configuration file:

{
    ...
    "containerDefinitions": [
        {
            "command": [
                "--site='...'",
                "--locationID='...'",
                "--accessKey='...'",
                "--datadogApiKey='...'",
                "--secretAccessKey='...'",
                "--privateKey='-----BEGIN RSA PRIVATE KEY-----XXXXXXXX-----END RSA PRIVATE KEY-----'",
                "--publicKey.pem='-----BEGIN PUBLIC KEY-----XXXXXXXX-----END PUBLIC KEY-----'",
                "--publicKey.fingerprint='...'"
            ],
            ...
            "image": "datadog/synthetics-private-location-worker:latest",
            ...
        }
    ],
    ...
    "compatibilities": [
        "EC2"
    ],
    ...
}

Notes:

  • If you have blocked reserved IPs, configure a linuxParameters to grant NET_ADMIN capabilities to your private location containers.
  • If you use the DATADOG_API_KEY, DATADOG_ACCESS_KEY, DATADOG_SECRET_ACCESS_KEY, DATADOG_PUBLIC_KEY_PEM and DATADOG_PRIVATE_KEY environment variables, you do not need to include them in the "command": [ ] section.

Create a new Fargate task definition that matches the following. Replace each parameter with the corresponding value found in your previously generated private location configuration file:

{
    ...
    "containerDefinitions": [
        {
            "command": [
                "--site='...'",
                "--locationID='...'",
                "--accessKey='...'",
                "--datadogApiKey='...'",
                "--secretAccessKey='...'",
                "--privateKey='-----BEGIN RSA PRIVATE KEY-----XXXXXXXX-----END RSA PRIVATE KEY-----'",
                "--publicKey.pem='-----BEGIN PUBLIC KEY-----XXXXXXXX-----END PUBLIC KEY-----'",
                "--publicKey.fingerprint='...'"
            ],
            ...
            "image": "datadog/synthetics-private-location-worker:latest",
            ...
        }
    ],
    ...
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    ...
}

Note: Because the private location firewall option is not supported on AWS Fargate, the enableDefaultBlockedIpRanges parameter cannot be set to true.

Because Datadog already integrates with Kubernetes and AWS, it is ready-made to monitor EKS.

  1. Create a Kubernetes Secret with the previously created JSON file by executing the following:

    kubectl create secret generic private-location-worker-config --from-file=<MY_WORKER_CONFIG_FILE_NAME>.json
    
  2. Use deployments to describe the desired state associated with your private locations. Create the following private-location-worker-deployment.yaml file:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: datadog-private-location-worker
      namespace: default
    spec:
      selector:
        matchLabels:
          app: private-location
      template:
        metadata:
          name: datadog-private-location-worker
          labels:
            app: private-location
        spec:
          containers:
          - name: datadog-private-location-worker
            image: datadog/synthetics-private-location-worker
            volumeMounts:
            - mountPath: /etc/datadog/synthetics-check-runner.json
              name: worker-config
              subPath: <MY_WORKER_CONFIG_FILE_NAME>
          volumes:
          - name: worker-config
            configMap:
              name: private-location-worker-config
    

    Note: If you have blocked reserved IPs, configure a security context to grant NET_ADMIN Linux capabilities to your private location containers.

  3. Apply the configuration:

    kubectl apply -f private-location-worker-deployment.yaml
    
  1. Download the datadog-synthetics-worker-<version>.amd64.msi file and run this file from the machine you want to install the private location on.

  2. Click Next on the welcome page, read the EULA, and accept the terms and conditions. Click Next.

  3. Modify where the application will be installed, or leave the default settings. Click Next.

  4. To configure your Windows private location, you can either:

    • Paste and enter a JSON configuration for your Datadog Synthetics Private Location Worker. This file is generated by Datadog when you create a private location.
    • Browse or type a file path to a file containing a JSON configuration for your Datadog Synthetics Private Location Worker.
    • You can leave it blank and run C:\\Program Files\Datadog-Synthetics\Synthetics\synthetics-pl-worker.exe --config=<PathToYourConfiguration> in the Windows command-line prompt after the installation is complete.
    Synthetics Private Location Worker wizard, MSI installer. The option 'Paste in a JSON configuration' is selected. A text field for this JSON configuration is displayed.
  5. You can apply the following configuration options:

    Synthetics Private Location Worker wizard, MSI installer. Firewall and log settings are displayed.
    Apply firewall rules needed by this program to Windows Firewall
    Allow the installer to apply firewall rules on install and remove them on uninstall.
    Apply rules to block reserved IPs in Windows Firewall
    Configure blocking rules for Chrome, Firefox, and Edge (if they are installed) and add rules to block reserved IP address ranges outbound in Windows Firewall.
    Enable File Logging
    Allow the Synthetics Private Location Worker to log files in the installation directory.
    Log Rotation Days
    Specifies how many days to keep logs before deleting them from the local system.
    Logging Verbosity
    Specifies the verbosity of the console and file logging for the Synthetics Private Location Worker.
  6. Click Next and Install to start the installation process.

Once the process is complete, click Finish on the installation completion page.

If you entered your JSON configuration, the Windows Service starts running using that configuration. If you did not enter your configuration, run C:\\Program Files\Datadog-Synthetics\Synthetics\synthetics-pl-worker.exe --config=< PathToYourConfiguration > from a command prompt or use the start menu shortcut to start the Synthetics Private Location Worker.
  1. Download the datadog-synthetics-worker-<version>.amd64.msi file and run this file from the machine you want to install the private location on.

  2. Run one of the following commands inside the directory where you downloaded the installer:

    • In a PowerShell Terminal:

      Start-Process msiexec "/i datadog-synthetics-worker-<version>-beta.amd64.msi /quiet /qn WORKERCONFIG_FILEPATH=C:\ProgramData\Datadog-Synthetics\worker-config.json";
      
    • Or in a Command Terminal:

      msiexec /i datadog-synthetics-worker-1.43.0-beta.amd64.msi /quiet /qn WORKERCONFIG_FILEPATH=C:\ProgramData\Datadog-Synthetics\worker-config.json
      

Additional parameters can be added:

Optional ParameterDefinitionValueDefault ValueType
APPLYDEFAULTFIREWALLRULESApplies firewall rules needed for the program.1N/A0: Disabled
1: Enabled
APPLYFIREWALLDEFAULTBLOCKRULESBlocks reserved IP addresses for each browser you have installed (Chrome, Edge, and Firefox). Blocking loopback connections is not possible in Windows Firewall.0N/A0: Disabled
1: Enabled
LOGGING_ENABLEDWhen enabled, this configures file logging. These logs are stored in the installation directory under the logs folder.0--enableFileLogging0: Disabled
1: Enabled
LOGGING_VERBOSITYConfigures the logging verbosity for the program. This affects console and file logs.This affects console and file logs.-vvv-v: Error
-vv: Warning
-vvv: Info
vvvv: Debug
LOGGING_MAXDAYSNumber of days to keep file logs on the system before deleting them. Can be any number when running an unattended installation.7--logFileMaxDaysInteger
WORKERCONFIG_FILEPATHThis should be changed to the path to your Synthetics Private Location Worker JSON configuration file. Wrap this path in quotes if your path contains spaces.--configString

Set up liveness and readiness probes

Add a liveness or readiness probe so your orchestrator can ensure the workers are running correctly.

For readiness probes, you need to enable private location status probes on port 8080 in your private location deployment. For more information, see Private Locations Configuration.

healthcheck:
  retries: 3
  test: [
    "CMD", "wget", "-O", "/dev/null", "-q", "http://localhost:8080/liveness"
  ]
  timeout: 2s
  interval: 10s
  start_period: 30s
livenessProbe:
  httpGet:
    path: /liveness
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
readinessProbe:
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  httpGet:
    path: /readiness
    port: 8080
livenessProbe:
  httpGet:
    path: /liveness
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
readinessProbe:
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  httpGet:
    path: /readiness
    port: 8080
"healthCheck": {
  "retries": 3,
  "command": [
    "CMD-SHELL", "/usr/bin/wget", "-O", "/dev/null", "-q", "http://localhost:8080/liveness"
  ],
  "timeout": 2,
  "interval": 10,
  "startPeriod": 30
}
"healthCheck": {
  "retries": 3,
  "command": [
    "CMD-SHELL", "wget -O /dev/null -q http://localhost:8080/liveness || exit 1"
  ],
  "timeout": 2,
  "interval": 10,
  "startPeriod": 30
}
livenessProbe:
  httpGet:
    path: /liveness
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
readinessProbe:
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  httpGet:
    path: /readiness
    port: 8080

Additional health check configurations

This method of adding private location health checks is no longer supported. Datadog recommends using liveness and readiness probes.

The /tmp/liveness.date file of private location containers gets updated after every successful poll from Datadog (2s by default). The container is considered unhealthy if no poll has been performed in a while, for example: no fetch in the last minute.

Use the configuration below to set up health checks on your containers with livenessProbe:

healthcheck:
  retries: 3
  test: [
    "CMD", "/bin/sh", "-c", "'[ $$(expr $$(cat /tmp/liveness.date) + 300000) -gt $$(date +%s%3N) ]'"
  ]
  timeout: 2s
  interval: 10s
  start_period: 30s
livenessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - '[ $(expr $(cat /tmp/liveness.date) + 300000) -gt $(date +%s%3N) ]'
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
livenessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - '[ $(expr $(cat /tmp/liveness.date) + 300000) -gt $(date +%s%3N) ]'
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
"healthCheck": {
  "retries": 3,
  "command": [
    "CMD-SHELL", "/bin/sh -c '[ $(expr $(cat /tmp/liveness.date) + 300000) -gt $(date +%s%3N) ]'"
  ],
  "timeout": 2,
  "interval": 10,
  "startPeriod": 30
}
"healthCheck": {
  "retries": 3,
  "command": [
    "CMD-SHELL", "/bin/sh -c '[ $(expr $(cat /tmp/liveness.date) + 300000) -gt $(date +%s%3N) ]'"
  ],
  "timeout": 2,
  "interval": 10,
  "startPeriod": 30
}
livenessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - '[ $(expr $(cat /tmp/liveness.date) + 300000) -gt $(date +%s%3N) ]'
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Upgrade a private location image

To upgrade an existing private location, click the Gear icon on the private location side panel and click Installation instructions.

Access the setup workflow for a private location

Then, run the configuration command based on your environment to get the latest version of the private location image.

Note: If you’re using docker run to launch your Private Location image and you’ve previously installed the Private Location image using the latest tag, make sure to add --pull=always to the docker run command to make sure the newest version is pulled rather than relying on the cached version of the image that may exist locally with the same latest tag.

Test your internal endpoint

Once at least one private location worker starts reporting to Datadog, the private location status displays green.

Private location reporting

You can see a REPORTING health status and an associated monitor status displayed on the Private Locations list in the Settings page.

Private location health and monitor status

Start testing your first internal endpoint by launching a fast test on one of your internal endpoints to see if you get the expected response:

Note: Datadog only sends outbound traffic from your private location, no inbound traffic is transmitted.

Launch Synthetic tests from your private location

Create an API, multistep API, or browser test, and select your Private Locations of interest.

Assign Synthetic test to private location

Use private locations just like your Datadog managed locations: assign Synthetic tests to private locations, visualize test results, retrieve Synthetic metrics, and more.

Scale your private location

Because you can run several workers for one single private location with a single configuration file, you can horizontally scale your private locations by adding or removing workers to them. When doing so, make sure to set a concurrency parameter and allocate worker resources that are consistent with the types and the number of tests you want your private location to execute.

You can also vertically scale your private locations by increasing the load your private location workers can handle. Similarly, you should use the concurrency parameter to adjust the maximum number of test your workers allowed to run and update the resources allocated to your workers.

For more information, see Dimensioning Private Locations.

In order to use private locations for Continuous Testing, set a value in the concurrency parameter to control your parallelization. For more information, see Continuous Testing.

Monitor your private location

While you initially add resources that are consistent with the number and type of tests to execute from your private location, the easiest way to know if you should downscale or upscale your private location is to closely monitor them. Private Location Monitoring provides insight about the performance and state of your private location as well as out-of-the-box metrics and monitors.

For more information, see Private Location Monitoring.

Permissions

By default, only users with the Datadog Admin Role can create private locations, delete private locations, and access private location installation guidelines.

Users with the Datadog Admin and Datadog Standard roles can view private locations, search for private locations, and assign Synthetic tests to private locations. Grant access to the Private Locations page by upgrading your user to one of these two default roles.

If you are using the custom role feature, add your user to a custom role that includes synthetics_private_location_read and synthetics_private_location_write permissions.

Further Reading