Run Synthetic Tests from Private Locations

Overview

Private locations allow you to monitor internal-facing applications or any private endpoints that aren’t accessible from the public internet. They can also be used to:

• Create custom Synthetic locations in areas that are mission-critical to your business.
• Verify application performance in your internal CI environment before you release new features to production with Continuous Testing and CI/CD.
• Compare application performance from both inside and outside your internal network.

Private locations come as Docker containers that you can install wherever makes sense inside of your private network. Once created and installed, you can assign Synthetic tests to your private location just like you would with any managed location.

Your private location worker pulls your test configurations from Datadog’s servers using HTTPS, executes the test on a schedule or on-demand, and returns the test results to Datadog’s servers. You can then visualize your private locations test results in a completely identical manner to how you would visualize tests running from managed locations:

Prerequisites

Continuous Testing

In order to use private locations for Continuous Testing tests, you need v1.27.0 or later.

Docker

Private locations are Docker containers that you can install anywhere inside your private network. You can access the private location worker image on Google Container Registry. It can run on a Linux-based OS or Windows OS if the Docker engine is available on your host and can run in Linux containers mode.

Datadog private locations endpoints

To pull test configurations and push test results, the private location worker needs access to the following Datadog API endpoints.

Set up your private location

Only users with the Admin role can create private locations. For more information, see Permissions.

Create your private location

Navigate to Synthetic Monitoring > Settings > Private Locations and click Add Private Location.

Fill out your private location details:

1. Specify your private location’s Name and Description.
2. Add any Tags you would like to associate with your private location. If you are configuring a private location for Windows, select This is a Windows Private Location.
3. Choose one of your existing API Keys. Selecting an API key allows communication between your private location and Datadog. If you don’t have an existing API key, click Generate API key to create one on the dedicated page. Only Name and API key fields are mandatory.
4. Set access for your private location and click Save Location and Generate Configuration File. Datadog creates your private location and generates the associated configuration file.

Configure your private location

Configure your private location by customizing the generated configuration file. When you add initial configuration parameters such as proxies and blocked reserved IPs in Step 3, your generated configuration file updates automatically in Step 4.

You can access advanced options to adjust the configuration based on your internal network setup. For more information about the help command, see Configuration.

Proxy configuration

If the traffic between your private location and Datadog has to go through a proxy, specify your proxy URL as http://<YOUR_USER>:<YOUR_PWD>@<YOUR_IP>:<YOUR_PORT> to add the associated proxyDatadog parameter to your generated configuration file.

Blocking reserved IPs

By default, Synthetic users can create Synthetic tests on endpoints using any IP. If you want to prevent users from creating tests on sensitive internal IPs in your network, toggle the Block reserved IPs button to block a default set of reserved IP ranges (IPv4 address registry and IPv6 address registry) and set the associated enableDefaultBlockedIpRanges parameter to true in your generated configuration file.

If some of the endpoints you are willing to test are located within one or several of the blocked reserved IP ranges, you can add their IPs and/or CIDRs to the allowed lists to add the associated allowedIPRanges parameters to your generated configuration file.

View your configuration file

After adding the appropriate options to your private location configuration file, you can copy and paste this file into your working directory. The configuration file contains secrets for private location authentication, test configuration decryption, and test result encryption.

Datadog does not store your secrets, so store them locally before clicking View Installation Instructions.

Note: You need to be able to reference these secrets again if you decide to add more workers or install workers on another host.

Install your private location

You can use DATADOG_API_KEY, DATADOG_ACCESS_KEY, DATADOG_SECRET_ACCESS_KEY, DATADOG_PUBLIC_KEY_PEM and DATADOG_PRIVATE_KEY environment variables in your task definition.

Launch your private location on:

docker run --rm -v $PWD/<MY_WORKER_CONFIG_FILE_NAME>.json:/etc/datadog/synthetics-check-runner.json datadog/synthetics-private-location-worker:latest

{
    ...
    "containerDefinitions": [
        {
            "command": [
                "--site='...'",
                "--locationID='...'",
                "--accessKey='...'",
                "--datadogApiKey='...'",
                "--secretAccessKey='...'",
                "--privateKey='-----BEGIN RSA PRIVATE KEY-----XXXXXXXX-----END RSA PRIVATE KEY-----'",
                "--publicKey.pem='-----BEGIN PUBLIC KEY-----XXXXXXXX-----END PUBLIC KEY-----'",
                "--publicKey.fingerprint='...'"
            ],
            ...
            "image": "datadog/synthetics-private-location-worker:latest",
            ...
        }
    ],
    ...
    "compatibilities": [
        "EC2"
    ],
    ...
}

{
    ...
    "containerDefinitions": [
        {
            "command": [
                "--site='...'",
                "--locationID='...'",
                "--accessKey='...'",
                "--datadogApiKey='...'",
                "--secretAccessKey='...'",
                "--privateKey='-----BEGIN RSA PRIVATE KEY-----XXXXXXXX-----END RSA PRIVATE KEY-----'",
                "--publicKey.pem='-----BEGIN PUBLIC KEY-----XXXXXXXX-----END PUBLIC KEY-----'",
                "--publicKey.fingerprint='...'"
            ],
            ...
            "image": "datadog/synthetics-private-location-worker:latest",
            ...
        }
    ],
    ...
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    ...
}

Set up liveness and readiness probes

Add a liveness or readiness probe so your orchestrator can ensure the workers are running correctly.

For readiness probes, you need to enable private location status probes on port 8080 in your private location deployment. For more information, see Private Locations Configuration.

healthcheck:
  retries: 3
  test: [
    "CMD", "wget", "-O", "/dev/null", "-q", "http://localhost:8080/liveness"
  ]
  timeout: 2s
  interval: 10s
  start_period: 30s

livenessProbe:
  httpGet:
    path: /liveness
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
readinessProbe:
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  httpGet:
    path: /readiness
    port: 8080

livenessProbe:
  httpGet:
    path: /liveness
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
readinessProbe:
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  httpGet:
    path: /readiness
    port: 8080

"healthCheck": {
  "retries": 3,
  "command": [
    "CMD-SHELL", "/usr/bin/wget", "-O", "/dev/null", "-q", "http://localhost:8080/liveness"
  ],
  "timeout": 2,
  "interval": 10,
  "startPeriod": 30
}

"healthCheck": {
  "retries": 3,
  "command": [
    "CMD-SHELL", "wget -O /dev/null -q http://localhost:8080/liveness || exit 1"
  ],
  "timeout": 2,
  "interval": 10,
  "startPeriod": 30
}

livenessProbe:
  httpGet:
    path: /liveness
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
readinessProbe:
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  httpGet:
    path: /readiness
    port: 8080

Additional health check configurations

The /tmp/liveness.date file of private location containers gets updated after every successful poll from Datadog (2s by default). The container is considered unhealthy if no poll has been performed in a while, for example: no fetch in the last minute.

Use the configuration below to set up health checks on your containers with livenessProbe:

healthcheck:
  retries: 3
  test: [
    "CMD", "/bin/sh", "-c", "'[ $$(expr $$(cat /tmp/liveness.date) + 300000) -gt $$(date +%s%3N) ]'"
  ]
  timeout: 2s
  interval: 10s
  start_period: 30s

livenessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - '[ $(expr $(cat /tmp/liveness.date) + 300000) -gt $(date +%s%3N) ]'
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

livenessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - '[ $(expr $(cat /tmp/liveness.date) + 300000) -gt $(date +%s%3N) ]'
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

"healthCheck": {
  "retries": 3,
  "command": [
    "CMD-SHELL", "/bin/sh -c '[ $(expr $(cat /tmp/liveness.date) + 300000) -gt $(date +%s%3N) ]'"
  ],
  "timeout": 2,
  "interval": 10,
  "startPeriod": 30
}

"healthCheck": {
  "retries": 3,
  "command": [
    "CMD-SHELL", "/bin/sh -c '[ $(expr $(cat /tmp/liveness.date) + 300000) -gt $(date +%s%3N) ]'"
  ],
  "timeout": 2,
  "interval": 10,
  "startPeriod": 30
}

livenessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - '[ $(expr $(cat /tmp/liveness.date) + 300000) -gt $(date +%s%3N) ]'
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Test your internal endpoint

Once at least one private location container starts reporting to Datadog, the private location status displays green:

You can also see a REPORTING health status displayed on the Private Locations list in the Settings page:

Start testing your first internal endpoint by launching a fast test on one of your internal endpoints to see if you get the expected response:

Note: Datadog only sends outbound traffic from your private location, no inbound traffic is transmitted.

Launch Synthetic tests from your private location

Create an API, multistep API, or browser test, and select your Private Locations of interest.

Use private locations just like your Datadog managed locations: assign Synthetic tests to private locations, visualize test results, retrieve Synthetic metrics, and more.

Scale your private location

Because you can run several containers for one single private location with a single configuration file, you can horizontally scale your private locations by adding or removing workers to them. When doing so, make sure to set a concurrency parameter and allocate worker resources that are consistent with the types and the number of tests you want your private location to execute.

You can also vertically scale your private locations by increasing the load your private location containers can handle. Similarly, you should use the concurrency parameter to adjust the maximum number of test your workers allowed to run and update the resources allocated to your workers.

For more information, see Dimensioning Private Locations.

In order to use private locations for continuous testing, set a value in the concurrency parameter to control your parallelization. For more information, see Continuous Testing.

Monitor your private location

While you initially add resources that are consistent with the number and type of tests to execute from your private location, the easiest way to know if you should downscale or upscale your private location is to closely monitor them. Private Location Monitoring provides insight about the performance and state of your private location as well as out-of-the-box metrics and monitors.

For more information, see Private Location Monitoring.

Permissions

By default, only users with the Datadog Admin Role can create private locations, delete private locations, and access private location installation guidelines.

Users with the Datadog Admin and Datadog Standard roles can view private locations, search for private locations, and assign Synthetic tests to private locations. Grant access to the Private Locations page by upgrading your user to one of these two default roles.

If you are using the custom role feature, add your user to a custom role that includes synthetics_private_location_read and synthetics_private_location_write permissions.

Further Reading

Additional helpful documentation, links, and articles:

Monitor your Synthetic private locations with DatadogBLOG Getting Started with Private LocationsDOCUMENTATION Monitor your Private LocationsDOCUMENTATION Dimension your Private LocationsDOCUMENTATION Configure an API TestDOCUMENTATION Create and manage Synthetic Private Locations with TerraformTERRAFORM