Synthetic Monitoring Troubleshooting

Docs > Synthetic Testing and Monitoring > Synthetic Monitoring Troubleshooting

Overview

If you experience issues setting up or configuring Datadog Synthetic Monitoring, use this page to start troubleshooting. If you continue to have trouble, contact Datadog Support.

API tests

Network timings are varied

If you see a sudden spike or overall increase in your API test timing metrics, this usually indicates a bottleneck or delay in the request. For more information, see this guide on API Test Timings and Variations.

Browser tests

Recording

The website is not loading in the iframe

After downloading the Datadog extension, you are unable to see your website in the iframe on the right side of your Browser test’s recorder and the iframe displays Your website does not support being loaded through an iframe.. This could mean that your application has some settings preventing it from being opened in an iframe.

Or, if you are unable to login to your website when recording in the iframe recorder, this could mean that your application has a request that is blocked.

Try opening your website in a pop-up window by clicking Open in Popup to record your user journey.

Some applications load in the iframe but some do not

This means your applications and environments have different restrictions, which causes some of them to be visualized in an iframe while the others are not viewable.

This most likely means you are trying to record steps on an http page. Only https is supported in the recorder iframe. You should open your page as a pop-up or change your URL to an https one to start recording on the page.

My website is not loading in the iframe and I cannot record any steps, even when opening my website in a pop-up

After downloading the Datadog extension, you are unable to see your website in the iframe on the right side of your Browser test’s recorder. Additionally, you cannot record any steps, regardless of whether you open your website in the iframe or in a pop-up:

If that happens, ensure the Datadog extension has the permissions to read and change data on the intended websites by specifying your website in the On specific sites section or by toggling On all sites:

I’m unable to record steps on my application

Your Chrome browser might have some policies preventing the extension from performing the recording as expected.

To find out, go to chrome://policy and look for any extension-related settings such as ExtensionSettings.

By default, the iframe/pop-up of the recorder uses your own browser. This means that if you’re already logged into your application, the iframe/pop-up might directly display a post login page, therefore preventing you from recording your login steps without logging out first.

To be able to record your steps without logging out from your application, just leverage the recorder’s incognito mode:

Opening a pop-up window in incognito mode allows you to start your test’s recording from the start URL set in your test configuration with a session completely isolated from your own browser’s main session and user data.

This incognito pop-up window ignores your previous browser history including cookies and local data. You are automatically logged out from your account and can start recording your login steps as if you were visiting your website for the first time.

Test results

My mobile small or tablet browser test results keep failing

If your website is using responsive techniques, its DOM might differ a lot depending on the device your test is running on. It might use a specific DOM when running from a Laptop Large, and have a different architecture when running from a Tablet or a Mobile Small.

This means that the steps you recorded from a Laptop Large viewport might not be applicable to the same website accessed from a Mobile Small, causing your Mobile Small test results to fail:

For these types of cases, Datadog recommends creating separate Mobile Small or Tablet specific tests where the recorded steps match the viewport your test is set to at runtime.

To record steps with a Mobile Small or Tablet viewport, selecting Mobile Small or Tablet in the recorder dropdown before hitting the Start Recording button.

Additionally, Datadog’s test browsers run in headless, meaning Browser tests do not support some features. For example, Browser tests do not support touch and cannot use touch to detect whether the website should appear with its mobile design.

`None or multiple elements detected` step warning appears in browser tests

One of your browser test steps is showing a None or multiple elements detected step warning:

This means that the user locator defined for that step is either targeting several elements, or none of them, consequently preventing the Browser test from knowing which element needs to be interacted with.

To fix it, go edit your recording, open the advanced options of the step that is having the issue, go to the page the step is testing, and click on Test. This highlights the located element or prints an error message. You can then go ahead and fix your user locator to have it match a single element of the page:

I am having issues with a CSS pointer property

Automated browsers do not support emulating the CSS pointer media feature. Browser tests have pointer: none for all tests and devices (laptop, tablet, or mobile).

Resource duration

A resource is of a longer duration than the actual step duration

Long-loading resources may span across multiple steps. Within a test result’s step, Datadog returns all resources initiated during that specific step. However, Datadog allows roughly 20 seconds for important network calls to finish. After this period, the synthetics worker proceeds to the subsequent step. The worker uses a hierarchy of timeouts, allowing it to balance speed and reliability. Because of this, Datadog does not advise using step duration to measure the speed or slowness of a web application. The step duration reflects the balanced time the worker needs to deliver a reliable result.

API and browser tests

Unauthorized errors

If one of your Synthetic tests is throwing a 401, it most likely means that it is unable to authenticate on the endpoint. You should use the method that you use to authenticate on that endpoint (outside of Datadog) and replicate it when configuring your Synthetic test.

Is your endpoint using header-based authentication?
- Basic Authentication: specify the associated credentials in the Advanced options of your HTTP or Browser test.
- Token based authentication: extract your token with a first HTTP test, create a global variable by parsing the response of that first test, and re-inject that variable in a second HTTP or Browser test requiring the authentication token.
- Session based authentication: add the required headers or cookies in the Advanced options of your HTTP or Browser test.
Is this endpoint using query parameters for authentication (for example, do you need to add a specific API key in your URL parameters?)
Is this endpoint using IP-based authentication? If so, you might need to allow part or all of the IPs from which Synthetic tests originate.

Forbidden errors

If you observe 403 Forbidden errors returned by Synthetic tests, it may be the result of your web server blocking or filtering requests that include the Sec-Datadog header. This header is added to each Synthetic request Datadog initiates to identify the source of the traffic and assist Datadog support in identifying the specific test execution.

Additionally, you might also have to ensure Datadog Synthetic Monitoring IP ranges are allowed as traffic sources by your firewalls.

Missing notifications

Synthetic tests by default do not renotify. This means that if you add your notification handle such as your email address or Slack handle after a transition is generated (for example: a test going into alert or recovering from a previous alert), a notification is not sent for that transition. A notification is sent for the next transition.

Mobile tests

Unable to launch a device recording

If there are security checks during application startup, such as verifying if USB debugging is enabled, Datadog recommends uploading a version of the application that does not contain these checks.

Ensure proper app functioning during test recording and execution

If certain features of your iOS app are not functioning as expected during test recording or test execution, it could be a result of the app resigning process. This resigning process is required so that mobile devices can trust the provided application. Issues with the resigning process may cause essential iOS entitlements to be removed (such as access to Contacts, Camera, Keychain, Photos, Health Kit, Home Kit, and so on).

To minimize the risk of entitlement-related issues, and for improved compatibility, Datadog recommends distributing your iOS app using Ad Hoc or Development provisioning profiles.

Private locations

My browser test results sometimes show `Page crashed` errors

This could uncover a resource exhaustion issue on your private location workers. Make sure your private location workers are provisioned with sufficient memory resources.

My tests are sometimes slower to execute

This could uncover a resource exhaustion issue on your private locations workers. Make sure your private location workers are provisioned with sufficient CPU resources.

My browser tests are taking too long to run

Confirm you are not seeing out of memory issues with your private location deployments. If you have tried scaling your workers instances following the dimensioning guidelines already, reach out to Datadog Support.

Requirements for browser tests running on private location

Browser tests require elevated privileges to spawn (when the test execution starts) and kill (when the test execution ends) the browser process. If your private location is configured with a security context that restricts elevated privileges, then the private location emits error logs when the browser test is executed. The reported logs vary based on the browser that is selected for test execution. Tests executed on Chrome/Edge report the following error:

Critical error in startBrowser: Failed to launch the browser process!
sudo: The "no new privileges" flag is set, which prevents sudo from running as root.
sudo: If sudo is running in a container, you may need to adjust the container configuration to disable the flag.

Firefox reports the following error:

Impossible to spawn Firefox: binary is not a Firefox executable
sudo: The "no new privileges" flag is set, which prevents sudo from running as root.
sudo: If sudo is running in a container, you may need to adjust the container configuration to disable the flag.

Requirements for ICMP tests running on private location

ICMP tests use the ping command to assess network routes and connectivity to a host. ping opens a raw socket to send ICMP packets through, so it requires the NET_RAW capability to allow for the creation of raw sockets. If your container is configured with a security context that drops or removes this capability, ICMP tests will not be able to function properly on the private location.

Additionally, ping requires elevated privileges to create the raw socket. The private location cannot execute ICMP tests if the private location is configured with a security context that restricts elevated privileges.

`TIMEOUT` errors appear in API tests executed from my private location

This might mean your private location is unable to reach the endpoint your API test is set to run on. Confirm that the private location is installed in the same network as the endpoint you are willing to test. You can also try to run your test on different endpoints to see if you get the same TIMEOUT error or not.

Resolving IPv4 forwarding issues for private location containers

Private locations require access to Datadog’s Synthetic Monitoring intake endpoints to pull test configurations and push test results. If IPv4 forwarding is disabled on a Linux server, the private location may lose access to the public internet and consequently cannot connect to the intake. Docker typically attempts to enable IP forwarding when a container starts, but if it remains disabled, then the container cannot reach external services like the intake.

If this is the case, the private location will report logs like:

WARNING: IPv4 forwarding is disabled. Networking will not work.

and

Queue error - onFetchMessagesLongPolling - getaddrinfo EAI_AGAIN intake.synthetics.datadoghq.com

To resolve this issue, ensure that net.ipv4.ip_forward is enabled on the host.

My security policy requires private location containers to run with a read-only root file system

Private location containers require read-write access to specific folders and files to function correctly. If the container is run with a read-only root file system, it will fail to start up properly due to several critical operations that depend on write access.

During startup, the container attempts to set Linux capabilities on certain binaries. This is necessary because, during the private location build process, metadata bits are stripped from the binaries for security reasons. By default, this restricts execution to the root user. Since private locations run as the dog user, the container reapplies the necessary permissions to allow execution. On a read-only root file system, these updates fail, resulting in errors when the container starts up.

My private location containers sometimes get killed `OOM`

Private location containers getting killed Out Of Memory generally uncover a resource exhaustion issue on your private location workers. Make sure your private location containers are provisioned with sufficient memory resources.

The `invalid mount config for type "bind": source path must be a directory` error appears when attempting to run a private location

This occurs when you attempt to mount a single file in a Windows-based container, which is not supported. For more information, see the Docker mount volume documentation. Ensure that the source of the bind mount is a local directory.

Restart the Synthetics Private Location Worker service without a reboot

First, ensure that you installed the private location with a configuration specified at installation time. You can either use a GUI or use Windows PowerShell to restart the service.

GUI

Open the MSI installer and search for Services in the Start menu.
Start Services on any user account.
Click Services (Local) and find the service called Datadog Synthetics Private Location.
Right-click on the service found in Step 2 and choose Restart.

The Synthetics Private Location Worker now runs under the Local Service account. To confirm this, launch Task Manager and look for the synthetics-pl-worker process on the Details tab.

PowerShell

Start Windows PowerShell on any Windows account that has the rights to execute PowerShell scripts.
Run the following command: Restart-Service -Name “Datadog Synthetics Private Location”.

Keep the Synthetics Private Location Worker running

First, ensure that you are logged in on the machine where the Synthetics Private Location Windows Service is installed, and you have the permissions to create scheduled tasks on the machine.

If the Synthetics Private Location Worker crashes, add a scheduled task in Windows that runs a PowerShell script to restart the application if it stops running. This ensures that a private location is restarted after a crash.

If you provided a configuration file when installing the application, a Windows service called Datadog Synthetics Private Location starts automatically after installation. To verify this, ensure that you can see the service running in the Services tool. This Windows service restarts the private location automatically.

I am being asked for a password for sudo/I am being asked for a password for the dog user

The Private Location user (dog) requires sudo for various reasons. Typically, this user is granted certain permissions to allow sudo access in the process of launching the Private Location on your container. Confirm if you have a policy in place that restricts the dog user’s ability to sudo, or prevents the container from launching as the dog user (UID 501).

Additionally, in Private Location versions >v1.27, Datadog depends on the use of the clone3 system call. In some older versions of container runtime environments (such as Docker versions <20.10.10), clone3 is not supported by the default seccomp policy. Confirm that your container runtime environment’s seccomp policy includes clone3. You can do this by updating the version of your runtime in use, manually adding clone3 to your seccomp policy, or using an unconfined seccomp policy. For more information, see Docker’s seccomp documentation.