For PHP, to start troubleshooting issues with the Datadog ASM extension, enable debug logs in the ASM extension’s .ini
file.
The extension’s ini
file is usually found in /etc/php/<version>/xxx/conf.d/98-ddtrace.ini
, but the location may differ depending on your installation. Look at the beginning of the phpinfo()
output to identify the directory that is scanned for .ini
files, if any. In the .ini
file, set the following configuration options with the following:
datadog.appsec.log_level='debug'
datadog.appsec.helper_extra_args='--log_level=debug'
datadog.appsec.helper_log_file='/tmp/helper.log'
The extension outputs logs to the default php_error
log file. If there are no logs in the file, add the following to the .ini
file:
datadog.appsec.log_file='tmp/extension.log'
Installation fails to find PHP
If the installation script is unable to find the correct PHP version, you can set the --php-bin
to the PHP binary location, for example:
$ php datadog-setup.php --php-bin /usr/bin/php7.4 --enable-appsec
Connection to helper failed
If the ASM extension is unable to communicate with the helper process, the following warning occurs:
PHP Warning: Unknown: [ddappsec] Connection to helper failed and we are not going to attempt to launch it: dd_error
The warning could be followed by one of these error messages:
PHP Warning: Unknown: [ddappsec] Could not open lock file /tmp/ddappsec.lock: Permission denied in Unknown on line 0
PHP Warning: Unknown: [ddappsec] Call to bind() failed: Permission denied
PHP Warning: Unknown: [ddappsec] Failed to unlink /tmp/ddappsec.sock: Operation not permitted
This indicates that the lock file or socket file used by the extension has invalid permissions, or the user executing the PHP process does not have write access to the tmp
directory.
If the lock file or socket file has invalid permissions, you can either delete them and restart Apache/FPM or adjust the user:group
to match the one used by Apache/FPM, for example, www-data
.
If the user doesn’t have write access to the tmp directory, you can change the location of the lock file and socket file by modifying the following settings in the extension’s .ini
file:
datadog.appsec.helper_runtime_path = /<directory with compatible permissions>/
Confirm ASM is enabled in the running application
Tracer startup logs show the tracer configuration and whether ASM is enabled or not. If appsec
is true
, then ASM is enabled and running.
For example, the following startup log shows that ASM is disabled:
2022/02/17 14:49:00 Datadog Tracer v1.36.0 INFO: DATADOG TRACER CONFIGURATION {"date":"2022-02-17T14:49:00+01:00","os_name":"Linux (Unknown Distribution)","os_version":"5.13.0","version":"v1.36.0","lang":"Go","lang_version":"go1.17.1","env":"prod","service":"grpcserver","agent_url":"http://localhost:8126/v0.4/traces","debug":false,"analytics_enabled":false,"sample_rate":"NaN","sampling_rules":null,"sampling_rules_error":"","service_mappings":null,"tags":{"runtime-id":"69d99219-b68f-4718-9419-fa173a79351e"},"runtime_metrics_enabled":false,"health_metrics_enabled":false,"profiler_code_hotspots_enabled":false,"profiler_endpoints_enabled":false,"dd_version":"","architecture":"amd64","global_service":"","lambda_mode":"false","appsec":false,"agent_features":{"DropP0s":false,"Stats":false,"StatsdPort":0}}
Enable debug logs
Enable debug logs with the environment variable DD_TRACE_DEBUG=1
. The ASM library will log to the standard error output.
Note: ASM only outputs logs when it is enabled. Use the environment variable DD_APPSEC_ENABLED=1
to enable ASM.
For Ruby, if you don’t see ASM threat information in the Trace and Signals Explorer after a few minutes, enable tracer diagnostics for debug logs. For example:
Datadog.configure do |c|
c.diagnostics.debug = true # increase general log level to debug
c.appsec.waf_debug = true # also enable WAF-specific log verbosity to highest level
end
Debug logs are verbose but useful. If you open up a ticket with Datadog support, forward the logs with your request.
Is ASM correctly enabled?
ASM has been correctly enabled if you see logs such as:
D, [2021-12-14T11:03:32.167125 #73127] DEBUG -- ddtrace: [ddtrace] (libddwaf/lib/datadog/appsec/waf.rb:296:in `block in logger=') {:level=>:ddwaf_log_info, :func=> "ddwaf_set_log_cb", :file=>"PowerWAFInterface.cpp", :message=>"Sending log messages to binding, min level trace"}
D, [2021-12-14T11:03:32.200491 #73127] DEBUG -- ddtrace: [ddtrace] (libddwaf/lib/datadog/appsec/waf.rb:296:in `block in logger=') {:level=>:ddwaf_log_debug, :func= >"parse", :file=>"parser_v2.cpp", :message=>"Loaded 124 rules out of 124 available in the ruleset"}
If you do not see those logs, check the following:
- If the correct ASM environment variables are set for your application process.
- The latest gem version is installed.
- The tracer is configured correctly and sending APM traces to your APM dashboard.
Is ASM called for each HTTP request?
To confirm that ASM is called for each HTTP request, trigger a test attack and look for these logs:
D, [2022-01-19T21:25:50.579745 #341792] DEBUG -- ddtrace: [ddtrace] (/home/lloeki/src/github.com/DataDog/dd-trace-rb/lib/datadog/appsec/reactive/operation.rb:14:in `initialize') operation: rack.request initialize
D, [2022-01-19T21:25:50.580300 #341792] DEBUG -- ddtrace: [ddtrace] (/home/lloeki/src/github.com/DataDog/dd-trace-rb/lib/datadog/appsec/contrib/rack/gateway/watcher.rb:25:in `block (2 levels) in watch') root span: 964736568335365930
D, [2022-01-19T21:25:50.580371 #341792] DEBUG -- ddtrace: [ddtrace] (/home/lloeki/src/github.com/DataDog/dd-trace-rb/lib/datadog/appsec/contrib/rack/gateway/watcher.rb:26:in `block (2 levels) in watch') active span: 964736568335365930
D, [2022-01-19T21:25:50.581061 #341792] DEBUG -- ddtrace: [ddtrace] (/home/lloeki/src/github.com/DataDog/dd-trace-rb/lib/datadog/appsec/contrib/rack/reactive/request.rb:34:in `block in subscribe') reacted to ["request.headers", "request.uri.raw", "request.query", "request.cookies", "request.body.raw"]: [{"version"=>"HTTP/1.1", "host"=>"127.0.0.1:9292", "accept"=>"*/*", "user-agent"=>"Nessus SOAP"}, "http://127.0.0.1:9292/", [], {}, ""]
If you don’t see those logs, try the following:
- Check that another upstream security system is not filtering requests based on the test header value, which would prevent the request from reaching the application.
- Send another test attack using another user agent value in the curl command to see if the threat information is successfully sent.
- Look in the application logs for the exact request you ran to confirm the request reached the application, and was not responded to by another upstream system.
If the Rack integration was configured manually, sometimes a known issue prevents ASM from working. For example:
Datadog.configure do |c|
c.tracing.instrument :rails
...
c.tracing.instrument :rack, web_service_name: "something", request_queuing: true
If c.tracing.instrument :rack
is present, remove it to see if the check passes.
Is ASM detecting HTTP request security threats?
To confirm that ASM is detecting security threats, trigger a test attack, and look for these logs:
D, [2021-12-14T22:39:53.268820 #106051] DEBUG -- ddtrace: [ddtrace] (ddtrace/lib/datadog/appsec/contrib/rack/reactive/request.rb:63:in `block in subscribe') WAF: #<struct Datadog::AppSec::WAF::Result action=:monitor, data=[{"rule"=>{"id"=>"ua0-600-10x", "name"=>"Nessus", "tags"=>{"type"=>"security_scanner", "category"=>"attack_attempt"}}, "rule_matches"=>[{"operator"=>"match_regex", "operator_value"=>"(?i)^Nessus(/|([ :]+SOAP))", "parameters"=>[{"address"=>"server.request.headers.no_cookies", "key_path"=>["user-agent"], "value"=>"Nessus SOAP", "highlight"=>["Nessus SOAP"]}]}]}], perf_data=nil, perf_total_runtime=20519>
If you don’t see those logs, check that another upstream security system is not filtering out the requests or altering them based on the test header value.
Is the tracer sending traces with security data?
ASM data is sent with APM traces. To confirm that ASM correctly detects and inserts security data into traces, trigger a test attack, and look for these tracer logs:
Tags: [
runtime-id => 0c3dfc67-9cf3-457c-a980-0229b203d048,
_dd.runtime_family => ruby,
appsec.event => true,
_dd.appsec.json => {"triggers":[{"rule":{"id":"ua0-600-10x","name":"Nessus","tags":{"type":"security_scanner","category":"attack_attempt"}},"rule_matches":[{"operator":"match_regex","operator_value":"(?i)^Nessus(/|([ :]+SOAP))","parameters":[{"address":"server.request.headers.no_cookies","key_path":["user-agent"],"value":"Nessus SOAP","highlight":["Nessus SOAP"]}]}]}]},
http.request.headers.host => 127.0.0.1:9292,
http.request.headers.accept => */*,
http.request.headers.user-agent => Nessus SOAP,
http.response.headers.content-type => text/plain,
http.host => 127.0.0.1,
http.useragent => Nessus SOAP,
network.client.ip => 127.0.0.1,
_dd.origin => appsec,
http.method => GET,
http.url => /,
http.base_url => http://127.0.0.1:9292,
http.status_code => 200,
http.response.headers.content_type => text/plain]
Metrics: [
_dd.agent_psr => 1.0,
system.pid => 155644.0,
_dd.appsec.enabled => 1.0,
_dd.measured => 1.0,
_sampling_priority_v1 => 2.0]]
Wait a minute for the agent to forward the traces, then check that the traces show up in the APM dashboard. The security information in the traces may take additional time to be processed by Datadog before showing up as suspicious requests in the ASM Trace and Signals Explorer.