New Relic

New Relic Dashboard

Overview

Connect to New Relic to:

  • See key New Relic metrics (like response time and Apdex score) in context with the rest of your Datadog metrics
    (only works with New Relic Pro accounts and above).
  • See New Relic alerts in your event stream.

Setup

Installation

New Relic alerts in event stream

  1. On the “Alerts & AI” tab, navigate to “Notification Channels”.

  2. Select “New Notification Channel”.

  3. Select “Webhook” as the channel type.

  4. Name your channel “Datadog”.

  5. Enter this base URL:

    https://app.datadoghq.com/intake/webhook/newrelic?api_key=<DATADOG_API_KEY>
    
  6. Click “Custom Payload” and ensure that the payload is in JSON format. Note: See section below for instructions on including custom tags in JSON.

  7. Click “Create Channel”.

  8. Click “Alert Policies”.

  9. Select any alert policies for which you would like alerts to be sent into Datadog.

New Relic APM metric collection

  1. Locate your REST API key on New Relic’s API Keys page (Account Settings -> Integrations -> API Keys) and enter it in the form on the Datadog New Relic Integration page.
  2. To tag all metrics with your New Relic account number, add an account tag.
  3. Choose whether you want to collect your metrics per hosts or app-wide. Note: Enabling these options imports New Relic hosts to Datadog.
New Relic custom metrics may take 5-10 minutes to show up in Datadog.
The first 5000 custom metrics are available for auto complete.

Data Collected

Metrics

new_relic.apdex.score
(gauge)
Ratio of satisfactory response times to unsatisfactory response times
Shown as apdex
new_relic.application_summary.apdex_score
(gauge)
Ratio of satisfactory response times to unsatisfactory response times
Shown as apdex
new_relic.application_summary.apdex_target
(gauge)
Threshold ratio of satisfactory response times to unsatisfactory response times
Shown as apdex
new_relic.application_summary.concurrent_instance_count
(gauge)
Number of concurrent instances serving the application
Shown as instance
new_relic.application_summary.error_rate
(gauge)
Ratio of the number of errors reported by the application to the total number of served requests
Shown as percent
new_relic.application_summary.host_count
(gauge)
Number of hosts serving the application
Shown as host
new_relic.application_summary.instance_count
(gauge)
Number of instances serving the application
Shown as instance
new_relic.application_summary.response_time
(gauge)
Application response time
Shown as millisecond
new_relic.application_summary.throughput
(gauge)
Number of requests served by the application
Shown as request
new_relic.database.all.average_exclusive_time
(gauge)
Average time spent in database queries, exclusive of any time instrumented by other metrics
Shown as millisecond
new_relic.errors.all.errors_per_minute
(gauge)
Number of errors reported by the application
Shown as error
new_relic.web_transaction.average_response_time
(gauge)
Average response time of web transactions served by the application
Shown as second
new_relic.web_transaction.requests_per_minute
(gauge)
Number of web transaction requests served by the application
Shown as request
new_relic.synthetics_summary.monitors.count
(gauge)
Count of monitors
Shown as monitor
new_relic.synthetics_summary.monitors.frequency
(gauge)
Frequency of the monitor
Shown as minute
new_relic.synthetics_summary.locations.count
(gauge)
Count of locations associated with the monitor
Shown as location
new_relic.synthetic_check.duration
(gauge)
Total time for the monitor run
Shown as millisecond
new_relic.synthetic_check.total_request_body_size
(gauge)
Size of the body request to the server
Shown as byte
new_relic.synthetic_check.total_request_header_size
(gauge)
Size of the header request to the server
Shown as byte
new_relic.synthetic_check.total_response_header_size
(gauge)
Size of the response header returned by the server
Shown as byte
new_relic.synthetic_check.total_response_body_size
(gauge)
Size of the response body returned by the server
Shown as byte
new_relic.synthetic_check.count
(count)
Count of monitor runs (use result tag to select only successes or failures)
Shown as check
new_relic.synthetic_check.errors
(count)
Count of individual request errors (not check failures)
Shown as error
new_relic.synthetic_request.count
(count)
Count of requests
Shown as request
new_relic.synthetic_request.duration_blocked.average
(gauge)
Average time the requests were blocked
Shown as millisecond
new_relic.synthetic_request.duration_connect.average
(gauge)
Average time the requests were establishing a connection
Shown as millisecond
new_relic.synthetic_request.duration_dns.average
(gauge)
Average time the requests were resolving DNS
Shown as millisecond
new_relic.synthetic_request.duration_receive.average
(gauge)
Average time the requests were receiving data
Shown as millisecond
new_relic.synthetic_request.duration_send.average
(gauge)
Average time the requests were sending data
Shown as millisecond
new_relic.synthetic_request.duration_ssl.average
(gauge)
Average time establishing an SSL connection
Shown as millisecond
new_relic.synthetic_request.duration_wait.average
(gauge)
Average time the requests were waiting
Shown as millisecond
new_relic.synthetic_request.resources_load_time
(gauge)
Average resources load time
Shown as millisecond
new_relic.synthetic_request.time_spent_third_parties
(gauge)
Average time spent by third parties
Shown as millisecond
new_relic.apm_mobile.mobile_application_summary.active_users
(gauge)
The number of active users for a mobile application
Shown as user
new_relic.apm_mobile.mobile_application_summary.launch_count
(count)
The number of mobile applications launched
Shown as execution
new_relic.apm_mobile.mobile_application_summary.throughput
(gauge)
Throughput for a mobile application
Shown as request
new_relic.apm_mobile.mobile_application_summary.response_time
(gauge)
Mobile application response time
Shown as millisecond
new_relic.apm_mobile.mobile_application_summary.calls_per_session
(gauge)
Number of calls per session
Shown as invocation
new_relic.apm_mobile.mobile_application_summary.interaction_time
(gauge)
Interaction time for a mobile application
Shown as millisecond
new_relic.apm_mobile.mobile_application_summary.failed_call_rate
(gauge)
Percentage of failed calls for a mobile application
Shown as percent
new_relic.apm_mobile.mobile_application_summary.remote_error_rate
(gauge)
Percentage of remote error for a mobile application
Shown as percent
new_relic.apm_mobile.mobile_application_summary.unresolved_crash_count
(count)
Number of unresolved crash count for a mobile application
Shown as execution
new_relic.apm_mobile.mobile_application_summary.crash_count
(count)
Crash count for a mobile application
Shown as execution
new_relic.apm_mobile.mobile_application_summary.crash_rate
(gauge)
Crash rate for a mobile application
Shown as percent
new_relic.apm_mobile.mobile.crash.all.call_count
(count)
Total numbers of crashed calls
Shown as execution
new_relic.apm_mobile.session.start.call_count
(count)
Number of started calls for a session
Shown as invocation
new_relic.apm_mobile.mobile.failed_call_rate.call_count
(count)
Failed call rate count for a mobile application
Shown as invocation
new_relic.apm_mobile.session.duration.call_count
(count)
Number of call counts for a session
Shown as invocation
new_relic.apm_mobile.cpu.user.utilization.percent
(gauge)
User space CPU usage utilization
Shown as percent
new_relic.apm_mobile.cpu.system.utilization.percent
(gauge)
Kernel space CPU usage utilization
Shown as percent
new_relic.web_transaction_total_time.average_response_time
(gauge)
Average response time of web transactions served by the application
Shown as second
new_relic.web_transaction_total_time.requests_per_minute
(gauge)
Number of web transaction requests served by the application
Shown as request
new_relic.web_transaction_total_time.calls_per_minute
(gauge)
Number of calls served by the application
Shown as invocation
new_relic.web_transaction_total_time.min_response_time
(gauge)
Minimum response time of web transactions served by the application
Shown as second
new_relic.web_transaction_total_time.max_response_time
(gauge)
Maximum response time of web transactions served by the application
Shown as second
new_relic.web_transaction_total_time.total_call_time_per_minute
(gauge)
Total time, in second per minute, spent in web application transactions
Shown as fraction
new_relic.web_transaction_total_time.standard_deviation
(gauge)
Standard deviation of the response time of web transactions served by the application
Shown as second
new_relic.web_transaction_total_time.average_exclusive_time
(gauge)
Average time spent in web application transactions, exclusive of any time instrumented by other metrics
Shown as millisecond
new_relic.web_transaction_total_time.average_value
(gauge)
Average response time over all instances in the call count
Shown as second

Events

The New Relic integration can ingest New Relic alerts in the event stream.

Service Checks

The New Relic integration does not include any service checks.

Troubleshooting

Collect metrics by host

When ‘Collect metrics by host’ is set, Datadog collects application metrics for every associated host, instead of the overall host throughput based average.

This makes sense when using the metrics separately, for example: “host X has aberrant error rate Y which is problematic, though application Z overall across many hosts has an acceptable error rate in aggregate”.

This also import New Relic hosts to Datadog Infrastructure section.

Different values in New Relic vs Datadog

When New Relic computes the aggregate application-level value for metrics that are measured at the host level, such as response time, they compute a weighted average based on each host’s measured throughput.

The closest value in Datadog is the avg aggregator, which computes the arithmetic mean of the values. This is also the default aggregator, and the result for the simplest query, for example: new_relic.web_transaction.average_response_time{*}. If your hosts all have approximately the same throughput, Datadog’s average aggregation and New Relic’s throughput-weighted aggregation yield similar numbers, but if the throughput is not balanced, different aggregate application-level values display in New Relic and Datadog.

For example, say you have an application with three hosts. At a specific point in time, the hosts have the following values:

           throughput    response time
hostA         305 rpm           240 ms
hostB         310 rpm           250 ms
hostC          30 rpm            50 ms

New Relic would compute the application-level response time as follows:

hostA: 240 ms * 305 rpm = 73200 total time
hostB: 250 ms * 310 rpm = 77500 total time
hostC:  50 ms *  30 rpm =  1500 total time

total throughput = 305 + 310 + 30 = 645 rpm
average response time = (73200 + 77500 + 1500) / 645 = 236.0 ms

Whereas Datadog would compute the arithmetic mean:

average response time = (240 + 250 + 50) / 3 = 180.0 ms

Include custom tags on beta alerts

You can include custom tags with the “Use Custom Payload” option through New Relic’s Beta Alerts feature. To configure this, navigate to your New Relic account, and click the ‘Alerts Beta’ button in the upper right-hand corner of the screen. Then, select the ‘Notification channels’ section and find the Webhook you’ve setup for Datadog. From here there should be a section called ‘Use Custom Payload’, and once selected, it expands to reveal a JSON payload. You need to modify this payload by adding a “tags” attribute. For example, a modified payload might look like this:

{
    "account_id": "$ACCOUNT_ID",
    "account_name": "$ACCOUNT_NAME",
    "condition_id": "$CONDITION_ID",
    "condition_name": "$CONDITION_NAME",
    "current_state": "$EVENT_STATE",
    "details": "$EVENT_DETAILS",
    "event_type": "$EVENT_TYPE",
    "incident_acknowledge_url": "$INCIDENT_ACKNOWLEDGE_URL",
    "incident_id": "$INCIDENT_ID",
    "incident_url": "$INCIDENT_URL",
    "owner": "$EVENT_OWNER",
    "policy_name": "$POLICY_NAME",
    "policy_url": "$POLICY_URL",
    "runbook_url": "$RUNBOOK_URL",
    "severity": "$SEVERITY",
    "targets": "$TARGETS",
    "timestamp": "$TIMESTAMP",
    "tags": ["application:yourapplication", "host:yourhostname", "sometag"]
}

After your modifications are complete, make sure you select ‘Update Chanel’, for your changes to be saved.