New Relic

Overview

Connect to New Relic to:

• See key New Relic metrics (like response time and Apdex score) in context with the rest of your Datadog metrics
  (only works with New Relic Pro accounts and above).
• See New Relic alerts in your event stream.

Setup

Installation

New Relic alerts in event stream

1. On the “Alerts & AI” tab, navigate to “Notification Channels”.

2. Select “New Notification Channel”.

3. Select “Webhook” as the channel type.

4. Name your channel “Datadog”.

5. Enter this base URL:

   https://app.datadoghq.com/intake/webhook/newrelic?api_key=<DATADOG_API_KEY>
   

6. Click “Custom Payload” and ensure that the payload is in JSON format. Note: See section below for instructions on including custom tags in JSON.

7. Click “Create Channel”.

8. Click “Alert Policies”.

9. Select any alert policies for which you would like alerts to be sent into Datadog.

New Relic APM metric collection

1. Locate your REST API key on New Relic’s API Keys page (Account Settings -> Integrations -> API Keys) and enter it in the form on the Datadog New Relic Integration page.
2. To tag all metrics with your New Relic account number, add an account tag.
3. Choose whether you want to collect your metrics per hosts or app-wide. Note: Enabling these options imports New Relic hosts to Datadog.

Data Collected

Metrics

Events

The New Relic integration can ingest New Relic alerts in the event stream.

Service Checks

The New Relic integration does not include any service checks.

Troubleshooting

Collect metrics by host

When ‘Collect metrics by host’ is set, Datadog collects application metrics for every associated host, instead of the overall host throughput based average.

This makes sense when using the metrics separately, for example: “host X has aberrant error rate Y which is problematic, though application Z overall across many hosts has an acceptable error rate in aggregate”.

This also import New Relic hosts to Datadog Infrastructure section.

Different values in New Relic vs Datadog

When New Relic computes the aggregate application-level value for metrics that are measured at the host level, such as response time, they compute a weighted average based on each host’s measured throughput.

The closest value in Datadog is the avg aggregator, which computes the arithmetic mean of the values. This is also the default aggregator, and the result for the simplest query, for example: new_relic.web_transaction.average_response_time{*}. If your hosts all have approximately the same throughput, Datadog’s average aggregation and New Relic’s throughput-weighted aggregation yield similar numbers, but if the throughput is not balanced, different aggregate application-level values display in New Relic and Datadog.

For example, say you have an application with three hosts. At a specific point in time, the hosts have the following values:

           throughput    response time
hostA         305 rpm           240 ms
hostB         310 rpm           250 ms
hostC          30 rpm            50 ms

New Relic would compute the application-level response time as follows:

hostA: 240 ms * 305 rpm = 73200 total time
hostB: 250 ms * 310 rpm = 77500 total time
hostC:  50 ms *  30 rpm =  1500 total time

total throughput = 305 + 310 + 30 = 645 rpm
average response time = (73200 + 77500 + 1500) / 645 = 236.0 ms

Whereas Datadog would compute the arithmetic mean:

average response time = (240 + 250 + 50) / 3 = 180.0 ms

Include custom tags on beta alerts

You can include custom tags with the “Use Custom Payload” option through New Relic’s Beta Alerts feature. To configure this, navigate to your New Relic account, and click the ‘Alerts Beta’ button in the upper right-hand corner of the screen. Then, select the ‘Notification channels’ section and find the Webhook you’ve setup for Datadog. From here there should be a section called ‘Use Custom Payload’, and once selected, it expands to reveal a JSON payload. You need to modify this payload by adding a “tags” attribute. For example, a modified payload might look like this:

{
    "account_id": "$ACCOUNT_ID",
    "account_name": "$ACCOUNT_NAME",
    "condition_id": "$CONDITION_ID",
    "condition_name": "$CONDITION_NAME",
    "current_state": "$EVENT_STATE",
    "details": "$EVENT_DETAILS",
    "event_type": "$EVENT_TYPE",
    "incident_acknowledge_url": "$INCIDENT_ACKNOWLEDGE_URL",
    "incident_id": "$INCIDENT_ID",
    "incident_url": "$INCIDENT_URL",
    "owner": "$EVENT_OWNER",
    "policy_name": "$POLICY_NAME",
    "policy_url": "$POLICY_URL",
    "runbook_url": "$RUNBOOK_URL",
    "severity": "$SEVERITY",
    "targets": "$TARGETS",
    "timestamp": "$TIMESTAMP",
    "tags": ["application:yourapplication", "host:yourhostname", "sometag"]
}

After your modifications are complete, make sure you select ‘Update Chanel’, for your changes to be saved.