Splunk

이 페이지는 아직 한국어로 제공되지 않으며 번역 작업 중입니다. 번역에 관한 질문이나 의견이 있으시면 언제든지 저희에게 연락해 주십시오.

Overview

Connect your Splunk log monitoring to be able to:

  • Get notified of your reports.
  • Correlate these reports with your other metrics
  • Collaborate with your team on those events

Setup

Installation

To receive your reports from Splunk into Datadog, you need to have the datadog python library installed on your splunk server:

pip install datadog

Once it is done, get your api key and an application key and drop the following dog-splunk.sh script into $SPLUNK_HOME/bin/scripts


export API_KEY=YOURAPIKEYHERE
export APP_KEY=YOURAPPKEYHERE

dog --api-key $API_KEY --application-key $APP_KEY event post \
"Found $SPLUNK_ARG_1 events in splunk" \
"Matching $SPLUNK_ARG_2 based on $SPLUNK_ARG_5," \
" from report $SPLUNK_ARG_4. More details at $SPLUNK_ARG_6." \
 --aggregation_key $SPLUNK_ARG_3 --type splunk

Make sure the script is executable and owned by the splunk user and group.

Once the script is in place, create a new report or navigate to an existing report. Click the Edit Schedule and check the checkbox to Schedule the Report. When you get to the option to Run a Script, enter dog-splunk.sh in the Filename textbox. Click Save and you should see the results start appearing in your Event Stream.

Troubleshooting

If you see an error code on each run of runshellscript in splunkd.log, try adding > dog_splunk_trace.txt 2>&1 to the end of the last command. This creates a $SPLUNK_HOME/etc/apps/search/bin/dog_splunk_trace.txt file, which provides more detail about the problem.

If the trace file has something like the usage help for the dog command followed by dog: error: unrecognized arguments: OR failed OR severe, add single quotes around \$SPLUNK_ARG_3 on the last line.

If the trace file includes a Traceback that ends with pkg_resources.DistributionNotFound or something similar, add three unsets to the top of your dog-splunk.sh script:

#!/bin/bash
unset PYTHONHOME
unset PYTHONPATH
unset LD_LIBRARY_PATH
export API_KEY=YOURAPIKEYHERE
export APP_KEY=YOURAPPKEYHERE

dog --api-key $API_KEY --application-key $APP_KEY event post \
"Found $SPLUNK_ARG_1 events in splunk" \
"Matching $SPLUNK_ARG_2 based on $SPLUNK_ARG_5," \
" from report $SPLUNK_ARG_4. More details at $SPLUNK_ARG_6." \
 --aggregation_key $SPLUNK_ARG_3 --type splunk

Further Reading

Knowledge base

The script file uses variables made available by Splunk. If you would like to customize the message, see the following table of variables:

$SPLUNK_ARG_0Script Name
$SPLUNK_ARG_1Number of events returned
$SPLUNK_ARG_2Search terms
$SPLUNK_ARG_3Fully qualified query string
$SPLUNK_ARG_4Name of saved search
$SPLUNK_ARG_5Trigger reason (for example, “The number of events was greater than 1”)
$SPLUNK_ARG_6Browser URL to view the saved search
$SPLUNK_ARG_7option removed in version 3.6
$SPLUNK_ARG_8File in which the results for this search are stored (contains raw results)

You can modify the text of the events by for example using datadog’s @mention to notify people of these reports.


Further reading

This documentation verified on October 28, 2015 using the Splunk Enterprise AMI on AWS