Graphs are the window onto your monitored systems. Most of the times that you visit Datadog, you look at dashboards made up of graphs. Other times you see email notifications that include a graph of some fluctuation in the system. And yet other times you see graphs in your Slack, HipChat, and other chat clients documenting the changes in metrics over the course of time. Graphs are at the heart of monitoring and observability, so it is essential to understand how to define great graphs.
There are two ways to interact with the Graphing Editor: using the GUI (the default method) and writing JSON (the more complete method). This page covers using the GUI. To learn more about using JSON, visit the JSON Graphing Primer Page
On each graph there is a pencil icon that opens the graph editor.
The graphing editor has three tabs:
When you first open the graphing editor window, you are on the Edit tab. Here you can use the UI to choose most settings to tweak your graphs. Here is an example of what you might see. This example comes from the first graph in the standard Postgres Integration dashboard:
Configuring a graph is a multi-step process:
When you create a graph, you probably have a metric in mind that you want to show. You can select that in the first dropdown in the Choose metrics and events section. If you aren’t sure exactly which metric to use, you might want to start with the Metrics Explorer or a Notebook. You can also look in the Metrics Summary.
The Metrics Explorer allows you to play around with different graph settings in a more ad-hoc way. The Metrics Summary allows to learn more about the type of metric as well as setting the default unit for a metric.
Once you have a metric in mind to display in your graph, select your visualization. Check the list of all visualizations.
Now that you have the metric and a visualization in place, you can filter down the hosts to be graphed. To the right of the metric is a dropdown which by default says (everywhere). Click this and choose the tag(s) you want to filter by. To learn more about tags, refer to the Tagging documentation.
Next to the filter dropdown is the aggregation method. This defaults to avg by but can be changed to max by, min by, or sum by. In most cases, the metric has many values for each time interval, coming from many hosts or instances.
The aggregation method chosen determines how the metrics is aggregated into a single line. So if you are graphing a metric that is from 100 hosts, sum by adds up all of those values and display the sum.
After the aggregation method you can determine what constitutes a line or grouping in a graph. If you choose host, then you have a line (in the case of line graphs) for every host. If you choose role, then there is a line for every role. Then that line is made up of metrics from all the hosts in that role, aggregated using the method you chose above.
Regardless of the options chosen above, there is always some aggregation of data due to the physical size constraints of the window holding the graph. If a metric is updated every second and you are looking at 4 hours of data, you need 14,400 points to display everything. Each graph we display has about 300 points shown at any given time.
In the example above, each point displayed on the screen represents 48 data points. In practice, metrics are collected by the Agent every 15-20 seconds. So one day’s worth of data is 4,320 data points. You might consider a rollup function that looks at 5 or 10 minutes worth of data if you would like to have more control over the display of your data for a graph that shows 1 day.
To use the rollup function, click the plus sign to the right of the aggregation group and choose rollup from the dropdown. Now choose how you want to aggregate the data and the interval in seconds.
To create a single line that represents the total available disk space on average across all machines rolled up in 60 seconds buckets, you would use a query like this:
When switching to the JSON view, the query looks like this:
"q": "avg:system.disk.free{*}.rollup(avg, 60)"
For more about using the JSON view, visit the JSON Graphing Primer page.
Depending on your analysis needs, you may choose to apply other mathematical functions to the query. Examples include rates and derivatives, smoothing, and more. For a list of available functions click here
The Datadog UI also supports the ability to graph your metrics with various arithmetic options. Use any of: +, -, /, * to modify the values that are displayed on your graphs.
This syntax allows for both integer values as well as arithmetic using multiple metrics.
You can modify how a metric value is displayed on a graph by performing an arithmetic operation on the metric.
For example, if you would like to visualize the double of a specific metric, say system.load.5.
This can be done inside a graph editor by clicking on the Graph Editor and selecting Advanced… .
From there, enter your arithmetic in the Formula box, in this case: a * 2.
If you would like to visualize the percentage of jvm.heap_memory used for instance, perform the following arithmetic across two metrics already being reported to your Datadog application:
jvm.heap_memory / jvm.heap_memory_max
This can be done in the same manner as above, utilizing the Advanced… option in the Graph Editor. From there, select:
Add Query +Once you have added all of the metrics you would like to visualize with their context, notice they are each assigned a letter: the first metric is represented by a, the second metric is represented b, and so on.
Then in the Formula box, enter the arithmetic you would like, in this case: ( a / b ):
To display only your formula, un-check your metrics a and b:
Note: Formula are not lettered, you cannot do Arithmetic between formulas.
Each query or formula can be aliased. The alias overrides the display on the graph and legend, which is particularly helpful for long metric names or to clarify a formula.
At the end of your query/formula click on the as… button, then enter your metric alias:
The Datadog y-axis controls are available via the UI and the JSON editor. They allow you to:
Change the Y-axis scale with the Show Y-Axis Controls button:
There are three configuration settings:
Min/max (optional): Specifies minimum (and/or maximum) value to show on y-axis. It takes a number, or “auto” for default behavior. Default value is “auto”.
Scale (optional): Specifies the scale type. Possible values:
Always include zero (optional): Specifies whether or not to always include zero or fit the axis to the data range. Default is to always include zero.
Note: as the mathematical log function doesn’t accept negative values, the Datadog log scale only works if values are of the same sign (everything > 0 or everything < 0). Otherwise an empty graph is returned.
Add events from related systems to add even more context to your graph. For example, you can add Github commits, Jenkins deploys, or Docker creation events. Click the Overlay Events button and enter a query to find and display your events. Use the same query format as for the event stream, for example:
| Query | Description |
|---|---|
sources:jenkins |
Shows all events from the Jenkins source |
tag:role:web |
Shows all events with the tag role:web |
tags:$<TEMPLATE_VAR> |
Shows all events from the selected <TEMPLATE_VAR> |
If you don’t enter a title, we automatically generate a title based on the selections you have made. But it may be more useful to the users of the dashboard to create a title that more aptly describes the purpose of the graph. Linking the technical purpose to the business benefits adds even more value.
The final step is to click Save. You can always come back in to the editor and tweak the graph further depending on your needs.
