Overview

This tutorial walks you through the steps for enabling tracing on a sample Go application installed on a host. In this scenario, you install a Datadog Agent on the same host as the application.

For other scenarios, including applications in containers or on cloud infrastructure, Agent in a container, and applications written in different languages, see the other Enabling Tracing tutorials.

See Tracing Go Applications for general comprehensive tracing setup documentation for Go.

Prerequisites

• A Datadog account and organization API key
• A physical or virtual Linux host with root access when using sudo. The host has the following requirements:
  • Git
  • Curl
  • Go version 1.18+
  • Make and GCC

Install the Agent

If you haven’t installed a Datadog Agent on your machine, go to Integrations > Agent and select your operating system. For example, on most Linux platforms, you can install the Agent by running the following script, replacing <YOUR_API_KEY> with your Datadog API key:

DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=<YOUR_API_KEY> DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

To send data to a Datadog site other than datadoghq.com, replace the DD_SITE environment variable with your Datadog site.

Verify that the Agent is running and sending data to Datadog by going to Events > Explorer, optionally filtering by the Datadog Source facet, and looking for an event that confirms the Agent installation on the host:

Install and run a sample Go application

Next, install a sample application to trace. The code sample for this tutorial can be found at github.com/DataDog/apm-tutorial-golang.git. Clone the git repository by running:

git clone https://github.com/DataDog/apm-tutorial-golang.git

Build the sample application using the following command. The command might take a while the first time you run it:

make runNotes

The sample notes application is a basic REST API that stores data in an in-memory database. Use curl to send a few API requests:

curl localhost:8080/notes

Returns [] because there is nothing in the database yet

curl -X POST 'localhost:8080/notes?desc=hello'

Adds a note with the description hello and an ID value of 1. Returns {"id":1,"description":"hello"}.

curl localhost:8080/notes/1

Returns the note with id value of 1: {"id":1,"description":"hello"}

curl -X POST 'localhost:8080/notes?desc=otherNote'

Adds a note with the description otherNote and an ID value of 2. Returns {"id":2,"description":"otherNote"}

curl localhost:8080/notes

Returns the contents of the database: [{"id":1,"description":"hello"},{"id";2,"description":"otherNote"}]

Run more API calls to see the application in action. When you’re done, run the following command to exit the application:

make exitNotes

Install Datadog tracing

Next, install the Go tracer. From your apm-tutorial-golang directory, run:

go get gopkg.in/DataDog/dd-trace-go.v1/ddtrace

Now that the tracing library has been added to go.mod, enable tracing support.

Uncomment the following imports in apm-tutorial-golang/cmd/notes/main.go:

  sqltrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/database/sql"
  chitrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/go-chi/chi"
  httptrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/net/http"
  "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
  "fmt"

Change the import:

_ "github.com/mattn/go-sqlite3"

to:

"github.com/mattn/go-sqlite3"

In the main() function, uncomment the following lines:

tracer.Start()
defer tracer.Stop()

client = httptrace.WrapClient(client, httptrace.RTWithResourceNamer(func(req *http.Request) string {
		return fmt.Sprintf("%s %s", req.Method, req.URL.Path)
	}))

r.Use(chitrace.Middleware(chitrace.WithServiceName("notes")))

In setupDB(), uncomment the following lines:

sqltrace.Register("sqlite3", &sqlite3.SQLiteDriver{}, sqltrace.WithServiceName("db"))
db, err := sqltrace.Open("sqlite3", "file::memory:?cache=shared")

Comment out the following line:

db, err := sql.Open("sqlite3", "file::memory:?cache=shared")

Once you’ve made these changes, run:

go mod tidy

Launch the Go application and explore automatic instrumentation

To start generating and collecting traces, launch the application again with make runNotes.

Use curl to again send requests to the application:

curl localhost:8080/notes

[]

curl -X POST 'localhost:8080/notes?desc=hello'

{"id":1,"description":"hello"}

curl localhost:8080/notes/1

{"id":1,"description":"hello"}

curl localhost:8080/notes

[{"id":1,"description":"hello"}]

Wait a few moments, and take a look at your Datadog UI. Navigate to APM > Traces. The Traces list shows something like this:

There are entries for the database (db) and the notes app. The traces list shows all the spans, when they started, what resource was tracked with the span, and how long it took.

If you don’t see traces, clear any filter in the Traces Search field (sometimes it filters on an environment variable such as ENV that you aren’t using).

Examine a trace

On the Traces page, click on a POST /notes trace, and you’ll see a flame graph that shows how long each span took and what other spans occurred before a span completed. The bar at the top of the graph is the span you selected on the previous screen (in this case, the initial entry point into the notes application).

The width of a bar indicates how long it took to complete. A bar at a lower depth represents a span that completes during the lifetime of a bar at a higher depth.

The flame graph for a POST trace looks something like this:

A GET /notes trace looks something like this:

Tracing configuration

You can configure the tracing library to add tags to the telemetry it sends to Datadog. Tags help group, filter, and display data meaningfully in dashboards and graphs. To add tags, specify environment variables when running the application. The project Makefile includes the environment variables DD_ENV, DD_SERVICE, and DD_VERSION, which are set to enable Unified Service Tagging:

run: build
  DD_TRACE_SAMPLE_RATE=1 DD_SERVICE=notes DD_ENV=dev DD_VERSION=0.0.1 ./cmd/notes/notes &

For more information on available configuration options, see Configuring the Go Tracing Library.

Use automatic tracing libraries

Datadog has several fully supported libraries for Go that allow for automatic tracing when implemented in the code. In the cmd/notes/main.go file, you can see the go-chi, sql, and http libraries being aliased to the corresponding Datadog libraries: chitrace, sqltrace, and httptrace respectively:

import (
  ...

  sqltrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/database/sql"
  chitrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/go-chi/chi"
  httptrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/net/http"
  ...
)

In cmd/notes/main.go, the Datadog libraries are initialized with the WithServiceName option. For example, the chitrace library is initialized as follows:

r := chi.NewRouter()
r.Use(middleware.Logger)
r.Use(chitrace.Middleware(chitrace.WithServiceName("notes")))
r.Mount("/", nr.Register())

Using chitrace.WithServiceName("notes") ensures that all elements traced by the library fall under the service name notes.

The main.go file contains more implementation examples for each of these libraries. For an extensive list of libraries, see Go Compatibility Requirements.

Use custom tracing with context

In cases where code doesn’t fall under a supported library, you can create spans manually.

Remove the comments around the makeSpanMiddleware function in notes/notesController.go. It generates middleware that wraps a request in a span with the supplied name. To use this function, comment out the following lines:

  r.Get("/notes", nr.GetAllNotes)                // GET /notes
  r.Post("/notes", nr.CreateNote)                // POST /notes
  r.Get("/notes/{noteID}", nr.GetNoteByID)       // GET /notes/123
  r.Put("/notes/{noteID}", nr.UpdateNoteByID)    // PUT /notes/123
  r.Delete("/notes/{noteID}", nr.DeleteNoteByID) // DELETE /notes/123

Remove the comments around the following lines:

  r.Get("/notes", makeSpanMiddleware("GetAllNotes", nr.GetAllNotes))               // GET /notes
  r.Post("/notes", makeSpanMiddleware("CreateNote", nr.CreateNote))                // POST /notes
  r.Get("/notes/{noteID}", makeSpanMiddleware("GetNote", nr.GetNoteByID))          // GET /notes/123
  r.Put("/notes/{noteID}", makeSpanMiddleware("UpdateNote", nr.UpdateNoteByID))    // PUT /notes/123
  r.Delete("/notes/{noteID}", makeSpanMiddleware("DeleteNote", nr.DeleteNoteByID)) // DELETE /notes/123

Also remove the comment around the following import:

"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

There are several examples of custom tracing in the sample application. Here are a couple more examples. Remove the comments to enable these spans:

The doLongRunningProcess function creates child spans from a parent context:

func doLongRunningProcess(ctx context.Context) {
	childSpan, ctx := tracer.StartSpanFromContext(ctx, "traceMethod1")
	childSpan.SetTag(ext.ResourceName, "NotesHelper.doLongRunningProcess")
	defer childSpan.Finish()

	time.Sleep(300 * time.Millisecond)
	log.Println("Hello from the long running process in Notes")
	privateMethod1(ctx)
}

The privateMethod1 function demonstrates creating a completely separate service from a context:

func privateMethod1(ctx context.Context) {
	childSpan, _ := tracer.StartSpanFromContext(ctx, "manualSpan1",
		tracer.SpanType("web"),
		tracer.ServiceName("noteshelper"),
	)
	childSpan.SetTag(ext.ResourceName, "privateMethod1")
	defer childSpan.Finish()

	time.Sleep(30 * time.Millisecond)
	log.Println("Hello from the custom privateMethod1 in Notes")
}

Uncomment the following imports:

  "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/ext"
  "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

Launch the application with make runNotes and try the curl commands again to observe the custom spans and traces you’ve just configured:

curl localhost:8080/notes

[]

curl -X POST 'localhost:8080/notes?desc=hello'

{"id":1,"description":"hello"}

curl localhost:8080/notes/1

{"id":1,"description":"hello"}

curl localhost:8080/notes

[{"id":1,"description":"hello"}]

For more information on custom tracing, see Go Custom Instrumentation.

Examine distributed traces

Tracing a single application is a great start, but the real value in tracing is seeing how requests flow through your services. This is called distributed tracing.

The sample project includes a second application called calendar that returns a random date whenever it is invoked. The POST endpoint in the notes application has a second query parameter named add_date. When it is set to y, the notes application calls the calendar application to get a date to add to the note.

To enable tracing in the calendar application, uncomment the following lines in cmd/calendar/main.go:

  chitrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/go-chi/chi"
  "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

  tracer.Start()
  defer tracer.Stop()

  r.Use(chitrace.Middleware(chitrace.WithServiceName("calendar")))

1. If the notes application is still running, use make exitNotes to stop it.

2. Run make run to start the sample application.

3. Send a POST request with the add_date parameter:

   Copy

   curl -X POST 'localhost:8080/notes?desc=hello_again&add_date=y'

4. In the Trace Explorer, click this latest notes trace to see a distributed trace between the two services:

This flame graph combines interactions from multiple applications:

• The first span is a POST request sent by the user and handled by the chi router through the supported go-chi library.
• The second span is a createNote function that was manually traced by the makeSpanMiddleware function. The function created a span from the context of the HTTP request.
• The next span is the request sent by the notes application using the supported http library and the client initialized in the main.go file. This GET request is sent to the calendar application. The calendar application spans appear in blue because they are separate service.
• Inside the calendar application, a go-chi router handles the GET request and the GetDate function is manually traced with its own span under the GET request.
• Finally, the purple db call is its own service from the supported sql library. It appears at the same level as the GET /Calendar request because they are both called by the parent span CreateNote.

Troubleshooting

If you’re not receiving traces as expected, set up debug mode for the Go tracer. Read Enable debug mode to find out more.

Further reading

Additional helpful documentation, links, and articles:

Additional tracing library configuration options Detailed tracing library setup instructions Supported Go frameworks for automatic instrumentation Manually configuring traces and spans Ingestion mechanisms Tracing library open source code repository