Overview

Use Observability Pipelines’ sources to receive logs from your different log sources.

Select and set up your source when you build a pipeline in the UI. This is step 3 in the pipeline setup process:

1. Navigate to Observability Pipelines.
2. Select a template.
3. Select and set up your source.
4. Select and set up your destinations.
5. Set up your processors.
6. Install the Observability Pipelines Worker.

Sources have different prerequisites and settings. Some sources also need to be configured to send logs to the Observability Pipelines Worker.

Standard metadata fields

All sources add the following standard metadata fields to ingested events:

For example, if this is the raw event:

{
  "foo": "bar"
}

Then the enriched event with the standard metadata fields is:

{
  "foo": "bar",
  "hostname": "ip-34-2-553.us.test",
  "timestamp": "2024-06-17T22:25:55.439Z",
  "source_type": "splunk_tcp"
}

You can see these standard metadata fields when you use the tap command to see the events sent through the source.

After events are ingested by the source, they get sent to different processors and destinations that might update those fields. For example, if the event is sent to the Datadog Logs destination, the timestamp field gets converted to UNIX format.

Note: The bytes in per second metric in the UI is for ingested raw events, not enriched events.

Further reading

Additional helpful documentation, links, and articles:

Set up pipelinesDOCUMENTATION Processors for your pipelinesDOCUMENTATION Observability Pipelines destinationsDOCUMENTATION