Join the Preview!
Federated Logs is in Preview. Use this form to submit your request today.
Request AccessOverview
An external source is a table that contains logs in your own data platform, such as Databricks. After you connect it, you can search it from Log Explorer with Federated Logs, without ingesting the logs into Datadog.
Prerequisites
- Access to the Federated Logs Preview. To request access, complete the Preview form.
- The Logs Federated Query Write (
logs_federated_query_write) permission, which lets you connect, edit, and delete sources. For more information, see Role Based Access Control.
Set up your source
Databricks prerequisites
Grant access to the service principal
Datadog runs queries with the Databricks SQL Statement Execution API, using the service principal of your Databricks integration. Use the service principal’s application (client) ID, not its display name, for <application_id>.
In your Databricks workspace settings, open the service principal, go to the Configurations tab, and select Databricks SQL access. New service principals don’t inherit this entitlement from the users group. For more information, see Manage entitlements in the Databricks documentation.
Give the service principal the CAN USE permission on the SQL warehouse configured in your Databricks integration. Datadog runs queries on this warehouse. CAN MANAGE is not required.
Grant read access to the table. In Databricks, open the SQL Editor and run the following commands:
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<application_id>`;
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<application_id>`;
GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO `<application_id>`;
The External Sources page lists only the catalogs, schemas, and tables that the service principal has access to. Queries consume compute on your SQL warehouse.
Supported column types
| Datadog field | Supported column types |
|---|
| Timestamp | timestamp, or string in ISO 8601 format |
Unique identifier (_id) | string |
| Status | string |
| Message, Host, Service, Source | string |
| Tags | array<string>, with each element in key:value format |
Numeric timestamps, such as epoch values, are not supported.
Source fields
When you connect the source, select Databricks as the source type and your integration account. Then select the catalog, schema, and table that contains your logs.
Schema mapping
When you connect a source, you map each Datadog field to a column of your table. Datadog reads each row of the table as one log.
| Datadog field | Required |
|---|
| Timestamp | Yes |
Unique identifier (_id) | Yes |
| Status | Yes |
| Message, Host, Service, Source | No |
| Tags | No |
Only columns with a compatible type are listed for each field. For the supported column types, see the tab for your source in the Set up your source section.
Connect a source
- Go to Logs > Configuration > External Sources and click Connect Source.
- On the Source tab, select the source type and the table that contains your logs. The fields depend on the source. Click Next.
- On the Schema Mapping tab, select the column to read each Datadog field from. See the Schema mapping section.
- Click Save.
You can connect multiple tables, each with its own schema mapping. After you save a source, you can edit only its schema mapping. To change the integration account or the table, delete the source and connect it again.
To search an external source, see Federated Logs.
Further reading
Additional helpful documentation, links, and articles: