--- title: AWS Glue description: >- Connect AWS Glue to Datadog Data Observability to monitor Iceberg table metadata, freshness, and quality. breadcrumbs: >- Docs > Data Observability Overview > Quality Monitoring > Data Lake Integrations > AWS Glue --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # AWS Glue {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}). {% /alert %} {% /callout %} ## Overview{% #overview %} If you're [using the Iceberg framework in AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html), you can see metadata from your Iceberg tables in Datadog through the [AWS Glue integration](https://docs.datadoghq.com/integrations/amazon-glue.md). Use this data to monitor table schemas, data freshness, row counts, and table sizes. ## Prerequisites{% #prerequisites %} Before you begin, make sure you have: - An AWS account with Glue Iceberg tables you want to monitor. - An [AWS account connected in Datadog](https://docs.datadoghq.com/integrations/amazon-web-services.md). - Log forwarding is not required for Data Observability. - IAM permissions to modify the Datadog role's policies. - (Optional) AWS Lake Formation access if you use it to manage table permissions. ## Configure the AWS account{% #configure-the-aws-account %} 1. Navigate to [Datadog Data Observability > Settings](https://app.datadoghq.com/data-obs/settings/integrations). 1. Click Configure next to AWS Glue. {% image source="https://docs.dd-static.net/images/data_observability/aws_glue/settings-configure-button.89c6340c4a9809eaf5de6e335b1be3f7.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/data_observability/aws_glue/settings-configure-button.89c6340c4a9809eaf5de6e335b1be3f7.png?auto=format&fit=max&w=850&dpr=2 2x" alt="AWS Glue configuration option in the Data Observability Settings page" /%} 1. Select an existing AWS account that is already connected to Datadog, or add a new one. For help adding a new account, see the [AWS integration documentation](https://docs.datadoghq.com/integrations/amazon-web-services.md). Then, click Next. {% image source="https://docs.dd-static.net/images/data_observability/aws_glue/account-selection.075b327ec679d7d21d2793c447293894.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/data_observability/aws_glue/account-selection.075b327ec679d7d21d2793c447293894.png?auto=format&fit=max&w=850&dpr=2 2x" alt="AWS account selection dropdown in the configuration flow" /%} ## Add required IAM permissions{% #add-required-iam-permissions %} The Data Observability crawler requires additional permissions to monitor Glue Iceberg tables. Attach the following policy to the Datadog IAM role configured for your AWS integration: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases", "glue:GetJobRun", "glue:GetJobRuns", "glue:GetJob", "glue:GetJobs", "glue:GetTable", "glue:GetTables", "glue:GetTags", "glue:ListJobs", "s3:ListBucket", "kms:Decrypt", "lakeformation:GetDataAccess" ], "Resource": ["*"] }, { "Sid": "AllowIcebergMetadataOnly", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::*/metadata/*" ] } ] } ``` ### (Optional) Restrict access to specific databases and tables{% #optional-restrict-access-to-specific-databases-and-tables %} The policy above grants access to all Glue resources. To monitor only specific databases or tables, replace `Resource: ["*"]` in the example policy above with explicit ARNs of the databases or tables to monitor. AWS Glue IAM permissions are hierarchical. To access a table, the policy must include the catalog, the database, and the table. Omitting any level results in an access denied error. | Resource | ARN format | Example | | -------- | ----------------------------------------------------------------- | ------------------------------------------------------------ | | Catalog | `arn:aws:glue:::catalog` | `arn:aws:glue:us-east-1:123456789012:catalog` | | Database | `arn:aws:glue:::database/` | `arn:aws:glue:us-east-1:123456789012:database/analytics` | | Table | `arn:aws:glue:::table//` | `arn:aws:glue:us-east-1:123456789012:table/analytics/events` | #### Example policies{% #example-policies %} {% tab title="Specific databases" %} To monitor all tables in specific databases, include the catalog, each database, and a wildcard for tables in those databases: ```json { "Effect": "Allow", "Action": [ "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables" ], "Resource": [ "arn:aws:glue:us-east-1:123456789012:catalog", "arn:aws:glue:us-east-1:123456789012:database/production_db", "arn:aws:glue:us-east-1:123456789012:database/analytics_db", "arn:aws:glue:us-east-1:123456789012:table/production_db/*", "arn:aws:glue:us-east-1:123456789012:table/analytics_db/*" ] } ``` {% /tab %} {% tab title="Specific tables" %} To monitor only specific tables, list each table explicitly. You can also use wildcards to match table name patterns: ```json { "Effect": "Allow", "Action": [ "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables" ], "Resource": [ "arn:aws:glue:us-east-1:123456789012:catalog", "arn:aws:glue:us-east-1:123456789012:database/production_db", "arn:aws:glue:us-east-1:123456789012:table/production_db/orders", "arn:aws:glue:us-east-1:123456789012:table/production_db/customers", "arn:aws:glue:us-east-1:123456789012:table/production_db/events_*" ] } ``` The wildcard `events_*` matches tables like `events_clicks`, `events_purchases`, and any other table starting with `events_`. {% /tab %} For more information, see the [AWS Glue identity-based policy examples](https://docs.aws.amazon.com/glue/latest/dg/security_iam_id-based-policy-examples.html). ## (Optional) Configure Lake Formation access{% #optional-configure-lake-formation-access %} If you use AWS Lake Formation to manage access to your Glue Catalog tables, grant the Datadog role access to the databases and tables you want to monitor. {% tab title="AWS CLI" %} Use the following commands, replacing the placeholder values with your actual account ID, role name, database name, and S3 bucket: ```bash PRINCIPAL=arn:aws:iam:::role/ aws lakeformation grant-permissions \ --principal DataLakePrincipalIdentifier=$PRINCIPAL \ --resource '{"Database":{"Name":""}}' \ --permissions DESCRIBE SELECT aws lakeformation grant-permissions \ --principal DataLakePrincipalIdentifier=$PRINCIPAL \ --resource '{"TableWildcard":{"DatabaseName":""}}' \ --permissions DESCRIBE SELECT aws lakeformation grant-permissions \ --principal DataLakePrincipalIdentifier=$PRINCIPAL \ --resource '{"DataLocation":{"ResourceArn":"arn:aws:s3:::"}}' \ --permissions DATA_LOCATION_ACCESS ``` {% /tab %} {% tab title="AWS Console" %} 1. In the AWS Console, navigate to Lake Formation > Data lake permissions. 1. Click Grant. 1. Under Principals, select IAM users and roles and choose your Datadog role. 1. Under LF-Tags or catalog resources, select the database and tables you want to monitor. 1. Under Permissions, select DESCRIBE and SELECT. 1. Click Grant. {% image source="https://docs.dd-static.net/images/data_observability/aws_glue/lakeformation-permissions.83f4fe5ace11a4d159435fe4c7f7c018.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/data_observability/aws_glue/lakeformation-permissions.83f4fe5ace11a4d159435fe4c7f7c018.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Lake Formation permissions grant dialog in AWS Console" /%} {% /tab %} ## Configure the crawler{% #configure-the-crawler %} 1. Select the AWS regions where your Glue Iceberg tables are located. 1. Enable the Quality Monitoring toggle. 1. (Optional) Enable the Job Monitoring toggle if you also want to monitor Glue job health and performance. 1. (Optional) In Excluded namespaces, enter the Glue databases that Datadog should not crawl. 1. Choose a Sync frequency. {% image source="https://docs.dd-static.net/images/data_observability/aws_glue/crawler-configuration.1df78d37f5778ce18f6857fcf2631d58.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/data_observability/aws_glue/crawler-configuration.1df78d37f5778ce18f6857fcf2631d58.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Crawler configuration showing region selection and sync frequency options" /%} 1. (Optional) To calculate metrics that require querying table data, enable Athena queries. 1. Click Add Account. ## Query tables with Amazon Athena{% #query-tables-with-amazon-athena %} {% callout %} # Important note for users on the following Datadog sites: app.datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, app.datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com, uk1.datadoghq.com {% callout %} ##### Preview Querying AWS Glue tables with Amazon Athena is in Preview. Contact your Datadog representative to enable it for your organization. {% /callout %} {% /callout %} By default, Datadog calculates Iceberg table metrics, such as freshness and row count, from Iceberg metadata files without reading table data. To monitor metrics that metadata can't answer, configure Datadog to run queries through an [Amazon Athena workgroup](https://docs.aws.amazon.com/athena/latest/ug/workgroups-manage-queries-control-costs.html) in your AWS account. With Athena queries enabled, you can: - Group metrics by one or more columns. - Apply custom filters to metrics. - Monitor Glue tables that don't use the Iceberg table format. For each monitor, Datadog uses Iceberg metadata when it can calculate the complete requested metric from metadata, and uses Athena only for the remaining work. If Athena queries are not configured, Datadog skips monitors that require a query and continues to calculate metadata-based metrics. **Note**: Datadog runs Athena queries in your AWS account. Athena bills these queries to your account based on the workgroup's pricing model. ### Add Athena IAM permissions{% #add-athena-iam-permissions %} Athena reads table data files, not only Iceberg metadata. Add the following statements to the Datadog IAM role, alongside the required IAM permissions: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AthenaQueryExecution", "Effect": "Allow", "Action": [ "athena:GetQueryExecution", "athena:GetQueryResults", "athena:GetWorkGroup", "athena:StartQueryExecution", "athena:StopQueryExecution", "glue:GetPartition", "glue:GetPartitions" ], "Resource": ["*"] }, { "Sid": "AthenaTableDataAndResults", "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:PutObject", "kms:GenerateDataKey" ], "Resource": ["*"] } ] } ``` To restrict access, scope the Athena actions to your workgroup ARN (`arn:aws:athena:::workgroup/`), `s3:GetObject` to the buckets that store your table data, and `s3:PutObject` to the Athena query result location. If you use Lake Formation, the `SELECT` permission described in Configure Lake Formation access is required for Athena to read table data. ### Enable Athena queries{% #enable-athena-queries %} 1. In the AWS Glue configuration, under Quality Monitoring, enable the Enable Athena queries toggle. 1. Enter the Athena workgroup that Datadog uses to run queries. 1. (Optional) Enter an Athena result location, such as `s3://example-bucket/athena-results/`. Use an S3 path when the workgroup does not manage query results. 1. Click Add Account, or Save if you are editing an account that is already configured. This setting controls only the queries that Datadog runs to calculate metrics. To monitor the queries that your own users and pipelines run in Athena, see [Amazon Athena Query History](https://docs.datadoghq.com/data_observability/quality_monitoring/data_lakes/amazon_athena.md). ## Next steps{% #next-steps %} After you complete the setup, Datadog begins syncing your Glue Iceberg table metadata in the background. Initial syncs can take up to an hour depending on the number of tables in your catalog. After the sync completes, your tables appear in the [Data Catalog](https://app.datadoghq.com/data-obs/catalog?integration=awsglue%2Fdatabase_account). You can also create a [Data Observability monitor](https://docs.datadoghq.com/monitors/types/data_observability.md) to start alerting on freshness and row count. ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Data Observability Overview](https://docs.datadoghq.com/data_observability.md) - [AWS Integration](https://docs.datadoghq.com/integrations/amazon-web-services.md) - [Data Observability Monitors](https://docs.datadoghq.com/monitors/types/data_observability.md)