Overview

If you’re using the Iceberg framework in AWS Glue, you can see metadata from your Iceberg tables in Datadog through the AWS Glue integration. Use this data to monitor table schemas, data freshness, row counts, and table sizes.

Prerequisites

Before you begin, make sure you have:

• An AWS account with Glue Iceberg tables you want to monitor.
• The Datadog AWS integration configured for the account.
• IAM permissions to modify the Datadog role’s policies.
• (Optional) AWS Lake Formation access if you use it to manage table permissions.

Configure the AWS account

1. Navigate to Datadog Data Observability > Settings.

2. Click Configure next to AWS Glue.

3. Select an existing AWS account that is already connected to Datadog, or add a new one. For help adding a new account, see the AWS integration documentation.

Add required IAM permissions

The Data Observability crawler requires additional permissions to monitor Glue Iceberg tables. Attach the following policy to the Datadog IAM role configured for your AWS integration:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetCatalog",
        "glue:GetDatabase",
        "glue:GetDatabases",
        "glue:GetJobRun",
        "glue:GetJobRuns",
        "glue:GetJob",
        "glue:GetJobs",
        "glue:GetTable",
        "glue:GetTables",
        "glue:ListJobs",
        "s3:ListBucket",
        "kms:Decrypt",
        "lakeformation:GetDataAccess"
      ],
      "Resource": ["*"]
    },
    {
      "Sid": "AllowIcebergMetadataOnly",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion"
      ],
      "Resource": [
        "arn:aws:s3:::*/metadata/*"
      ]
    }
  ]
}

(Optional) Restrict access to specific databases and tables

The policy above grants access to all Glue resources. To monitor only specific databases or tables, replace Resource: ["*"] in the example policy above with explicit ARNs of the databases or tables to monitor.

AWS Glue IAM permissions are hierarchical. To access a table, the policy must include the catalog, the database, and the table. Omitting any level results in an access denied error.

Example policies

{
  "Effect": "Allow",
  "Action": [
    "glue:GetCatalog",
    "glue:GetDatabase",
    "glue:GetDatabases",
    "glue:GetTable",
    "glue:GetTables"
  ],
  "Resource": [
    "arn:aws:glue:us-east-1:123456789012:catalog",
    "arn:aws:glue:us-east-1:123456789012:database/production_db",
    "arn:aws:glue:us-east-1:123456789012:database/analytics_db",
    "arn:aws:glue:us-east-1:123456789012:table/production_db/*",
    "arn:aws:glue:us-east-1:123456789012:table/analytics_db/*"
  ]
}

{
  "Effect": "Allow",
  "Action": [
    "glue:GetCatalog",
    "glue:GetDatabase",
    "glue:GetDatabases",
    "glue:GetTable",
    "glue:GetTables"
  ],
  "Resource": [
    "arn:aws:glue:us-east-1:123456789012:catalog",
    "arn:aws:glue:us-east-1:123456789012:database/production_db",
    "arn:aws:glue:us-east-1:123456789012:table/production_db/orders",
    "arn:aws:glue:us-east-1:123456789012:table/production_db/customers",
    "arn:aws:glue:us-east-1:123456789012:table/production_db/events_*"
  ]
}

For more information, see the AWS Glue identity-based policy examples.

(Optional) Configure Lake Formation access

If you use AWS Lake Formation to manage access to your Glue Catalog tables, grant the Datadog role access to the databases and tables you want to monitor.

Use the following commands, replacing the placeholder values with your actual account ID, role name, database name, and S3 bucket:

PRINCIPAL=arn:aws:iam::<YOUR_AWS_ACCOUNT_ID>:role/<YOUR_DATADOG_ROLE_NAME>

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=$PRINCIPAL \
  --resource '{"Database":{"Name":"<YOUR_DATABASE_NAME>"}}' \
  --permissions DESCRIBE SELECT

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=$PRINCIPAL \
  --resource '{"TableWildcard":{"DatabaseName":"<YOUR_DATABASE_NAME>"}}' \
  --permissions DESCRIBE SELECT

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=$PRINCIPAL \
  --resource '{"DataLocation":{"ResourceArn":"arn:aws:s3:::<YOUR_S3_BUCKET_NAME>"}}' \
  --permissions DATA_LOCATION_ACCESS

1. In the AWS Console, navigate to Lake Formation > Data lake permissions.
2. Click Grant.
3. Under Principals, select IAM users and roles and choose your Datadog role.
4. Under LF-Tags or catalog resources, select the database and tables you want to monitor.
5. Under Permissions, select DESCRIBE and SELECT.
6. Click Grant.

Configure the crawler

1. Select the AWS regions where your Glue Iceberg tables are located.

2. Enable the Quality Monitoring for Apache Iceberg toggle.

3. (Optional) Enable the Job Monitoring toggle if you also want to monitor Glue job health and performance.

4. Choose a sync frequency.

5. (Optional) Enter a catalog name if you use nested Glue catalog features. Leave this field empty for the default catalog.

6. Click Save.

Next steps

After you complete the setup, Datadog begins syncing your Glue Iceberg table metadata in the background. Initial syncs can take up to an hour depending on the number of tables in your catalog.

After the sync completes, your tables appear in the Data Catalog.

Further reading

추가 유용한 문서, 링크 및 기사:

Data Observability OverviewDOCUMENTATION AWS IntegrationDOCUMENTATION