--- title: AWS Glue description: >- A managed ETL service that categorizes, cleans, enriches, and moves data between different data stores. breadcrumbs: Docs > Integrations > AWS Glue --- # AWS Glue Integration version1.0.1 ## Overview{% #overview %} AWS Glue is a fully managed ETL (extract, transform, and load) service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores. Enable this integration to see all your Glue metrics in Datadog. ## Setup{% #setup %} ### Installation{% #installation %} If you haven't already, set up the [Amazon Web Services integration](https://docs.datadoghq.com/integrations/amazon_web_services.md) first. ### Metric collection{% #metric-collection %} 1. In the [AWS integration page](https://app.datadoghq.com/integrations/amazon-web-services), ensure that `Glue` is enabled under the `Metric Collection` tab. 1. Install the [Datadog - AWS Glue integration](https://app.datadoghq.com/integrations/amazon-glue). ### Log collection{% #log-collection %} #### Enable logging{% #enable-logging %} Configure AWS Glue to send logs either to an S3 bucket or to CloudWatch. AWS Glue sends job logs to CloudWatch by default. The log group depends on the job type: - Spark jobs: `/aws-glue/jobs/output` and `/aws-glue/jobs/error` - Python Shell jobs: `/aws-glue/python-jobs/output` and `/aws-glue/python-jobs/error` - Ray jobs: `/aws-glue/ray/jobs/*` **Notes**: - Datadog's [automatic trigger setup](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function.md?tab=awsconsole#automatically-set-up-triggers) is available for CloudWatch log groups only. For S3 buckets, use the [manual trigger setup](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function.md#collecting-logs-from-s3-buckets). If you use the automatic trigger setup, ensure that your [Datadog IAM policy](https://docs.datadoghq.com/integrations/amazon_web_services.md#installation) has the following permissions: | AWS Permission | Description | | ------------------- | -------------------------------------------------- | | `glue:ListJobs` | Used to list available Glue jobs. | | `glue:GetJobs` | Used to get details on all Glue jobs. | | `glue:GetJob` | Used to get detailed information about a Glue job. | | `glue:BatchGetJobs` | Used to retrieve metadata for multiple Glue jobs. | - If you log to a S3 bucket, make sure that `amazon_glue` is set as *Target prefix*. - If you use the `--custom-logGroup-prefix` argument for your Glue Jobs, use the log group name prefix `/aws-glue/` for Datadog to identify the source of the logs and parse them automatically. Otherwise, they cannot be identified properly and will be tagged as `source:cloudwatch`. #### Send logs to Datadog{% #send-logs-to-datadog %} 1. If you haven't already, set up the [Datadog Forwarder Lambda function](https://docs.datadoghq.com/logs/guide/forwarder.md). 1. Once the Lambda function is installed, manually add a trigger on the S3 bucket or CloudWatch log group that contains your AWS Glue logs in the AWS console: - [Add a manual trigger on the S3 bucket](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function.md#collecting-logs-from-s3-buckets) - [Add a manual trigger on the CloudWatch Log Group](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function.md#collecting-logs-from-cloudwatch-log-group) ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **aws.glue.driver.executor\_allocation\_manager.executors.number\_all\_executors**(gauge) | The number of actively running job executors. | | **aws.glue.driver.executor\_allocation\_manager.executors.number\_max\_needed\_executors**(gauge) | The number of maximum (actively running and pending) job executors needed to satisfy the current load. | | **aws.glue.glue\_alljvm\_heap\_usage**(gauge) | The average fraction of memory used by the JVM heap for this driver (scale: 0-1) for all executors.*Shown as percent* | | **aws.glue.glue\_alljvm\_heap\_used**(gauge) | The number of memory bytes used by the JVM heap for all executors.*Shown as byte* | | **aws.glue.glue\_alls\_3filesystem\_readbytes**(gauge) | The average number of bytes read from Amazon S3 all executors since the previous report. | | **aws.glue.glue\_allsystem\_cpu\_system\_load**(gauge) | The average fraction of CPU system load used (scale: 0-1) by all executors.*Shown as percent* | | **aws.glue.glue\_driver\_aggregate\_bytes\_read**(count) | The number of bytes read from all data sources by all completed Spark tasks running in all executors.*Shown as byte* | | **aws.glue.glue\_driver\_aggregate\_elapsed\_time**(count) | The ETL elapsed time in milliseconds (does not include the job bootstrap times).*Shown as millisecond* | | **aws.glue.glue\_driver\_aggregate\_num\_completed\_stages**(count) | The number of completed stages in the job. | | **aws.glue.glue\_driver\_aggregate\_num\_completed\_tasks**(count) | The number of completed tasks in the job. | | **aws.glue.glue\_driver\_aggregate\_num\_failed\_tasks**(count) | The number of failed tasks. | | **aws.glue.glue\_driver\_aggregate\_num\_killed\_tasks**(count) | The number of tasks killed. | | **aws.glue.glue\_driver\_aggregate\_records\_read**(count) | The number of records read from all data sources by all completed Spark tasks running in all executors. | | **aws.glue.glue\_driver\_aggregate\_shuffle\_bytes\_written**(count) | The number of bytes written by all executors to shuffle data between them since the previous report. | | **aws.glue.glue\_driver\_aggregate\_shuffle\_local\_bytes\_read**(count) | The number of bytes read by all executors to shuffle data between them since the previous report. | | **aws.glue.glue\_driver\_block\_manager\_disk\_disk\_space\_used\_mb**(gauge) | The average number of megabytes of disk spaced used across all executors. | | **aws.glue.glue\_driver\_jvm\_heap\_usage**(gauge) | The average fraction of memory used by the JVM heap for this driver (scale: 0-1) for driver.*Shown as percent* | | **aws.glue.glue\_driver\_jvm\_heap\_used**(gauge) | The number of memory bytes used by the JVM heap for the driver.*Shown as byte* | | **aws.glue.glue\_driver\_s3\_filesystem\_readbytes**(gauge) | The average number of bytes read from Amazon S3 by the driver since the previous report. | | **aws.glue.glue\_driver\_s3\_filesystem\_writebytes**(gauge) | The average number of bytes written to Amazon S3 by the driver since the previous report. | | **aws.glue.glue\_driver\_system\_cpu\_system\_load**(gauge) | The average fraction of CPU system load used (scale: 0-1) by the driver.*Shown as percent* | | **aws.glue.glue\_executor\_id\_jvm\_heap\_usage**(gauge) | The average fraction of memory used by the JVM heap for this driver (scale: 0-1) for executor identified.*Shown as percent* | | **aws.glue.glue\_executor\_id\_jvm\_heap\_used**(gauge) | The number of memory bytes used by the JVM heap for the executor identified.*Shown as byte* | | **aws.glue.glue\_executor\_id\_system\_cpu\_system\_load**(gauge) | The average fraction of CPU system load used (scale: 0-1) by the executor identified.*Shown as percent* | | **aws.glue.glue\_executor\_ids\_3\_filesystem\_readbytes**(gauge) | The average number of bytes read from Amazon S3 by the executor identified since the previous report. | | **aws.glue.glue\_executor\_ids\_3\_filesystem\_writebytes**(gauge) | The average number of bytes written to Amazon S3 by the executor identified since the previous report. | ### Events{% #events %} The AWS Glue integration does not include any events. ### Service Checks{% #service-checks %} The AWS Glue integration does not include any service checks. ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).