--- title: Fly.io description: Monitor your Fly.io apps and machines. breadcrumbs: Docs > Integrations > Fly.io --- # Fly.io Supported OS Integration version3.4.1 ## Overview{% #overview %} This check monitors [Fly.io](https://fly.io/) metrics through the Datadog Agent. **Minimum Agent version:** 7.57.0 ## Setup{% #setup %} Follow the instructions below to install and configure this check for an Agent running on a Fly application. ### Installation{% #installation %} The Fly.io check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package. We recommend deploying one Fly.io application dedicated to running the Datadog Agent. This agent can run the Fly.io check, which collects [Prometheus metrics](https://fly.io/docs/metrics-and-logs/metrics/#prometheus-on-fly-io) as well as some additional data from the [Machines API](https://fly.io/docs/machines/api/). Additionally, you can configure the Agent to receive traces and custom metrics from all of your Fly.io applications inside the organization. #### Deploying the Agent as a Fly.io application{% #deploying-the-agent-as-a-flyio-application %} 1. Create a new application in Fly.io with the image set as the [Datadog Agent](https://console.cloud.google.com/artifacts/docker/datadoghq/us/gcr.io/agent) when launching, or provide the image in the `fly.toml` file: ``` [build] image = 'gcr.io/datadoghq/agent:7' ``` 1. Set a [secret](https://fly.io/docs/flyctl/secrets/) for your Datadog API key called `DD_API_KEY`, and optionally your [site](https://docs.datadoghq.com/agent/troubleshooting/site.md) as `DD_SITE`. 1. Create a [read-only](https://fly.io/docs/flyctl/tokens-create-readonly/) authentication token. 1. In your app's directory, create a `conf.yaml` file for the Fly.io integration, configure the integration, and mount it in the Agent's `conf.d/fly_io.d/` directory as `conf.yaml`: ``` instances: - empty_default_hostname: true headers: Authorization: FlyV1 machines_api_endpoint: http://_api.internal:4280 org_slug: ``` 1. [Deploy](https://fly.io/docs/flyctl/deploy/) your app. **Note**: To collect traces and custom metrics from your applications, see Application traces. ### Configuration{% #configuration %} 1. Edit the `fly_io.d/conf.yaml` file, located in the `conf.d/` folder at the root of your Agent's configuration directory, to start collecting your Fly.io performance data. See the [sample fly_io.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/fly_io/datadog_checks/fly_io/data/conf.yaml.example) for all available configuration options. 1. [Restart the Agent](https://docs.datadoghq.com/agent/guide/agent-commands.md#start-stop-and-restart-the-agent). ### Validation{% #validation %} [Run the Agent's status subcommand](https://docs.datadoghq.com/agent/guide/agent-commands.md#agent-status-and-information) and look for `fly_io` under the Checks section. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **fly\_io.app.concurrency**(gauge) | | **fly\_io.app.connect\_time.bucket**(count) | *Shown as second* | | **fly\_io.app.connect\_time.count**(count) | | **fly\_io.app.connect\_time.sum**(count) | *Shown as second* | | **fly\_io.app.count**(gauge) | Count of apps | | **fly\_io.app.http\_response\_time.bucket**(count) | *Shown as second* | | **fly\_io.app.http\_response\_time.count**(count) | | **fly\_io.app.http\_response\_time.sum**(count) | *Shown as second* | | **fly\_io.app.http\_responses.count**(gauge) | *Shown as response* | | **fly\_io.app.tcp\_connects.count**(gauge) | | **fly\_io.app.tcp\_disconnects.count**(gauge) | | **fly\_io.edge.data\_in**(gauge) | *Shown as byte* | | **fly\_io.edge.data\_out**(gauge) | *Shown as byte* | | **fly\_io.edge.http\_response\_time.bucket**(count) | *Shown as second* | | **fly\_io.edge.http\_response\_time.count**(count) | | **fly\_io.edge.http\_response\_time.sum**(count) | *Shown as second* | | **fly\_io.edge.http\_responses.count**(gauge) | *Shown as response* | | **fly\_io.edge.tcp\_connects.count**(gauge) | | **fly\_io.edge.tcp\_disconnects.count**(gauge) | | **fly\_io.edge.tls\_handshake\_errors**(gauge) | *Shown as error* | | **fly\_io.edge.tls\_handshake\_time.bucket**(count) | *Shown as second* | | **fly\_io.edge.tls\_handshake\_time.count**(count) | | **fly\_io.edge.tls\_handshake\_time.sum**(count) | *Shown as second* | | **fly\_io.instance.cpu.count**(count) | The amount of time each CPU (cpu_id) has spent performing different kinds of work (mode) in centiseconds | | **fly\_io.instance.disk.io\_in\_progress**(gauge) | Incremented as requests are given to appropriate struct request_queue and decremented as they finish. | | **fly\_io.instance.disk.reads\_completed.count**(count) | This is the total number of reads completed successfully. | | **fly\_io.instance.disk.reads\_merged.count**(count) | Reads and writes which are adjacent to each other may be merged for efficiency. This field lets you know how often this was done. | | **fly\_io.instance.disk.sectors\_read.count**(count) | This is the total number of sectors read successfully. | | **fly\_io.instance.disk.sectors\_written.count**(count) | This is the total number of sectors written successfully. | | **fly\_io.instance.disk.time\_io.count**(count) | Counts jiffies when at least one request was started or completed. If request runs more than 2 jiffies then some I/O time might be not accounted in case of concurrent requests.*Shown as millisecond* | | **fly\_io.instance.disk.time\_io\_weighted.count**(count) | Incremented at each I/O start, I/O completion, I/O merge, or read of these stats by the number of I/Os in progress (field 9) times the number of milliseconds spent doing I/O since the last update of this field.*Shown as millisecond* | | **fly\_io.instance.disk.time\_reading.count**(count) | This is the total number of milliseconds spent by all reads.*Shown as millisecond* | | **fly\_io.instance.disk.time\_writing.count**(count) | This is the total number of milliseconds spent by all writes*Shown as millisecond* | | **fly\_io.instance.disk.writes\_completed.count**(count) | This is the total number of writes completed successfully. | | **fly\_io.instance.disk.writes\_merged.count**(count) | Reads and writes which are adjacent to each other may be merged for efficiency. This field lets you know how often this was done. | | **fly\_io.instance.filefd.allocated**(gauge) | Number of allocated file descriptors | | **fly\_io.instance.filefd.max**(gauge) | Number of maximum file descriptors | | **fly\_io.instance.filesystem.block\_size**(gauge) | File system block size. | | **fly\_io.instance.filesystem.blocks**(gauge) | Total number of blocks on file system | | **fly\_io.instance.filesystem.blocks\_avail**(gauge) | Total number of available blocks. | | **fly\_io.instance.filesystem.blocks\_free**(gauge) | Total number of free blocks. | | **fly\_io.instance.load.avg**(gauge) | System load average measuring the number of processes in the system run queue, with samples representing averages over 1, 5, and 15 minutes.*Shown as process* | | **fly\_io.instance.memory.active**(gauge) | Memory that has been used more recently and usually not reclaimed unless absolutely necessary.*Shown as byte* | | **fly\_io.instance.memory.buffers**(gauge) | Relatively temporary storage for raw disk blocks*Shown as byte* | | **fly\_io.instance.memory.cached**(gauge) | In-memory cache for files read from the disk (the pagecache) as well as tmpfs & shmem. Doesn't include SwapCached.*Shown as byte* | | **fly\_io.instance.memory.dirty**(gauge) | Memory which is waiting to get written back to the disk*Shown as byte* | | **fly\_io.instance.memory.inactive**(gauge) | Memory which has been less recently used. It is more eligible to be reclaimed for other purposes*Shown as byte* | | **fly\_io.instance.memory.mem\_available**(gauge) | An estimate of how much memory is available for starting new applications, without swapping.*Shown as byte* | | **fly\_io.instance.memory.mem\_free**(gauge) | Total free RAM.*Shown as byte* | | **fly\_io.instance.memory.mem\_total**(gauge) | Total usable RAM (i.e. physical RAM minus a few reserved bits and the kernel binary code)*Shown as byte* | | **fly\_io.instance.memory.pressure\_full**(gauge) | Memory pressure for all processes | | **fly\_io.instance.memory.pressure\_some**(gauge) | Memory pressure for at least one process | | **fly\_io.instance.memory.shmem**(gauge) | Total memory used by shared memory (shmem) and tmpfs*Shown as byte* | | **fly\_io.instance.memory.slab**(gauge) | in-kernel data structures cache*Shown as byte* | | **fly\_io.instance.memory.swap\_cached**(gauge) | Memory that once was swapped out, is swapped back in but still also is in the swapfile*Shown as byte* | | **fly\_io.instance.memory.swap\_free**(gauge) | Memory which has been evicted from RAM, and is temporarily on the disk*Shown as byte* | | **fly\_io.instance.memory.swap\_total**(gauge) | total amount of swap space available*Shown as byte* | | **fly\_io.instance.memory.vmalloc\_chunk**(gauge) | largest contiguous block of vmalloc area which is free*Shown as byte* | | **fly\_io.instance.memory.vmalloc\_total**(gauge) | total size of vmalloc virtual address space*Shown as byte* | | **fly\_io.instance.memory.vmalloc\_used**(gauge) | amount of vmalloc area which is used*Shown as byte* | | **fly\_io.instance.memory.writeback**(gauge) | Memory which is actively being written back to the disk*Shown as byte* | | **fly\_io.instance.net.recv\_bytes.count**(count) | Number of good bytes received by the interface.*Shown as byte* | | **fly\_io.instance.net.recv\_compressed.count**(count) | Number of correctly received compressed packets. | | **fly\_io.instance.net.recv\_drop.count**(count) | Number of packets received but not processed, e.g. due to lack of resources or unsupported protocol.*Shown as packet* | | **fly\_io.instance.net.recv\_errs.count**(count) | Total number of bad packets received on this network device.*Shown as packet* | | **fly\_io.instance.net.recv\_fifo.count**(count) | Receiver FIFO overflow event counter. | | **fly\_io.instance.net.recv\_frame.count**(count) | Receiver frame alignment errors. | | **fly\_io.instance.net.recv\_multicast.count**(count) | Multicast packets received.*Shown as packet* | | **fly\_io.instance.net.recv\_packets.count**(count) | Number of good packets received by the interface.*Shown as packet* | | **fly\_io.instance.net.sent\_bytes.count**(count) | Number of good transmitted bytes.*Shown as byte* | | **fly\_io.instance.net.sent\_carrier.count**(count) | Number of frame transmission errors due to loss of carrier during transmission. | | **fly\_io.instance.net.sent\_colls.count**(count) | Number of collisions during packet transmissions. | | **fly\_io.instance.net.sent\_compressed.count**(count) | Number of transmitted compressed packets. | | **fly\_io.instance.net.sent\_drop.count**(count) | Number of packets dropped on their way to transmission, e.g. due to lack of resources.*Shown as packet* | | **fly\_io.instance.net.sent\_errs.count**(count) | Total number of transmit problems. | | **fly\_io.instance.net.sent\_fifo.count**(count) | Sent FIFO overflow event counter. | | **fly\_io.instance.net.sent\_packets.count**(count) | Number of packets successfully transmitted.*Shown as packet* | | **fly\_io.instance.up**(gauge) | Reports 1 if the VM is reporting correctly | | **fly\_io.instance.volume.size**(gauge) | Volume size in bytes.*Shown as byte* | | **fly\_io.instance.volume.used**(gauge) | Percentage of volume used.*Shown as byte* | | **fly\_io.machine.count**(gauge) | Count of running machines | | **fly\_io.machine.cpus.count**(gauge) | Number of cpus | | **fly\_io.machine.gpus.count**(gauge) | Number of gpus | | **fly\_io.machine.memory**(gauge) | Memory of a machine*Shown as megabyte* | | **fly\_io.machine.swap\_size**(gauge) | Swap space to reserve for the Fly Machine*Shown as megabyte* | | **fly\_io.machines\_api.up**(gauge) | Whether the check can access the machines API or not | | **fly\_io.pg.database.size**(gauge) | Database size*Shown as byte* | | **fly\_io.pg.replication.lag**(gauge) | Replication lag | | **fly\_io.pg\_stat.activity.count**(gauge) | number of connections in this state | | **fly\_io.pg\_stat.activity.max\_tx\_duration**(gauge) | max duration in seconds any active transaction has been running*Shown as second* | | **fly\_io.pg\_stat.archiver.archived\_count**(gauge) | Number of WAL files that have been successfully archived | | **fly\_io.pg\_stat.archiver.failed\_count**(gauge) | Number of failed attempts for archiving WAL files | | **fly\_io.pg\_stat.bgwriter.buffers\_alloc**(gauge) | Number of buffers allocated | | **fly\_io.pg\_stat.bgwriter.buffers\_backend**(gauge) | Number of buffers written directly by a backend | | **fly\_io.pg\_stat.bgwriter.buffers\_backend\_fsync**(gauge) | Number of times a backend had to execute its own fsync call (normally the background writer handles those even when the backend does its own write) | | **fly\_io.pg\_stat.bgwriter.buffers\_checkpoint**(gauge) | Number of buffers written during checkpoints | | **fly\_io.pg\_stat.bgwriter.buffers\_clean**(gauge) | Number of buffers written by the background writer | | **fly\_io.pg\_stat.bgwriter.checkpoint\_sync\_time**(gauge) | Total amount of time that has been spent in the portion of checkpoint processing where files are synchronized to disk, in milliseconds*Shown as millisecond* | | **fly\_io.pg\_stat.bgwriter.checkpoint\_write\_time**(gauge) | Total amount of time that has been spent in the portion of checkpoint processing where files are written to disk, in milliseconds*Shown as millisecond* | | **fly\_io.pg\_stat.bgwriter.checkpoints\_req**(gauge) | Number of requested checkpoints that have been performed | | **fly\_io.pg\_stat.bgwriter.checkpoints\_timed**(gauge) | Number of scheduled checkpoints that have been performed | | **fly\_io.pg\_stat.bgwriter.maxwritten\_clean**(gauge) | Number of times the background writer stopped a cleaning scan because it had written too many buffers | | **fly\_io.pg\_stat.bgwriter.stats\_reset**(gauge) | Time at which these statistics were last reset | | **fly\_io.pg\_stat.database.blk\_read\_time**(gauge) | Time spent reading data file blocks by backends in this database, in milliseconds*Shown as millisecond* | | **fly\_io.pg\_stat.database.blk\_write\_time**(gauge) | Time spent writing data file blocks by backends in this database, in milliseconds*Shown as millisecond* | | **fly\_io.pg\_stat.database.blks\_hit**(gauge) | Number of times disk blocks were found already in the buffer cache, so that a read was not necessary (this only includes hits in the PostgreSQL buffer cache, not the operating system's file system cache) | | **fly\_io.pg\_stat.database.blks\_read**(gauge) | Number of disk blocks read in this database | | **fly\_io.pg\_stat.database.conflicts**(gauge) | Number of queries canceled due to conflicts with recovery in this database. Conflicts occur only on standby servers | | **fly\_io.pg\_stat.database.conflicts\_confl\_bufferpin**(gauge) | Number of queries in this database that have been canceled due to pinned buffers | | **fly\_io.pg\_stat.database.conflicts\_confl\_deadlock**(gauge) | Number of queries in this database that have been canceled due to deadlocks | | **fly\_io.pg\_stat.database.conflicts\_confl\_lock**(gauge) | Number of queries in this database that have been canceled due to lock timeouts | | **fly\_io.pg\_stat.database.conflicts\_confl\_snapshot**(gauge) | Number of queries in this database that have been canceled due to old snapshots | | **fly\_io.pg\_stat.database.conflicts\_confl\_tablespace**(gauge) | Number of queries in this database that have been canceled due to dropped tablespaces | | **fly\_io.pg\_stat.database.deadlocks**(gauge) | Number of deadlocks detected in this database | | **fly\_io.pg\_stat.database.numbackends**(gauge) | Number of backends currently connected to this database. This is the only column in this view that returns a value reflecting current state; all other columns return the accumulated values since the last reset. | | **fly\_io.pg\_stat.database.stats\_reset**(gauge) | Time at which these statistics were last reset | | **fly\_io.pg\_stat.database.tup\_deleted**(gauge) | Number of rows deleted by queries in this database | | **fly\_io.pg\_stat.database.tup\_fetched**(gauge) | Number of rows fetched by queries in this database | | **fly\_io.pg\_stat.database.tup\_inserted**(gauge) | Number of rows inserted by queries in this database | | **fly\_io.pg\_stat.database.tup\_returned**(gauge) | Number of rows returned by queries in this database | | **fly\_io.pg\_stat.database.tup\_updated**(gauge) | Number of rows updated by queries in this database | | **fly\_io.pg\_stat.database.xact\_commit**(gauge) | Number of transactions in this database that have been committed | | **fly\_io.pg\_stat.database.xact\_rollback**(gauge) | Number of transactions in this database that have been rolled back | | **fly\_io.pg\_stat.replication.pg\_current\_wal\_lsn\_bytes**(gauge) | WAL position in bytes*Shown as byte* | | **fly\_io.pg\_stat.replication.pg\_wal\_lsn\_diff**(gauge) | Lag in bytes between master and slave*Shown as byte* | | **fly\_io.pg\_stat.replication.reply\_time**(gauge) | Send time of last reply message received from standby server | | **fly\_io.volume.block\_size**(gauge) | The size of each memory block in bytes*Shown as byte* | | **fly\_io.volume.blocks.count**(gauge) | The total number of blocks in the volume | | **fly\_io.volume.blocks\_avail**(gauge) | The number of blocks available for data in the volume | | **fly\_io.volume.blocks\_free**(gauge) | The total number of blocks free for data and root user ops | | **fly\_io.volume.created**(gauge) | Whether the volume has been created or not | | **fly\_io.volume.encrypted**(gauge) | Whether the volume is encrypted or not | | **fly\_io.volume.size**(gauge) | The size of the volume in GB*Shown as gigabyte* | ### Events{% #events %} The Fly.io integration does not include any events. ### Service Checks{% #service-checks %} The Fly.io integration does not include any service checks. ### Application traces{% #application-traces %} Follow these steps to collect traces for an application in your Fly.io environment. 1. [Instrument](https://docs.datadoghq.com/tracing/trace_collection.md#instrumentation-types) your application. 1. Deploy the Datadog Agent as a Fly.io application. 1. Set the required environment variables in the `fly.toml` or `Dockerfile` of your application and deploy the app. Set the following as an environment variable to submit metrics to the Datadog Agent application: ``` [env] DD_AGENT_HOST=".internal" ``` Set the following environment variable to ensure you report the same host for logs and metrics: ``` DD_TRACE_REPORT_HOSTNAME="true" ``` To utilize [unified service tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md?tab=docker#configuration-1), set these environment variables: ``` DD_SERVICE="APP_NAME" DD_ENV="ENV_NAME" DD_VERSION="VERSION" ``` To correlate logs and traces, follow these [steps](https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces.md) and set this environment variable: ``` DD_LOGS_INJECTION="true" ``` 1. Set the following environment variables in your Datadog Agent application's `fly.toml` and deploy the app: ``` [env] DD_APM_ENABLED = "true" DD_APM_NON_LOCAL_TRAFFIC = "true" DD_DOGSTATSD_NON_LOCAL_TRAFFIC = "true" DD_BIND_HOST = "fly-global-services" ``` **Note**: Ensure that the settings on your Fly.io instances do not publicly expose the ports for APM and DogStatsD, if enabled. ### Log collection{% #log-collection %} Use the [fly_logs_shipper](https://github.com/superfly/fly-log-shipper) to collect logs from your Fly.io applications. 1. Clone the logs shipper [project](https://github.com/superfly/fly-log-shipper). 1. Modify the `vector-configs/vector.toml` file to set the logs source as `fly_io`: ``` [transforms.log_json] type = "remap" inputs = ["nats"] source = ''' . = parse_json!(.message) .ddsource = "fly-io" .host = .fly.app.instance .env = ''' ``` This configuration will parse basic fly-specific log attributes. To fully parse all log attributes, set `ddsource` to a [known logs integration](https://docs.datadoghq.com/logs/log_configuration/pipelines.md?tab=source#integration-pipeline-library) on a per-app basis using [vector transforms](https://vector.dev/docs/reference/configuration/transforms/lua/). Set [secrets](https://fly.io/docs/flyctl/secrets/) for [NATS](https://github.com/superfly/fly-log-shipper?tab=readme-ov-file#nats-source-configuration): `ORG` and `ACCESS_TOKEN` Set [secrets](https://fly.io/docs/flyctl/secrets/) for [Datadog](https://github.com/superfly/fly-log-shipper?tab=readme-ov-file#datadog): `DATADOG_API_KEY` and `DATADOG_SITE` [Deploy](https://docs.datadoghq.com/agent/guide/agent-commands.md#agent-status-and-information) the logs shipper app. ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).