--- title: Troubleshoot Database Monitoring setup for MariaDB description: Troubleshoot Database Monitoring setup breadcrumbs: >- Docs > Database Monitoring > Setting up MariaDB > Troubleshoot Database Monitoring setup for MariaDB --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Troubleshoot Database Monitoring setup for MariaDB This page details common issues with setting up and using Database Monitoring with MariaDB, and how to resolve them. Datadog recommends staying on the latest stable Agent version and adhering to the latest [setup documentation](https://docs.datadoghq.com/database_monitoring/setup_mariadb.md), as it can change with Agent version releases. ## Diagnosing common problems{% #diagnosing-common-problems %} ### No data is showing after configuring Database Monitoring{% #no-data-is-showing-after-configuring-database-monitoring %} If you do not see any data after following the [setup instructions](https://docs.datadoghq.com/database_monitoring/setup_mariadb.md) and configuring the Agent, there is most likely an issue with the Agent configuration or API key. Follow the [troubleshooting guide](https://docs.datadoghq.com/agent/troubleshooting.md), which helps confirm you are receiving data from the Agent. If you are receiving other data such as system metrics, but not Database Monitoring data (such as query metrics and query samples), there is probably an issue with the Agent or database configuration. Compare your Agent configuration to the example in the [setup instructions](https://docs.datadoghq.com/database_monitoring/setup_mariadb.md) to check that it matches, and double-check the location of the configuration files. To debug, start by running the [Agent status command](https://docs.datadoghq.com/agent/configuration/agent-commands.md?tab=agentv6v7#agent-status-and-information) to collect debugging information about data collected and sent to Datadog. Check the `Config Errors` section to confirm the configuration file is valid. For instance, the following indicates a missing instance configuration or invalid file: ``` Config Errors ============== mysql ----- Configuration file contains no valid instances ``` If the configuration is valid, the output looks like this: ``` ========= Collector ========= Running Checks ============== mysql (5.0.4) ------------- Instance ID: mysql:505a0dd620ccaa2a Configuration Source: file:/etc/datadog-agent/conf.d/mysql.d/conf.yaml Total Runs: 32,439 Metric Samples: Last Run: 175, Total: 5,833,916 Events: Last Run: 0, Total: 0 Database Monitoring Query Metrics: Last Run: 2, Total: 51,074 Database Monitoring Query Samples: Last Run: 1, Total: 74,451 Service Checks: Last Run: 3, Total: 95,993 Average Execution Time : 1.798s Last Execution Date : 2021-07-29 19:28:21 UTC (1627586901000) Last Successful Execution Date : 2021-07-29 19:28:21 UTC (1627586901000) metadata: flavor: MariaDB version.build: unspecified version.major: 10 version.minor: 11 version.patch: 6 version.raw: 10.11.6-MariaDB version.scheme: semver ``` Check that these lines are in the output and have values greater than zero: ``` Database Monitoring Query Metrics: Last Run: 2, Total: 51,074 Database Monitoring Query Samples: Last Run: 1, Total: 74,451 ``` When you are confident the Agent configuration is correct, [check the Agent logs](https://docs.datadoghq.com/agent/configuration/agent-log-files.md) for warnings or errors attempting to run the database integrations. You can also explicitly execute a check by running the `check` CLI command on the Datadog Agent and inspecting the output for errors: ```bash # For self-hosted installations of the Agent DD_LOG_LEVEL=debug DBM_THREADED_JOB_RUN_SYNC=true datadog-agent check mysql -t 2 # For container-based installations of the Agent DD_LOG_LEVEL=debug DBM_THREADED_JOB_RUN_SYNC=true agent check mysql -t 2 ``` ### Queries are missing explain plans{% #queries-are-missing-explain-plans %} Some or all queries may not have plans available. This can be due to unsupported query commands, queries made by unsupported client applications, an outdated Agent, or incomplete database setup. Below are possible causes for missing explain plans. #### Missing event statements consumer{% #events-statements-consumer-missing %} To capture explain plans, you must enable an event statements consumer. You can do this by adding the following option to your configuration files (for example, `mysql.conf`): ``` performance-schema-consumer-events-statements-current=ON ``` Datadog additionally recommends enabling the following: ``` performance-schema-consumer-events-statements-history-long=ON ``` This option enables the tracking of a larger number of recent queries across all threads. Turning it on increases the likelihood of capturing execution details from infrequent queries. #### Missing explain plan procedure{% #explain-plan-procedure-missing %} The Agent requires the procedure `datadog.explain_statement(...)` to exist in the `datadog` schema. Read the [setup instructions](https://docs.datadoghq.com/database_monitoring/setup_mariadb.md) for details on the creation of the `datadog` schema. Create the `explain_statement` procedure to enable the Agent to collect explain plans: ```sql DELIMITER $$ CREATE PROCEDURE datadog.explain_statement(IN query TEXT) SQL SECURITY DEFINER BEGIN SET @explain := CONCAT('EXPLAIN FORMAT=json ', query); PREPARE stmt FROM @explain; EXECUTE stmt; DEALLOCATE PREPARE stmt; END $$ DELIMITER ; ``` #### Missing full qualified explain plan procedure{% #explain-plan-fq-procedure-missing %} The Agent requires the procedure `explain_statement(...)` to exist in **all schemas** the Agent can collect samples from. Create this procedure **in every schema** from which you want to collect explain plans. Replace `` with your database schema: ```sql DELIMITER $$ CREATE PROCEDURE .explain_statement(IN query TEXT) SQL SECURITY DEFINER BEGIN SET @explain := CONCAT('EXPLAIN FORMAT=json ', query); PREPARE stmt FROM @explain; EXECUTE stmt; DEALLOCATE PREPARE stmt; END $$ DELIMITER ; GRANT EXECUTE ON PROCEDURE .explain_statement TO datadog@'%'; ``` #### Agent is running an unsupported version{% #agent-is-running-an-unsupported-version %} Check that the Agent is running version 7.61.0 or newer. Datadog recommends regular updates of the Agent to take advantage of new features, performance improvements, and security updates. #### Queries are truncated{% #queries-are-truncated %} See the section on truncated query samples for instructions on how to increase the size of sample query text. #### Query cannot be explained{% #query-cannot-be-explained %} Some queries such as BEGIN, COMMIT, SHOW, USE, and ALTER queries cannot yield a valid explain plan from the database. Only SELECT, UPDATE, INSERT, DELETE, and REPLACE queries have support for explain plans. #### Query is relatively infrequent or executes fast{% #query-is-relatively-infrequent-or-executes-fast %} The query may not have been sampled for selection because it does not represent a significant proportion of the database's total execution time. Try [raising the sampling rates](https://docs.datadoghq.com/database_monitoring/setup_mariadb/advanced_configuration.md) to capture the query. ### Query metrics are missing{% #query-metrics-are-missing %} Before following these steps to diagnose missing query metric data, check that the Agent is running successfully and you have followed the steps to diagnose missing agent data. Below are possible causes for missing query metrics. Prepared-statement metrics require MariaDB 10.5.2 or later (`performance_schema.prepared_statements_instances`). Query metrics from `events_statements_summary_by_digest` are collected on all supported MariaDB versions. If `performance_schema` is disabled, neither query metrics nor prepared-statement metrics are collected. See `performance_schema` is not enabled. ### Index metrics are missing{% #index-metrics-are-missing %} If the Agent displays this error: ``` Error querying mysql.innodb_index_stats: (1142, "SELECT command denied to user 'datadog'@'172.20.0.5' for table 'innodb_index_stats'") ``` Resolve the error by granting the `datadog` user the SELECT privilege to collect index metrics: ```sql GRANT SELECT ON mysql.innodb_index_stats TO datadog@'%'; ``` #### `performance_schema` is not enabled{% #performance-schema-not-enabled %} The Agent requires the `performance_schema` option to be enabled. **Unlike MySQL, MariaDB does not enable `performance_schema` by default.** Follow the [setup instructions](https://docs.datadoghq.com/database_monitoring/setup_mariadb.md) for enabling it. ### Blocking queries are missing or incomplete{% #blocking-queries-are-missing-or-incomplete %} #### Blocking-query collection is disabled{% #blocking-query-collection-is-disabled %} Blocking-query collection is disabled by default. Enable it with `query_activity.collect_blocking_queries: true` in your instance configuration. It requires no additional grants beyond the `PROCESS` and `SELECT ON performance_schema.*` privileges from the [setup instructions](https://docs.datadoghq.com/database_monitoring/setup_mariadb.md). #### Fewer blocking-query columns than MySQL 8.0{% #fewer-blocking-query-columns-than-mysql-80 %} MariaDB always uses the same, simpler set of blocking-query columns and joins that MySQL 5.7 uses, even on the newest MariaDB versions. The richer blocking-query columns available on MySQL 8.0 aren't available on MariaDB. #### Deadlock counts appear flat{% #deadlock-counts-appear-flat %} Deadlock counts aren't updated on MariaDB. The deadlock metric can remain at zero regardless of actual deadlocks occurring on the database. ### Certain queries are missing{% #certain-queries-are-missing %} If you have data from some queries, but are expecting to see a particular query or set of queries in Database Monitoring, follow this guide. | Possible cause | Solution | | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The query is not a "top query," meaning the sum of its total execution time is not in the top 200 normalized queries at any point in the selected time frame. | It may be grouped into the "Other Queries" row. For more information on which queries are tracked, see [Data Collected](https://docs.datadoghq.com/database_monitoring/data_collected.md#which-queries-are-tracked). The number of top queries tracked can be raised by contacting Datadog Support. | | The `events_statements_summary_by_digest` may be full. | The MariaDB table `events_statements_summary_by_digest` in `performance_schema` has a maximum limit on the number of digests (normalized queries) it stores. Regular truncation of this table as a maintenance task helps track all queries over time. See [Advanced configuration](https://docs.datadoghq.com/database_monitoring/setup_mariadb/advanced_configuration.md) for more information. | | The query has been executed a single time since the agent last restarted. | Query metrics are only emitted after having been executed at least once over two separate ten second intervals since the Agent was restarted. | ### Query samples are truncated{% #query-samples-are-truncated %} Longer queries may not show their full SQL text due to database configuration. Some tuning is necessary to adjust for your workload. The MariaDB SQL text length visible to the Datadog Agent is determined by the following [system variables](https://mariadb.com/kb/en/server-system-variables/#max_digest_length): ``` max_digest_length=4096 performance_schema_max_digest_length=4096 performance_schema_max_sql_text_length=4096 ``` ### Query activity is missing{% #query-activity-is-missing %} Before following these steps to diagnose missing query activity, check that the Agent is running successfully and you have followed the steps to diagnose missing agent data. Below are possible causes for missing query activity. #### `performance-schema-consumer-events-waits-current` is not enabled{% #events-waits-current-not-enabled %} The Agent requires the `performance-schema-consumer-events-waits-current` option to be enabled. It is disabled by default. Follow the [setup instructions](https://docs.datadoghq.com/database_monitoring/setup_mariadb.md) for enabling it. Alternatively, to avoid bouncing your database, consider setting up a runtime setup consumer. Create the following procedure to give the Agent the ability to enable `performance_schema.events_*` consumers at runtime. ```SQL DELIMITER $$ CREATE PROCEDURE datadog.enable_events_statements_consumers() SQL SECURITY DEFINER BEGIN UPDATE performance_schema.setup_consumers SET enabled='YES' WHERE name LIKE 'events_statements_%'; UPDATE performance_schema.setup_consumers SET enabled='YES' WHERE name = 'events_waits_current'; END $$ DELIMITER ; GRANT EXECUTE ON PROCEDURE datadog.enable_events_statements_consumers TO datadog@'%'; ``` **Note:** This option additionally requires `performance_schema` to be enabled. ### Tables are missing from collected schemas{% #tables-are-missing-from-collected-schemas %} If the Agent logs a warning starting with: ``` No tables were found across any of the N databases. ``` MariaDB exposes a table in `INFORMATION_SCHEMA` only to users that hold a privilege on that table, so the `datadog` user sees no tables at all without one. Resolve the warning by granting the `REFERENCES` privilege, which makes your table metadata visible without giving the Agent any ability to read your data: ```sql GRANT REFERENCES ON *.* TO datadog@'%'; ``` See [Collecting schemas](https://docs.datadoghq.com/database_monitoring/setup_mariadb/selfhosted.md#collecting-schemas) for more information. ### Schema or Database missing on MariaDB Query Metrics & Samples{% #schema-or-database-missing-on-mariadb-query-metrics--samples %} The `schema` tag (also known as "database") is present on MariaDB Query Metrics and Samples only when a Default Database is set on the connection that made the query. The Default Database is configured by the application by specifying the "schema" in the database connection parameters, or by executing the [USE Statement](https://mariadb.com/kb/en/use/) on an already existing connection. If there is no default database configured for a connection, then none of the queries made by that connection have the `schema` tag on them. ## MariaDB known limitations{% #mariadb-known-limitations %} MariaDB is monitored using the same MySQL integration, and metrics and events are tagged with `dbms_flavor:mariadb` to distinguish them from MySQL data. The following features differ from MySQL or aren't supported on MariaDB. ### Incompatible InnoDB metrics{% #incompatible-innodb-metrics %} The following InnoDB metrics are not available for certain MariaDB versions: | Metric Name | MariaDB Versions | | ---------------------------------------- | ----------------------- | | `mysql.innodb.hash_index_cells_total` | 10.5, 10.6, 10.11, 11.4 | | `mysql.innodb.hash_index_cells_used` | 10.5, 10.6, 10.11, 11.4 | | `mysql.innodb.os_log_fsyncs` | 10.11, 11.4 | | `mysql.innodb.os_log_pending_fsyncs` | 10.11, 11.4 | | `mysql.innodb.os_log_pending_writes` | 10.11, 11.4 | | `mysql.innodb.pending_log_flushes` | 10.11, 11.4 | | `mysql.innodb.pending_log_writes` | 10.5, 10.6, 10.11, 11.4 | | `mysql.innodb.pending_normal_aio_reads` | 10.5, 10.6, 10.11, 11.4 | | `mysql.innodb.pending_normal_aio_writes` | 10.5, 10.6, 10.11, 11.4 | | `mysql.innodb.rows_deleted` | 10.11, 11.4 | | `mysql.innodb.rows_inserted` | 10.11, 11.4 | | `mysql.innodb.rows_updated` | 10.11, 11.4 | | `mysql.innodb.rows_read` | 10.11, 11.4 | | `mysql.innodb.s_lock_os_waits` | 10.6, 10.11, 11.4 | | `mysql.innodb.s_lock_spin_rounds` | 10.6, 10.11, 11.4 | | `mysql.innodb.s_lock_spin_waits` | 10.6, 10.11, 11.4 | | `mysql.innodb.x_lock_os_waits` | 10.6, 10.11, 11.4 | | `mysql.innodb.x_lock_spin_rounds` | 10.6, 10.11, 11.4 | | `mysql.innodb.x_lock_spin_waits` | 10.6, 10.11, 11.4 | ### MariaDB explain plan{% #mariadb-explain-plan %} MariaDB does not produce the same JSON format as MySQL for explain plans. Certain explain plan fields may be missing from MariaDB explain plans, including `cost_info`, `rows_examined_per_scan`, `rows_produced_per_join`, and `used_columns`. Explain plan collection itself works the same way as on MySQL, but plan visualizations that rely on these fields may show less detail for MariaDB queries. ### `mysql.performance.errors_raised` is not collected{% #mysqlperformanceerrors_raised-is-not-collected %} This metric is collected only for MySQL 8.0 and later, and isn't available for MariaDB. ### Functional indexes aren't reflected in schema metadata{% #functional-indexes-arent-reflected-in-schema-metadata %} MariaDB doesn't support functional indexes. Index metadata collection always uses the plain index query, without the extra expression information available for MySQL 8.0.13 and later. ### Cluster tags aren't supported{% #cluster-tags-arent-supported %} Cluster tags aren't collected for MariaDB.