Dashboard overview with health check and AI output quality metrics.
Safety and Security Metrics.
Tone metrics with a list of response metadata logs.
Description of a response evaluation log.
Overview
AIMon provides tools that help organizations evaluate, monitor, and improve the quality and reliability of their AI applications. It surfaces key response metrics such as groundedness, instruction adherence, context relevance, and toxicity to help teams improve output quality and build more reliable AI systems.
The Datadog integration enables real-time or asynchronous visualization of these metrics, supporting faster anomaly detection, prompt evaluation, and tuning of LLM pipelines. AIMon also logs evaluation metadata such as user prompts, model responses, timestamps, and evaluation scores for traceability and debugging at scale.
Setup
- Within Datadog, navigate to the Integrations page and select AIMon.
- Click Install Integration, then select Connect Accounts. You will be redirected to AIMon login page.
- Log in to your AIMon account using your credentials or continue with Google. You will be redirected to the
/integrations tab on AIMon. - Toggle on the Datadog integration to enable the connection.
Note: If your organization already has a Datadog API key and an AIMon account, make sure you register under the same company name in AIMon. AIMon requires a one-to-one mapping between companies and Datadog organizations. Using a different company name results in an error.
When the integration is active and the AIMon API key is in use, the platform will automatically transmit:
- Evaluation metrics, such as hallucination scores and instruction adherence.
- Evaluation logs with relevant metadata, such as user prompts and model responses.
Data Collected
Metrics
| |
|---|
aimon.groundedness
(gauge) | Detects fabricated or inaccurate content in LLM responses
Shown as fraction |
aimon.conciseness
(gauge) | Evaluates whether the LLM output is succinct and clear
Shown as fraction |
aimon.completeness
(gauge) | Determines whether the response fully answers the input query
Shown as fraction |
aimon.toxicity
(gauge) | Identifies toxic or harmful language in the LLM output
Shown as fraction |
aimon.retrieval_relevance
(gauge) | Measures how relevant the retrieved context is to the query
Shown as fraction |
aimon.context_classification
(gauge) | Classifies whether the provided context was used appropriately and aligns with user intent
Shown as fraction |
aimon.instruction_adherence
(gauge) | Measures how well the instructions are followed while generating a response
Shown as fraction |
aimon.health_check
(gauge) | Indicates whether AIMon is reporting metrics correctly
Shown as fraction |
aimon.output_relevance
(gauge) | Measures how well the response aligns with the user query
Shown as fraction |
aimon.prompt_injection
(gauge) | Detects attempts in the user query to override system instructions using adversarial language
Shown as fraction |
aimon.cbrn
(gauge) | Flags model outputs that include references to chemical, biological, radiological, or nuclear threats or related instructions
Shown as fraction |
aimon.personal_harm
(gauge) | Detects mentions of self-harm, suicide, or encouragement of physically dangerous behavior in the model’s output
Shown as fraction |
aimon.unsafe_stereotypes
(gauge) | Detects biased or discriminatory content in model outputs based on race, gender, religion, or identity groups
Shown as fraction |
aimon.code_detection
(gauge) | Detects unsafe or system-level code patterns in the user query to prevent unintended execution or exploitation
Shown as fraction |
aimon.sql_prevention
(gauge) | Flags SQL keywords or injection patterns in the user query that may indicate data access attempts or prompt attacks
Shown as fraction |
aimon.jailbreak
(gauge) | Identifies jailbreak tactics in the user query designed to elicit harmful or filtered responses from the model
Shown as fraction |
aimon.pii
(gauge) | Flags exposure of personally identifiable information such as names, addresses, or contact details in the model’s output
Shown as fraction |
Uninstallation
- Log in to your account on the AIMon app and navigate to the Integrations tab.
- Disable the Datadog integration.
- Log in to Datadog and navigate to the Integrations tab.
- Disable the AIMon integration.
- Search for the integration name on the API Keys page and disable any associated API keys to fully revoke access.
Support
For support, please reach out to the AIMon team.
Further Reading
Additional helpful documentation, links, and articles:
