Algorithmia is an MLOps platform that includes capabilities for data
scientists, application developers, and IT operators to deploy, manage, govern,
and secure machine learning and other probabilistic models in production.
Algorithmia Insights is a feature of Algorithmia Enterprise and provides a
metrics pipeline that can be used to instrument, measure, and monitor your
machine learning models. Use cases for monitoring inference-related metrics from
machine learning models include detecting model drift, data drift, model bias,
etc.
This integration allows you to stream operational metrics as well as
user-defined, inference-related metrics from Algorithmia to Kafka to the metrics
API in Datadog.
Setup
From your Algorithmia instance, configure Algorithmia Insights to connect to
a Kafka broker (external to Algorithmia).
See the Algorithmia Integrations repository
to install, configure, and start the Datadog message forwarding service used
in this integration, which forwards metrics from a Kafka topic to the
metrics API in Datadog.
Validation
From Algorithmia, query an algorithm that has Insights enabled.
In the Datadog interface, navigate to the Metrics summary page.
Verify that the metrics from Insights are present in Datadog by filtering for
algorithmia.
Streaming metrics
This integration streams metrics from Algorithmia when a model that has Insights
enabled is queried. Each log entry includes operational metrics and
inference-related metrics.
The duration_milliseconds metric is one of the operational metrics that is
included in the default payload from Algorithmia. To help you get started, this
integration also includes a dashboard and monitor for this default metric.
Additional metrics can include any user-defined, inference-related metrics that
are specified in Algorithmia by the algorithm developer. User-defined metrics
depend on your specific machine learning framework and use case, but might
include values such as prediction probabilities from a regression model in
scikit-learn, confidence scores from an image classifier in TensorFlow, or input
data from incoming API requests. Note: The message forwarding script
provided in this integration prefixes user-defined metrics with
algorithmia. in Datadog.
Data Collected
Metrics
algorithmia.duration_milliseconds (gauge)
Duration of algorithm run Shown as millisecond
Service Checks
The Algorithmia check does not include any service checks.
Events
The Algorithmia check does not include any events.
