Superwise

Supported OS Linux Windows

Overview

Superwise provides model observability for high-scale machine learning (ML) operations. Superwise’s model observability gives you visibility and context into your models’ behaviors, so you can easily monitor model risks based on different use cases. With Superwise, data scientists, ML engineers, and business ops get model observability without alert fatigue, so you can be confident about your model management.

Superwise Dashboard

Superwise’s model metrics and incidents integration sends out-of-the-box model metrics, including drift, activity, incidents, and custom metrics, directly to Datadog. You get an overview of which models are not predicting the expected results, which can be configured to any use case, logic, segmentation, threshold, and sensitivity.

With the Datadog integration configured in Superwise, standard model metrics are sent to Datadog, and users get model observability dashboards in Datadog. You can configure any specific model metric and incident policy, and send them to Datadog for model observability that is tailored to your use case.

Setup

  1. Go to Superwise portal and select Integrations.

  2. Click Create a new channel and select Datadog.

    Superwise - Add new integration

  3. Input your Datadog API and application keys, and click Test. A test request is sent to your Datadog account to validate the integration. If the request was successfully sent, there is a message in Superwise saying the test was delivered successfully. To finish the setup, click Create channel.

    Superwise - Add new Datadog channel

  4. When setup is complete, the new Datadog integration widget is available:

    Superwise Integration

Validation

In Datadog, go to Metrics Explorer and search for the metric superwise.integration.test to verify that the integration between Superwise and Datadog is working.

superwise.integration.test graph in Datadog

Data Collected

Metrics

superwise.metric.overall.data_drift
(gauge)
Average distribution shifts across all features.
Shown as day
superwise.metric.overall.weighted_data_drift
(gauge)
Average distribution shifts across all features weighted by importance.
Shown as day
superwise.metric.overall.quantity
(gauge)
Number of records sent.
Shown as day

Events

The Superwise integration does not include any events.

Service Checks

The Superwise integration does not include any service checks.

Troubleshooting

Need help? See the Superwise documentation.

Further Reading

Additional helpful documentation, links, and articles: