Federator.ai

Agent Check

Overview

ProphetStor Federator.ai is an AI-based solution that helps enterprise manage, optimize, auto-scale resources for any applications on Kubernetes. Using advanced machine learning algorithms to predict application workload, Federator.ai scales the right amount of resources at the right time for optimized application performance.

• AI-based workload prediction for Kafka or any applications
• Resource recommendation based on workload prediction, application, Kubernetes and other related metrics
• Automatic scaling of application containers through Datadog Watermark Pod Autoscaler (WPA)

With integration of ProphetStor Federator.ai, users can easily track the Kafka message production/consumption rate, as well as the prediction of message production rate from Federator.ai dashboard. Based on the prediction or message production rate, Federator.ai automatically scales Kafka consumer replicas to handle the workload. This can be visualized from Federator.ai dashboard where the recommended consumer replicas and the current number of consumer replicas are shown. Additionally, overall consumer lags as well as the average latency in the queue before a message is received by a consumer are also shown on the dashboard for better performance monitoring.

• Federator.ai Dashboard Overview

• Recommended Replicas vs Current/Desired Replicas
  • This timeseries graph shows the recommended replicas from the Federator.ai and the desired and current replicas in the system.

• Production vs Consumption vs Production Prediction
  • This timeseries graph shows the Kafka message production rate and consumption rate and the production rate predicted by Federator.ai.

• Kafka Consumer Lag
  • This timeseries graph shows the sum of consumer lags from all partitions.

• Consumer Queue Latency (msec)
  • This timeseries graph shows the average latency of a message in the message queue before it is received by a consumer.

• Deployment Memory Usage
  • This timeseries graph shows the memory usage of consumers.

• Deployment CPU Usage
  • This timeseries graph shows the CPU usage of consumers.

Setup

Installation

1. Log in to OpenShift/Kubernetes cluster

2. Install the Federator.ai for OpenShift/Kubernetes by the following command

   $ curl https://raw.githubusercontent.com/containers-ai/federatorai-operator/datadog/deploy/install.sh |bash

   $ curl https://raw.githubusercontent.com/containers-ai/federatorai-operator/datadog/deploy/install.sh |bash
   Checking environment version...
   ...Passed
   Please input Federator.ai Operator tag: datadog
   Enter the namespace you want to install Federator.ai [default: federatorai]:
   .........
   (snipped)
   .........
   You can now access GUI through https://federatorai-dashboard-frontend-federatorai.apps.jc-ocp4.172-31-17-84.nip.io
   Default login credential is admin/admin
   
   Also, you can start to apply alamedascaler CR for the namespace you would like to monitor.
   Review administration guide for further details.
   ========================================
   .........
   (snipped)
   .........
   Install Alameda successfully
   
   Downloaded YAML files are located under /tmp/install-op

3. Verify Federator.ai pods are running properly

   $ kubectl get pod -n federatorai

4. Log in to Federator.ai GUI, URL and login credential could be found in the output of Step 2.

Configuration

1. A Datadog account is required for connecting and using Datadog. If you don’t have an account, visit the Datadog website and sign up for a free trial account.

2. Log in to Datadog with your account and get an API key and Application key for using Datadog API.

3. Configure the Federator.ai Data-Adapter.

   • Download the Data-Adapter configuration script from Github.

     $ curl https://raw.githubusercontent.com/containers-ai/federatorai-operator/datadog/deploy/federatorai-setup-for-datadog.sh -O

   • Change the execution permission.

     $ chomd +x federatorai-setup-for-datadog.sh

   • Prepare .kubeconfig (sh -c “export KUBECONFIG=.kubeconfig; oc login ”) or use an existing one. For example:

     $ sh -c "export KUBECONFIG=.kubeconfig; oc login https://api.ocp4.example.com:6443"

   • Run the configuration script and follow the steps to fill in configuration parameters:

     $ ./federatorai-setup-for-datadog.sh -k .kubeconfig

     $ ./federatorai-setup-for-datadog.sh -k .kubeconfig
     You are connecting to cluster: https://api.jc-ocp4.172-31-17-84.nip.io:6443
     
     Getting Datadog info...
     Input a Datadog API Key []:xxxxx9273dxxcbc155xx3a7331xxxxx
     Input a Datadog Application Key []:xxxxx7220db1478xxxxxcb5c323fcb02a11xxxxx
     
     Getting the Kafka info... No.1
     Input Kafka consumer deployment name []: consumer
     Input Kafka consumer deeployment namespace []: myproject
     Input Kafka consumer minimum replica number []: 1
     Input Kafka consumer maximum replica number []: 30
     Input Kafka consumer group name []: group0001
     Input Kafka consumer group namespace []: myproject
     Input Kafka consumer topic name []: topic0001
     Input Kafka consumer topic namespace []: myproject
     
     Do you want to input another set? [default: n]:
     Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
     secret/federatorai-data-adapter-secret configured
     Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
     configmap/federatorai-data-adapter-config configured
     
     Setup Federator.ai for datadog successfully

4. Please refer to Federator.ai and Datadog Integration - Installation and Configuration Guide for more details.

Data Collected

Metrics

Service Checks

Federator.ai does not include any service checks.

Events

Federator.ai does not include any events.

Troubleshooting

Need help? Read ProphetStor Federator.ai documentations or contact Datadog support.