ProphetStor Federator.ai is an AI-based solution that helps enterprise manage, optimize, auto-scale resources for any applications on Kubernetes. Using advanced machine learning algorithms to predict application workload, Federator.ai scales the right amount of resources at the right time for optimized application performance.
With integration of ProphetStor Federator.ai, users can easily track the Kafka message production/consumption rate, as well as the prediction of message production rate from Federator.ai dashboard. Based on the prediction or message production rate, Federator.ai automatically scales Kafka consumer replicas to handle the workload. This can be visualized from Federator.ai dashboard where the recommended consumer replicas and the current number of consumer replicas are shown. Additionally, overall consumer lags as well as the average latency in the queue before a message is received by a consumer are also shown on the dashboard for better performance monitoring.
ProphetStor Federator.ai Cluster Overview
Cluster Resource Usage Predictions and Recommendations
Cluster Node Resource Usage Predictions and Recommendations
Node Current/Predicted Memory Usage (Daily)
Node Current/Predicted Memory Usage (Weekly)
Node Current/Predicted Memory Usage (Monthly)
Node Current/Predicted CPU Usage (Daily)
Node Current/Predicted CPU Usage (Weekly)
Node Current/Predicted CPU Usage (Monthly)
ProphetStor Federator.ai Application Overview
Workload Prediction for Next 24 Hours
Workload Prediction for Next 7 Days
Workload Prediction for Next 30 Days
Current/Predicted CPU Usage (Daily)
Current/Predicted CPU Usage (Weekly)
Current/Predicted CPU Usage (Monthly)
Current/Predicted Memory Usage (Daily)
Current/Predicted Memory Usage (Weekly)
Current/Predicted Memory Usage (Monthly)
Current/Desired/Recommended Replicas
Memory Usage/Request/Limit vs Rec Memory Limit
CPU Usage/Request/Limit vs Rec CPU Limit
CPU Usage/Limit Utilization
ProphetStor Federator.ai Kafka Overview
Recommended Replicas vs Current/Desired Replicas
Production vs Consumption vs Production Prediction
Kafka Consumer Lag
Consumer Queue Latency (msec)
Deployment Memory Usage
Deployment CPU Usage
ProphetStor Federator.ai Cost Analysis Overview
Current Cluster Cost and Current Cluster Configuration
Recommended Cluster - AWS and Recommended Cluster Configuration - AWS
Recommended Cluster - Azure and Recommended Cluster Configuration - Azure
Recommended Cluster - GCP and Recommended Cluster Configuration - GCP
Namespace with Highest Cost ($/day)
Namespace with Highest Predicted Cost ($/month)
Log in to OpenShift/Kubernetes cluster
Install the Federator.ai for OpenShift/Kubernetes by the following command
$ curl https://raw.githubusercontent.com/containers-ai/federatorai-operator/master/deploy/federatorai-launcher.sh | bash
curl https://raw.githubusercontent.com/containers-ai/federatorai-operator/master/deploy/federatorai-launcher.sh | bash
Please input Federator.ai version tag: datadog
Downloading scripts ...
Done
Do you want to use private repository URL? [default: n]:
Do you want to launch Federator.ai installation script? [default: y]:
Executing install.sh ...
Checking environment version...
...Passed
Enter the namespace you want to install Federator.ai [default: federatorai]:
.........
(snipped)
.........
All federatorai pods are ready.
========================================
You can now access GUI through https://<YOUR IP>:31012
Default login credential is admin/admin
Also, you can start to apply alamedascaler CR for the target you would like to monitor.
Review administration guide for further details.Review administration guide for further details.
========================================
.........
(snipped)
.........
Install Federator.ai successfully
Do you want to monitor this cluster? [default: y]:
Use "cluster-demo" as cluster name and DD_TAGS
Applying file alamedascaler_federatorai.yaml ...
alamedascaler.autoscaling.containers.ai/clusterscaler created
Done
Downloaded YAML files are located under /tmp/install-op
Verify Federator.ai pods are running properly
$ kubectl get pod -n federatorai
Log in to Federator.ai GUI, URL and login credential could be found in the output of Step 2.
A Datadog account is required for connecting and using Datadog. If you don’t have an account, visit the Datadog website and sign up for a free trial account.
Log in to Datadog with your account and get an API key and Application key for using Datadog API.
Configure the Federator.ai Data-Adapter.
$ curl https://raw.githubusercontent.com/containers-ai/federatorai-operator/master/deploy/federatorai-launcher.sh | bash
Please input Federator.ai version tag: datadog
Downloading scripts ...
Done
Do you want to use private repository URL? [default: n]:
Do you want to launch Federator.ai installation script? [default: y]: n
$ chomd +x /tmp/federatorai-scripts/datadog/federatorai-setup-for-datadog.sh
$ ./federatorai-setup-for-datadog.sh -k .kubeconfig
Checking environment version...
...Passed
You are connecting to cluster: https://<YOUR IP>:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Getting Datadog info...
Input a Datadog API Key []:xxxxx9273dxxcbc155xx3a7331xxxxx
Input a Datadog Application Key []:xxxxx7220db1478xxxxxcb5c323fcb02a11xxxxx
Getting Kafka info... No.1
You can use command "kubectl get cm cluster-info -n <namespace> --template={{.metadata.uid}}" to get cluster name
Where '<namespace>' is either 'default' or 'kube-public' or 'kube-service-catalog'.
If multiple cluster-info exist, pick either one would work as long as you always use the same one to configure Datadog Agent/Cluster Agent/WPA and other data source agents.
Input cluster name []: cluster-demo
Input Kafka exporter namespace []: myproject
Input Kafka consumer group kind (Deployment/DeploymentConfig/StatefulSet) []: Deployment
Input Kafka consumer group kind name []: consumer1-topic0001-group-0001
Input Kafka consumer group namespace []: myproject
Input Kafka consumer topic name []: topic0001
You can use Kafka command-line tool 'kafka-consumer-group.sh' (download separately or enter into a broker pod, in /bin directory) to list consumer groups.
e.g.: "/bin/kafka-consumer-groups.sh --bootstrap-server <kafka-bootstrap-service>:9092 --describe --all-groups --members"
The first column of output is the 'kafkaConsumerGroupId'.
Input Kafka consumer group id []: group0001
Input Kafka consumer minimum replica number []: 1
Input Kafka consumer maximum replica number []: 20
Do you want to input another set? [default: n]:
.........
(snipped)
.........
Please refer to Federator.ai and Datadog Integration - Installation and Configuration Guide for more details.
federatorai.integration.status (gauge) | integration status for showing Federator.ai health status. |
federatorai.recommendation (gauge) | recommended deployment/statefulset replicas. |
federatorai.prediction.kafka (gauge) | Workload prediction for Kafka metrics. |
federatorai.kafka.broker_offset_rate (gauge) | The delta of kafka.broker_offset timeseries in one minute. |
federatorai.kafka.consumer_offset_rate (gauge) | The delta of kafka.consumer_offset timeseries in one minute. |
federatorai.prediction.node (gauge) | Workload prediction for a Kubernetes node. |
federatorai.prediction.node.avg (gauge) | The average value of workload predictions for a Kubernetes node over a prediction window. |
federatorai.prediction.node.min (gauge) | The minimum value of workload predictions for a Kubernetes node over a prediction window. |
federatorai.prediction.node.max (gauge) | The maximum value of workload predictions for a Kubernetes node over a prediction window. |
federatorai.prediction.controller (gauge) | Workload prediction for a specific controller |
federatorai.prediction.controller.avg (gauge) | The average value of workload predictions for a specific controller over a prediction window. |
federatorai.prediction.controller.min (gauge) | The minimum value of workload predictions for a specific controller over a prediction window. |
federatorai.prediction.controller.max (gauge) | The maximum value of workload predictions for a specific controller over a prediction window. |
federatorai.resource_planning.node (gauge) | Workload predictions for resource planning of a Kubernetes node. |
federatorai.resource_planning.controller (gauge) | Workload predictions for resource planning of a Kubernetes controller. |
federatorai.recommendation.instance (gauge) | Cost of a recommended cloud instance. |
federatorai.cost_analysis.instance.cost (gauge) | Cost analysis for a cloud instance. |
federatorai.cost_analysis.namespace.cost (gauge) | Cost analysis for a namespace in a Kubernetes cluster |
federatorai.prediction.namespace.cost (gauge) | Cost prediction for a namespace in a Kubernetes cluster |
federatorai.kubernetes.cpu.usage.total.controller (gauge) | The number of cores (in millicore) used by the Kubernetes controller. |
federatorai.kubernetes.memory.usage.controller (gauge) | The memory usage (in bytes) of the Kubernetes controller. |
federatorai.kubernetes.cpu.usage.total.node (gauge) | The number of cores (in millicore) used by the Kubernetes node. |
federatorai.kubernetes.memory.usage.node (gauge) | The memory usage (in bytes) of the Kubernetes node. |
Federator.ai does not include any service checks.
Federator.ai does not include any events.
Need help? Read ProphetStor Federator.ai documentations or contact Datadog support.