AMP with Kubecost Prometheus (remote_write)
See also
Overview
When the Amazon Managed Service for Prometheus integration is enabled, the bundled Prometheus server in the Kubecost Helm Chart is configured in the remote write mode. The bundled Prometheus server sends the collected metrics to Amazon Managed Service for Prometheus using the AWS SigV4 signing process. All metrics and data are stored in Amazon Managed Service for Prometheus, and Kubecost queries the metrics directly from Amazon Managed Service for Prometheus instead of the bundled Prometheus. It helps customers not worry about maintaining and scaling the local Prometheus instance.
Kubecost has multiple methods for multi-cluster. There may be performance limits to how many clusters/nodes can be supported on a single AMP instance. Please contact Kubecost support for more information.
Quick-Start architecture
The following architecture diagram illustrates what this configuration guide aims to achieve:
It assumes the following prerequisites:
You have an existing AWS account.
You have IAM credentials to create Amazon Managed Service for Prometheus and IAM roles programmatically.
You have an existing Amazon EKS cluster with OIDC enabled.
Your Amazon EKS clusters have Amazon EBS CSI driver installed
Creating Amazon Managed Service for Prometheus workspace
Run the following command to get the information of your current EKS cluster:
The example output should be in this format:
Run the following command to create new a Amazon Managed Service for Prometheus workspace:
The Amazon Managed Service for Prometheus workspace should be created in a few seconds.
Run the following command to get the workspace ID:
Setting up the environment:
Run the following command to set environment variables for integrating Kubecost with Amazon Managed Service for Prometheus:
Set up IRSA to allow Kubecost and Prometheus to read & write metrics from Amazon Managed Service for Prometheus by running the following commands:
These commands help to automate the following tasks:
Create an IAM role with the AWS-managed IAM policy and trusted policy for the following service accounts:
kubecost-cost-analyzer-amp
,kubecost-prometheus-server-amp
.Modify current K8s service accounts with annotation to attach a new IAM role.
For more information, you can check AWS documentation at IAM roles for service accounts and learn more about Amazon Managed Service for Prometheus managed policy at Identity-based policy examples for Amazon Managed Service for Prometheus
Integrating Kubecost with Amazon Managed Service for Prometheus
Preparing the configuration file
Run the following command to create a file called config-values.yaml, which contains the defaults that Kubecost will use for connecting to your Amazon Managed Service for Prometheus workspace.
Primary cluster
Run this command to install Kubecost and integrate it with the Amazon Managed Service for Prometheus workspace as the primary:
Secondary clusters
These installation steps are similar to those for a primary cluster setup, except you do not need to follow the steps in the section "Create Amazon Managed Service for Prometheus workspace", and you need to update these environment variables below to match with your secondary clusters. Please note that the AMP_WORKSPACE_ID
and KC_BUCKET
are the same as the primary cluster.
Run this command to install Kubecost and integrate it with the Amazon Managed Service for Prometheus workspace as the additional cluster:
Your Kubecost setup is now writing and collecting data from AMP. Data should be ready for viewing within 15 minutes.
To verify that the integration is set up, select Settings in the Kubecost UI, and check the 'Prometheus Status' section.
Troubleshooting
See more troubleshooting steps at the bottom of AMP Overview.
See also
Last updated