Sections in this category

Prometheus Configuration Guide

Bring your own Prometheus

There are several considerations when disabling the Kubecost included Prometheus deployment. Kubecost strongly recommends installing Kubecost with the bundled Prometheus in most environments.

The Kubecost Prometheus deployment is optimized to not interfere with other observability instrumentation and by default only contains metrics that are useful to the Kubecost product. This results in 70-90% fewer metrics than a Prometheus deployment using default settings.

Additonally, if multi-cluster metric aggregation is required, Kubecost provides a turnkey solution that is highly tuned and simple to support using the included Prometheus deployment.

Note: the Kubecost team provides best efforts support for free/community users when integrating with an existing Prometheus deployment.

If you have node-exporter and/or KSM running on your cluster, follow this step to disable the Kubecost included versions. Additional detail on KSM requiments.

Note: In contrast to our recommendation above, we do recommend disabling the Kubecost’s node-exporter and kube-state-metrics if you already have them running in your cluster.

  helm upgrade --install kubecost \
    --repo cost-analyzer \
    --namespace kubecost --create-namespace \
    --set prometheus.nodeExporter.enabled=false \
    --set prometheus.serviceAccounts.nodeExporter.create=false \
    --set prometheus.kubeStateMetrics.enabled=false

Dependency Requirements

Kubecost requires the following minimum versions:

  • kube-state-metrics - v1.6.0+ (May 19)
  • cAdvisor - kubelet v1.11.0+ (May 18)
  • node-exporter - v0.16+ (May 18) [Optional]

Before contintuing, see the note above about Kubecost’s bundled prometheus

  1. Pass the following parameters in your helm install:

    helm upgrade --install kubecost \
      --repo cost-analyzer \
      --namespace kubecost --create-namespace \
      --set global.prometheus.fqdn=http://<prometheus-server-service-name>.<prometheus-server-namespace>.svc \
      --set global.prometheus.enabled=false

    Note The fqdn can be a full path: if you use Grafana Cloud managed Prometheus. Learn more at Grafana Cloud Integration for Kubecost

  2. Have your Prometheus scrape the cost-model /metrics endpoint. These metrics are needed for reporting accurate pricing data. Here is an example scrape config:

- job_name: kubecost
      honor_labels: true
      scrape_interval: 1m
      scrape_timeout: 10s
      metrics_path: /metrics
      scheme: http
      - names:
        - kubecost-cost-analyzer.<namespace-of-your-kubecost>
        type: 'A'
        port: 9003

This config needs to be added to extraScrapeConfigs in the Prometheus configuration. Example extraScrapeConfigs.yaml

To confirm this job is successfully scraped by Prometheus, you can view the Targets page in Prometheus and look for a job named kubecost.

Prometheus Targets

Node exporter metric labels

Note: This step is optional, and only impacts certain efficiency metrics. View issue/556 for a description of what will be missing if this step is skipped.

You’ll need to add the following relabel config to the job that scrapes the node exporter DaemonSet.

  - job_name: 'kubernetes-service-endpoints'

      - role: endpoints

      - source_labels: [__meta_kubernetes_pod_node_name]
        action: replace
        target_label: kubernetes_node

Note that this does not override the source label. It creates a new label called “kubernetes_node” and copies the value of pod into it.


Visiting <your-kubecost-endpoint>/diagnostics.html provides diagnostics info on this integration. More details

Common issues include the following:

Wrong Prometheus FQDN: Evidenced by the following pod error message No valid prometheus config file at ... and the init pods hanging. We recommend running curl <your_prometheus_url>/api/v1/status/config from a pod in the cluster to confirm that your Prometheus config is returned. Here is an example, but this needs to be updated based on your Prometheus address:

kubectl exec kubecost-cost-analyzer-db55d88f6-fr6kc -c cost-analyzer-frontend -n kubecost \
-- curl http://kubecost-prometheus-server.kubecost/api/v1/status/config

If the config file is not returned, this is an indication that an incorrect Prometheus address has been provided. If a config file is returned from one pod in the cluster but not the Kubecost pod, then the Kubecost pod likely has its access restricted by a network policy, service mesh, etc.

Prometheus throttling: Ensure Prometheus isn’t being CPU throttled due to a low resource request.

Wrong dependency version: Review the Dependency Requirements section above

Missing scrape configs: Visit Prometheus Targets page (screenshot above)

Data incorrectly is a single namespace: Make sure that honor_labels is enabled

Negative idle reported: Make sure the kubecost job is scraping Kubecost. Metrics for node_total_hourly_cost should exist in Prometheus.

You can visit Settings in Kubecost to see basic diagnostic information on these Prometheus metrics:

Prometheus status diagnostic

Have a question not answered on this page? Email us at or join the Kubecost Slack community!

Edit this doc on GitHub