2.2 (latest)
GitHub Contact Us Status

AWS Cloud Integration using IAM Roles for Service Accounts (IRSA)

There are many ways to integrate your AWS Cost and Usage Report (CUR) with Kubecost. This tutorial is intended as the best-practice method for users whose environments meet the following assumptions:

  1. Kubecost will run in a different account than the AWS Payer Account

  2. The IAM permissions will utilize AWS IRSA to avoid shared secrets

  3. The configuration of Kubecost will be done using a cloud-integration.json file, and not via Kubecost UI (following infrastructure as code practices)

If this is not an accurate description of your environment, see our AWS Cloud Integration doc for more options.

Overview of Kubecost CUR integration

This guide is a one-time setup per AWS payer account and is typically one per organization. It can be automated, but may not be worth the effort given that it will not be needed again.

Basic diagram when the below steps are complete:

Kubecost supports multiple AWS payer accounts as well as multiple cloud providers from a single Kubecost primary cluster. For multiple payer accounts, create additional entries inside the array below.

Detail for multiple cloud provider setups is here.

Configuration

Step 1: Download configuration files

To begin, download the recommended configuration template files from our poc-common-config repo. You will need the following files from this folder:

  • cloud-integration.json

  • iam-payer-account-cur-athena-glue-s3-access.json

  • iam-payer-account-trust-primary-account.json

  • iam-access-cur-in-payer-account.json

The bottom three files are found in /aws/iam-policies-cur.

Begin by opening cloud_integration.json, which should look like this:

{
    "aws": [
        {
            "athenaBucketName": "s3://ATHENA_RESULTS_BUCKET_NAME",
            "athenaRegion": "ATHENA_REGION",
            "athenaDatabase": "ATHENA_DATABASE",
            "athenaTable": "ATHENA_TABLE",
            "athenaWorkgroup": "ATHENA_WORKGROUP",
            "projectID": "ATHENA_PROJECT_ID",
            "masterPayerARN": "PAYER_ACCOUNT_ROLE_ARN"
        }
    ]
}

Update athenaWorkgroup to primary, then save the file and close it. The remaining values will be obtained during this tutorial.

Step 2: Create a CUR Export (and wait 24 hours)

Follow the AWS documentation to create a CUR export using the settings below.

  • For time granularity, select Daily.

  • Select the checkbox to enable Resource IDs in the report.

  • Select the checkbox to enable Athena integration with the report.

  • Select the checkbox to enable the JSON IAM policy to be applied to your bucket.

Screenshots from select CUR creation in the above AWS documentation

If this CUR data is only used by Kubecost, it is safe to expire or delete the objects after seven days of retention.

AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.

While you wait, update the following configuration files:

  • Update your cloud-integration.json file by providing a projectID value, which will be the AWS payer account number where the CUR is located and where the Kubecost primary cluster is running.

  • Update your iam-payer-account-cur-athena-glue-s2-access.json file by replacing all instances of CUR_BUCKET_NAME to the name of the bucket you created for CUR data.

Step 3: Setting up Athena

As part of the CUR creation process, Amazon creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket under s3-path-prefix/cur-name and typically has the filename crawler-cfn.yml. This .yml is your CloudFormation template. You will need it in order to complete the CUR Athena integration. You can read more about this here.

Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your bucket's report. In the Report details tab, you will find the S3 path prefix.

Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results. The bucket used for the CUR cannot be used for the Athena output.

  1. Navigate to the S3 Management Console.

  2. Select Create bucket. The Create Bucket page opens.

  3. Provide a name for your bucket. This is the value for athenaBucketName in your cloud-integration.json file. Use the same region used for the CUR bucket.

  4. Select Create bucket at the bottom of the page.

  5. Navigate to the Amazon Athena dashboard.

  6. Select Settings, then select Manage. The Manage settings window opens.

  7. Set Location of query result to the S3 bucket you just created, then select Save.

Navigate to Athena in the AWS Console. Be sure the region matches the one used in the steps above. Update your cloud-integration.json file with the following values. Use the screenshots below for help.

  • athenaBucketName: the name of the Athena bucket your created in this step

  • athenaDatabase: the value in the Database dropdown

  • athenaRegion: the AWS region value where your Athena query is configured

  • athenaTable: the partitioned value found in the Table list

For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after one day of retention.

Step 4: Setting up payer account IAM permissions

From the AWS payer account

In iam-payer-account-cur-athena-glue-s3-access.json, replace all ATHENA_RESULTS_BUCKET_NAME instances with your Athena S3 bucket name (the default will look like aws-athena-query-results-xxxx).

In iam-payer-account-trust-primary-account.json, replace SUB_ACCOUNT_222222222 with the account number of the account where the Kubecost primary cluster will run.

In the same location as your downloaded configuration files, run the following command to create the appropriate policy (jq is not required):

aws iam create-role --role-name kubecost-cur-access \
  --assume-role-policy-document file://iam-payer-account-trust-primary-account.json \
  --output json | jq -r .Role.Arn

Now we can obtain the last value masterPayerARN for cloud-integration.json as the ARN associated with the newly-created IAM role, as seen below in the AWS console:

Step 5: Setting up IAM permissions for the primary cluster

By arriving at this step, you should have been able to provide all values to your cloud-integration.json file. If any values are missing, reread the tutorial and follow any steps needed to obtain those values.

From the AWS Account where the Kubecost primary cluster will run

In iam-access-cur-in-payer-account.json, update PAYER_ACCOUNT_11111111111 with the AWS account number of the payer account and create a policy allowing Kubecost to assumeRole in the payer account:

aws iam create-policy --policy-name kubecost-access-cur-in-payer-account \
  --policy-document file://iam-access-cur-in-payer-account.json \
  --output json |jq -r .Policy.Arn

Note the output ARN (used in the iamserviceaccount --attach-policy-arn below):

arn:aws:iam::SUB_ACCOUNT_222222222:policy/kubecost-access-cur-in-payer-account

Create a namespace and set environment variables:

kubectl create ns kubecost
export CLUSTER_NAME=YOUR_CLUSTER
export AWS_REGION=YOUR_REGION

Enable the OIDC-Provider:

eksctl utils associate-iam-oidc-provider \
    --cluster $CLUSTER_NAME --region $AWS_REGION \
    --approve

Linking default Kubecost Service Account to an IAM Role

Kubecost's default service account kubecost-cost-analyzer is automatically created in the kubecost namespace upon installation. This service account can be linked to an IAM Role via Annotation + IAM Trust Policy.

In the Helm values for your deployment, add the following section:

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<accountNumber>:role/<kubecost-role>

Go to the IAM Role and attach the proper IAM trust policy. Use the sample trust policy here. Verify you have replaced the example OIDC URL with your cluster OIDC URL.

Alternative method: Create a new dedicated service account for Kubecost using eksctl

This method creates a new service account via eksctl command line tools, instead of using the default service account. Eksctl automatically creates the trust policy and IAM Role that are linked to the new dedicated Kubernetes service account.

Replace SUB_ACCOUNT_222222222 with the AWS account number where the primary Kubecost cluster will run.

eksctl create iamserviceaccount \
    --name kubecost-serviceaccount \
    --namespace kubecost \
    --cluster $CLUSTER_NAME --region $AWS_REGION \
    --attach-policy-arn arn:aws:iam::SUB_ACCOUNT_222222222:policy/kubecost-access-cur-in-payer-account \
    --override-existing-serviceaccounts \
    --approve

Create the secret (in this setup, there are no actual secrets in this file):

kubectl create secret generic cloud-integration -n kubecost --from-file=cloud-integration.json

Install Kubecost using the service account and cloud-integration secret:

helm install kubecost \
  --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \
  --namespace kubecost \
  --set serviceAccount.name=kubecost-serviceaccount \
  --set serviceAccount.create=false \
  --set kubecostProductConfigs.cloudIntegrationSecret=cloud-integration

Add the following section to your Helm values. This will tell Kubecost to use your newly created service account, instead of creating one.

serviceAccount:
  create: false
  name: kubecost-serviceaccount

Validation

It can take over an hour to process the billing data for large AWS accounts. In the short-term, follow the logs and look for a message similar to (7.7 complete), which should grow gradually to (100.0 complete). Some errors (ERR) are expected, as seen below.

kubectl logs -l app=cost-analyzer --tail -1 --follow |grep -i athena
------------------
Defaulted container "cost-model" out of: cost-model, cost-analyzer-frontend
2023-05-24T19:41:31.63093249Z ERR Failed to lookup reserved instance data: no reservation data available in Athena
2023-05-24T19:41:31.630973097Z ERR Failed to lookup savings plan data: Error fetching Savings Plan Data: QueryAthenaPaginated: athena configuration incomplete
2023-05-24T19:41:34.577437245Z INF Adding AWS Provider to map with key: 440082503234/s3://aws-athena-query-results-
2023-05-24T19:41:34.57901059Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Starting PipelineController
2023-05-24T19:41:34.579037927Z INF CloudCost: IngestionManager: creating integration with key: 440082503234/s3://aws-athena-query-results-
2023-05-24T19:41:34.581131953Z INF RunBuildProcess[CloudCost][440082503234/s3://aws-athena-query-results-]: build[NLzAH]: Starting build back to 2023-02-22 00:00:00 +0000 UTC in blocks of 7d
2023-05-24T19:41:34.581715777Z INF CloudCost[440082503234/s3://aws-athena-query-results-]: ingestor: building window [2023-05-17T00:00:00+0000, 2023-05-24T00:00:00+0000)
2023-05-24T19:41:34.581771159Z INF AthenaIntegration[440082503234/s3://aws-athena-query-results-]: StoreCloudCost: [2023-05-17T00:00:00+0000, 2023-05-24T00:00:00+0000)
2023-05-24T19:41:34.608040758Z ERR ETL:  CloudUsage[440082503234/s3://aws-athena-query-results-]: Build[XPQMa]:  failed to load range from back up 2023-05-18 00:00:00 +0000 UTC - 2023-05-25 00:00:00 +0000 UTC: file does not exist
2023-05-24T19:41:34.60806207Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Build[XPQMa]: QueryCloudUsage [2023-05-18T00:00:00+0000, 2023-05-25T00:00:00+0000)
2023-05-24T19:41:34.588661972Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Run[JkSYH]: QueryCloudUsage [2023-05-21T00:00:00+0000, 2023-05-25T00:00:00+0000)
2023-05-24T19:41:50.528544821Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Build[XPQMa]: coverage [2023-05-18T00:00:00+0000, 2023-05-25T00:00:00+0000) (7.7 complete)

Troubleshooting

For help with troubleshooting, see the section in our original AWS integration guide.

PreviousAWS Cloud Billing IntegrationNextAWS Out of Cluster

Last updated