Kubecost pulls asset prices from the public AWS pricing API by default. To have accurate pricing information from AWS, you can integrate directly with your account. This integration will properly account for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, spot usage, and more. This resource describes the required steps for achieving this.
Your user will need necessary permissions to create the Cost and Usage Report, add IAM credentials for Athena and S3. Optional permission is the ability to add and execute CloudFormation templates. Note we do not require root access in the AWS account.
A github repository with sample files used in below instructions can be found here: https://github.com/kubecost/poc-common-configurations/tree/main/aws
Follow these steps to set up a Cost and Usage Report. For time granularity, select Daily. Be sure to enable Resource Ids and Athena integration when creating the CUR. https://docs.aws.amazon.com/cur/latest/userguide/cur-create.html
Note the name of the bucket you create for CUR data. This will be used in the following step.
If you believe you have the correct permissions, but cannot access the Billing and Cost Management page, have the owner of your organization's root account follow these instructions https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/control-access-billing.html
AWS may take several hours to publish data(up to 24 hours), wait until this is complete before continuing to the next step.
As part of the CUR creation process, Amazon also creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket under your-billing-prefix/cur-name and typically has the filename crawler-cfn.yml. You will need to deploy this CloudFormation template in order to complete the CUR Athena integration.
https://docs.aws.amazon.com/cur/latest/userguide/use-athena-cf.html
Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results.
- Navigate to https://console.aws.amazon.com/s3
- Select Create Bucket
- Be sure to use the same region as was used for the CUR bucket and pick a name that follows the format
aws-athena-query-results-* - Select Create Bucket
- Navigate to https://console.aws.amazon.com/athena
- Click Settings
- Set Query result location to the S3 bucket you just created
Kubecost offers a set of CloudFormation templates to help set your IAM roles up. If you’re new to provisioning IAM roles, we suggest downloading our templates and using the CloudFormation wizard to set these up: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html. Download template files from the URLs provided below and upload them as the stack template in the Creating a stack > Selecting a stack template step.
My kubernetes clusters all run in the same account as the master payer account.
Download this file: https://raw.githubusercontent.com/kubecost/cloudformation/master/kubecost-single-account-permissions.yaml
Navigate to https://console.aws.amazon.com/cloudformation
Click Create New Stack if you have never used AWS CloudFormation before. Otherwise, click Create Stack. and select With new resource (standard)
Under Prepare template, choose Template is ready.
Under Template source, choose Upload a template file.
Select Choose file.
Choose the downloaded .yaml template, and then choose Open.
Choose Next.
For Stack name, enter a name for your template
Set the following parameters:
- AthenaCURBucket: The bucket where the CUR is sent from the “Setting up the CUR” step
- SpotDataFeedBucketName: Optional. The bucket where the spot data feed is sent from the “Setting up the Spot Data feed” step (see below)
Choose Next.
Choose Next
At the bottom of the page, select I acknowledge that AWS CloudFormation might create IAM resources.
Choose Create Stack
My kubernetes clusters run in different accounts from the master payer account
- On each sub account running Kubecost
- Download this file: https://raw.githubusercontent.com/kubecost/cloudformation/master/kubecost-sub-account-permissions.yaml
- Navigate to https://console.aws.amazon.com/cloudformation
- Choose Create New Stack if you have never used AWS CloudFormation before. Otherwise, choose Create Stack.
- Under Prepare template, choose Template is ready.
- Under Template source, choose Upload a template file.
- Select Choose file.
- Choose the downloaded .yaml template, and then choose Open.
- Choose Next.
- For Stack name, enter a name for your template
- Set the following parameters:
- MasterPayerAccountID: The account ID of the master payer account where the CUR has been created
- SpotDataFeedBucketName: The bucket where the spot data feed is sent from the “Setting up the Spot Data feed” step
- Choose Next.
- Choose Next
- At the bottom of the page, select I acknowledge that AWS CloudFormation might create IAM resources.
- Choose Create Stack
- On the master payer account
- Follow the same steps to create a CloudFormation stack as above, but with the following as your yaml file instead: https://raw.githubusercontent.com/kubecost/cloudformation/master/kubecost-masterpayer-account-permissions.yaml , and with these parameters:
- AthenaCURBucket: The bucket where the CUR is set from the “Setting up the CUR” step
- KubecostClusterID: An account that Kubecost is running on that requires access to the Athena CUR
- Download this file: https://raw.githubusercontent.com/kubecost/cloudformation/master/kubecost-sub-account-permissions.yaml
My Kubernetes clusters run in the same account as the master payer account
Attach both of the following policies to the same role or user. Use a user if you intend to integrate via servicekey, and a role if via IAM annotation (See more below under Via Pod Annotation by EKS). The SpotDataAccess policy statement is optional if the spot data feed is configured (see “Setting up the Spot Data feed” step below)
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AthenaAccess",
"Effect": "Allow",
"Action": [
"athena:*"
],
"Resource": [
"*"
]
},
{
"Sid": "ReadAccessToAthenaCurDataViaGlue",
"Effect": "Allow",
"Action": [
"glue:GetDatabase*",
"glue:GetTable*",
"glue:GetPartition*",
"glue:GetUserDefinedFunction",
"glue:BatchGetPartition"
],
"Resource": [
"arn:aws:glue:*:*:catalog",
"arn:aws:glue:*:*:database/athenacurcfn*",
"arn:aws:glue:*:*:table/athenacurcfn*/*"
]
},
{
"Sid": "AthenaQueryResultsOutput",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload",
"s3:CreateBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::aws-athena-query-results-*"
]
},
{
"Sid": "S3ReadAccessToAwsBillingData",
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::${AthenaCURBucket}*"
]
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SpotDataAccess",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:ListBucket",
"s3:HeadBucket",
"s3:HeadObject",
"s3:List*",
"s3:Get*"
],
"Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
}
]
}
My Kubernetes clusters run in different accounts
On each sub account running kubecost, attach both of the following policies to the same role or user. Use a user if you intend to integrate via servicekey, and a role if via IAM annotation (See more below under Via Pod Annotation by EKS). The SpotDataAccess policy statement is optional if the spot data feed is configured (see “Setting up the Spot Data feed” step below)
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AssumeRoleInMasterPayer",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::${MasterPayerAccountID}:role/KubecostRole-${This-account’s-id}"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SpotDataAccess",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:ListBucket",
"s3:HeadBucket",
"s3:HeadObject",
"s3:List*",
"s3:Get*"
],
"Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
}
]
}
On the masterpayer account, attach this policy to a role (replace ${AthenaCURBucket} variable):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AthenaAccess",
"Effect": "Allow",
"Action": [
"athena:*"
],
"Resource": [
"*"
]
},
{
"Sid": "ReadAccessToAthenaCurDataViaGlue",
"Effect": "Allow",
"Action": [
"glue:GetDatabase*",
"glue:GetTable*",
"glue:GetPartition*",
"glue:GetUserDefinedFunction",
"glue:BatchGetPartition"
],
"Resource": [
"arn:aws:glue:*:*:catalog",
"arn:aws:glue:*:*:database/athenacurcfn*",
"arn:aws:glue:*:*:table/athenacurcfn*/*"
]
},
{
"Sid": "AthenaQueryResultsOutput",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:AbortMultipartUpload",
"s3:CreateBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::aws-athena-query-results-*"
]
},
{
"Sid": "S3ReadAccessToAwsBillingData",
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::${AthenaCURBucket}*"
]
}
]
}
You will then need to add the following trust statement to the role the policy is attached to (replace ${KubecostClusterID} variable):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${KubecostClusterID}:root"
},
"Action": [
"sts:AssumeRole"
]
}
]
}
Now that the policies have been created, we will need to attach those policies to Kubecost. We support the following methods:
Attach via Service Key And Kubernetes Secret
Navigate to https://console.aws.amazon.com/iam Access Management > Users. Find the Kubecost User and select Security Credentials > Create Access Key. Note the Access key ID and Secret access key. You’ll use it to either Create a secret from helm values or Create and use an existing secret.
Create a secret from helm values
Create a secret from helm values
- Set
.Values.kubecostProductConfigs.awsServiceKeyNametoAccess key ID - Set
.Values.kubecostProductConfigs.awsServiceKeyPasswordto Secret access key - Note that this will leave your secrets unencrypted in values.yaml. Use an existing secret as in the next step to avoid this.
- Set
Create and use an existing secret
If you commit your helm values to source control, you may want to create a secret in a different way and import that secret to kubecost.
* Create a json file named service-key.json of the following format
{
"aws_access_key_id": "<ACCESS_KEY_ID>",
"aws_secret_access_key": "<ACCESS_KEY_SECRET>"
}
* Create a secret from file in the namespace kubecost is deployed in:
kubectl create secret generic <name> --from-file=service-key.json --namespace <kubecost>
* Set .Values.kubecostProductConfigs.serviceKeySecretName to the name of this secet. Note also that .Values.kubecostProductConfigs.awsServiceKeyName and .Values.kubecostProductConfigs.awsServiceKeyPassword should be unset if adding the service key from values this way.
Attach via Service Key on Kubecost frontend
- Navigate to https://console.aws.amazon.com/iam Access Management > Users. Find the Kubecost User and select Security Credentials > Create Access Key. Note the Access key ID and Secret access key.
- You can add the Access key ID and Secret access key on /settings.html > External Cloud Cost Configuration (AWS) > Update and setting Service key name to Access key ID and Service key secret to Secret access key
Attach via Pod Annotation on EKS
- First, create an OIDC provider for your cluster with these steps
- Next, create a Role with these steps.
- When asked to attach policies, you’ll want to attach the policies created above in Step 2
- When asked for “namespace” and “serviceaccountname” use the namespace Kubecost is installed in and the name of the serviceaccount attached to the cost-analyzer pod. You can find that name by running
kubectl get pods kubecost-cost-analyzer-69689769b8-lf6nq -n <kubecost-namespace> -o yaml | grep serviceAccount
- Then, you need to add an annotation to that service account as described in these docs. This annotation can be added to the Kubecost service account by setting
.Values.serviceAccount.annotationsin the helm chart toeks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>
Note: If you see the error:
User: ***/assumed-role/<role-name>/### is not authorized to perform: sts:AssumeRole on resource..., you can add the following to your policy permissions to allow the role the correct permissions:
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": "*"
}
]
These values can either be set from the kubecost frontend or via .Values.kubecostProductConfigs in the helm chart. Note that if you set any kubecostProductConfigs from the helm chart, all changes via the frontend will be overridden on pod restart.
athenaProjectIDe.g. "530337586277" # The AWS AccountID where the Athena CUR is.athenaBucketNameAn S3 bucket to store Athena query results that you’ve created that Kubecost has permission to access- The name of the bucket should match
s3://aws-athena-query-results-*, so the IAM roles defined above will automatically allow access to it - The bucket can have a Canned ACL of
Privateor other permissions as you see fit.
- The name of the bucket should match
athenaRegionThe aws region athena is running inathenaDatabasethe name of the database created by the Athena setup- The athena database name is available as the value (physical id) of
AWSCURDatabasein the CloudFormation stack created above (in Step 2: Setting up Athena)
- The athena database name is available as the value (physical id) of
athenaTablethe name of the table created by the Athena setup- The table name is typically the database name with the leading
athenacurcfn_removed (but is not available as a CloudFormation stack resource)
- The table name is typically the database name with the leading
Make sure use only underscore as a delimiter if needed for tables and views, using dash will not work even though you might be able to create it see docs.
- If you are using a multi-account setup, you will also need to set
.Values.kubecostProductConfigs.masterPayerARNTo the arn of the role in the masterpayer account, e.g.arn:aws:iam::530337586275:role/KubecostRole.
Once you've integrated with the CUR, you can visit /diagnostics.html in Kubecost to determine if Kubecost has been successfully integrated with your CUR. If any problems are detected, you will see a yellow warning sign under the cloud provider permissions status header:
You can check pod logs for authentication errors by running
kubectl get pods -n <namespace>
kubectl logs <kubecost-pod-name> -n <namespace> -c cost-model
If you do not see any authentication errors, log in to your AWS console and visit the Athena dashboard. You should be able to find the CUR. Ensure that the database with the CUR matches the athenaTable entered in step 4-- it likely has a prefix with athenacurfn_ :
You can also check query history to see if any queries are failing:
- Activating User-Defined Cost Allocation Tags - AWS Billing and Cost Management
- See Step 2 here for more information on how to supply tags or use existing tags.
Kubecost will reconcile your spot prices with CUR billing reports as they become available (usually 1-2 days), but pricing data can be pulled hourly by integrating directly with the AWS spot feed. To enable, follow these steps:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-data-feeds.html
Note the name of the bucket you created for spot data. This will be used in the following step.
These values can either be set from the kubecost frontend or via .Values.kubecostProductConfigs in the helm chart. Note that if you set any kubecostProductConfigs from the helm chart, all changes via the frontend will be deleted on pod restart
projectID the Account ID of the AWS Account on which the spot nodes are running.
awsSpotDataRegion region of your spot data bucket
awsSpotDataBucket the configured bucket for the spot data feed
awsSpotDataPrefix optional configured prefix for your spot data feed bucket
spotLabel optional Kubernetes node label name designating whether a node is a spot node. Used to provide pricing estimates until exact spot data becomes available from the CUR
spotLabelValue optional Kubernetes node label value designating a spot node. Used to provide pricing estimates until exact spot data becomes available from the CUR. For example, if your spot nodes carry a label lifecycle:spot, then the spotLabel would be "lifecycle" and the spotLabelValue would be "spot"
AWS services used here are:
Kubecost's cost-model requires roughly 2 CPU and 10GB of RAM per 50,000 pods monitored. The backing Prometheus database requires roughly 2CPU and 25GB per million metrics ingested per minute. You can pick the ec2 instances necessary to run kubecost accordingly.
Kubecost can write its cache to disk. Roughly 32 GB per 100,000 pods monitored is sufficient. (Optional-- our cache can exist in memory)
- Cloudformation (Optional-- manual IAM configuration or via Terraform is fine)
- EKS (Optional-- all k8s flavors are supported)
Edit this doc on GitHub