The true cost of Kubernetes: People, Time and Productivity

December 21, 2021

Yann Léger

Yann Léger
@yann_eu

Alisdair Broshar

Alisdair Broshar
@AlisdairBroshar

8 min read

While writing a comparison of Kubernetes and Koyeb, we tried to determine how much operating a Kubernetes cluster really costs. This section of our comparison took us hours to write and ended up being so long that we decided to write a dedicated post about it.

Full disclaimer: At Koyeb, we're building a serverless platform and we have a purpose-built orchestration engine. We're heavy users of Kubernetes for our own control plane, which, sadly, can't run on Koyeb as it would otherwise generate circular dependencies. We also previously ran our users workloads on Kubernetes, but decided to switch to a custom stack composed of Nomad, Firecracker, and Kuma.

Kubernetes is a proven technology for container orchestration and is a great tool for running resilient applications, but the true cost is often underestimated. Our comparison explored the opportunity costs related to developer experience and managed responsibilities, this post investigates the actual financial costs of using Kubernetes.

When building a production platform, you can select different layers of abstraction. We distinguish four major abstraction layers you can pick from. If you're not an infrastructure company, we advocate building on top of cloud service provider primitives. Spoiler alert: We estimate that operating Kubernetes clusters costs at least $100k and can easily cost more than $500k annually for a ridiculously low amount of resources.

The cost of Kubernetes on Azure, GCP, and AWS

Kubernetes is an open-source technology, but open-source doesn't mean it's free to operate. The operational costs depend on how you choose to use Kubernetes. When opting to use Kubernetes, you have two main options:

  1. Self-hosted Kubernetes
  2. Managed Kubernetes

Self-hosted Kubernetes

If you go with a self-hosted Kubernetes solution, you will need to budget for the nodes needed to run your workloads, but you will also need to add the costs of the nodes to run your control plane and of the team to manage the operation.

For a reliable Kubernetes production environment, you will need to create and manage nodes that host your control plane, in addition to separate worker nodes to host your application workloads. The control plane is where the main Kubernetes components needed to configure and orchestrate your workloads live. The control plane runs key processes including:

  • kube-api-server to expose the Kubernetes API,
  • etcd, a highly-available key value store, to store all cluster data,
  • kube-schedule, which schedules newly created Pods,
  • and more processes like the kube-controller-manager and cloud-controller-manager.

Technically, you could use the same nodes to host the control plane and worker nodes, but it is not recommended for production environments.

Self-hosted Kubernetes total cost

For this calculation, we decided to do the cost simulation using nodes running on Azure. We did the estimation using D2s v3 Azure VMs machines, which have 2 vCPUs and 8 GB RAM and used 3 nodes which is the minimum for redundancy purposes. With that, you should have 5.4GB of RAM per node, so ~16GB of usable RAM and 6vCPU.

Secondly, we took the average DevOps compensation in the US, $141,000 (cited by Builtin) and multiplied it by 4 to represent a team of four engineers, which is needed to properly cover 24/7 operations.

As a result, here is what the total cost of ownership (TCO) for self-hosted Kubernetes looks like:

ExpenseAnnual Cost
Nodes to host the control plane of your clusters$0.096 x 24 x 365 x 3 = $2523
Nodes to host application workload$0.096 x 24 x 365 x 3 = $2523
Compensation for members on your DevOps/SRE team$141,000 x 4 = $564,000

Annual total base cost: $2523 + $2523 + $564,000 = $569,046

This cost estimation doesn't include the sanity of your DevOps team when issues arise, which is, we believe, priceless.

Managed Kubernetes on Azure, GCP, or AWS

Then there is the managed Kubernetes route. With a managed Kubernetes solution, you do not need to handle creating or managing your Kubernetes clusters, nor do you need to worry about creating the control plane or installing its components like etcd, kube-apiserver, or kube-scheduler.

GKE, EKS, and AKS: The cost of the managed control plane

The 3 biggest managed Kubernetes providers are GCP, AWS, and Azure with Google Kubernetes Engine (GKE), Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS). From what we found, the pricing is similar for the managed control plane of all these services at $0.10/hour.

Managed Kubernetes ProviderCost to run one cluster
GCP GKE$0.10 x 24 x 365 = $876
AWS EKS$0.10 x 24 x 365 = $876
Azure AKS$0.10 x 24 x 365 = $876

For all these offerings, there are no automatic version updates or auto-recovery and you still need to pay for the computing resources like CPU, memory, and ephemeral storage that your worker pods consume.

Cost of the worker nodes

The cost of worker nodes also varies across cloud service providers and depends on the amount of computing resources you need as well as in which region of the world your servers are running. We multiply by 3 for redundancy purposes.

ProviderResourcesAnnual Cost
AWS m6gd.large EC2 instances2 vCPUs and 8 GB RAM$0.0904 x 24 x 365 x 3 = $2376
Azure D2s v3 VM machines2 vCPUs and 8 GB RAM$0.096 x 24 x 365 x 3 = $2523
GCP e2-standard-2 VMs2 vCPUs and 8 GB RAM$0.067 x 24 x 365 x 3 = $1761

With these instances, you should have 5.4GB of RAM per node, so ~16GB of usable RAM and 6vCPU.

Managed Kubernetes total cost

You also need to account for the maintenance responsibilities not covered. You'll need at least one dedicated DevOps person to monitor this managed solution and handle the responsibilities not covered by these offerings.

Managed Kubernetes ProviderCost to run one clusterCost of worker nodesSalary for dedicated DevOpsTCO
AWS EKS$876$2376$141,000$144,252
Azure AKS$876$2523$141,000$144,399
GCP GKE$876$1761$141,000$143,637

This option buys you a lot more time and energy to focus on developing and improving your applications, but you will still be stuck with a lot of the Kubernetes responsibilities and complexity depending on the cloud service provider you use.

Serverless container solutions are a more efficient alternative

Kubernetes is Greek for helmsman or pilot, so the question becomes: is this a fleet you really want to command? If it's not your core business, we advocate you shouldn't.

We're strong believers that, in the majority of cases, a serverless container solution combined with managed databases and object storage will be way more efficient than going with Kubernetes. Kubernetes is a generalist platform that leaves a lot of operational work in the hands of the user but all modern cloud providers now offer advanced serverless container solutions.

The key benefits of serverless container solutions are:

  • you have no upfront cost with no minimum number of nodes running
  • the nodes supporting your containers are completely managed, you don't have to think about security upgrades or orchestration of your containers
  • you keep the flexibility and portability of containers
  • scaling is completely seamless

Some might argue that resources (namely RAM and CPU) are more costly with serverless solutions but this is without accounting for the management costs of Kubernetes we highlighted above.

Serverless solutions provide different levels of complexity and maturity and we would, in a totally biased manner, recommend Koyeb. If you want to go with AWS or GCP, we recommend to look at AWS Fargate and GCP Cloud Run. All of these solutions remove the hassle of managing Kubernetes clusters.

For databases, we previously wrote about managed databases solutions for Postgres, MySQL, and Redis, that can be combined with serverless container solutions.

Serverless solutions are not magic, they require some configuration, but you don't need strong system engineering skills to work on it and that's the fundamental difference.

Looking forward to the Serverless future

Looking only at the execution cost of your containers is like only looking at the tip of the iceberg. We believe serverless solutions should also encompass advanced features including:

  • seamless continuous deployment
  • transparent load-balancing with native TLS
  • native secure and private networking with service mesh capabilities
  • easy global deployment

That's why built them natively into the Koyeb serverless platform. You can use Koyeb to host web apps and services, Docker containers, APIs, event-driven functions, cron jobs, and more. The platform has built-in Docker container deployment and also provides git-driven continuous deployment!

Discover the serverless experience by signing up today. Also, feel free to join us over on Slack.

And if you want to help build a serverless cloud service provider, we're hiring!