Advanced EKS Multi-Tenancy with Fargate and Karpenter

Introduction

Amazon Elastic Kubernetes Service (EKS) is a powerful platform for running containerized workloads at scale. However, as organizations grow, they often need to support multi-tenancy—running multiple teams, applications, or customers on a single cluster while ensuring isolation, security, and cost efficiency.

This article provides a comprehensive guide to implementing advanced multi-tenancy on EKS using Fargate (serverless compute) and Karpenter (autoscaling). We’ll cover:

1. Kubernetes Namespaces for logical isolation.
2. Resource Quotas to enforce resource limits.
3. IAM Roles for Service Accounts (IRSA) for fine-grained access control.
4. Fargate and Karpenter for efficient resource provisioning.

By the end of this guide, you’ll have a fully functional multi-tenant EKS cluster with cost optimization, scalability, and security.

1. Overview of Key Components

1.1. Kubernetes Namespaces

• What: Namespaces provide logical isolation for resources within a cluster.
• Use Case: Separate teams, applications, or environments (e.g., dev, prod).

1.2. Resource Quotas

• What: Enforce limits on resource usage (CPU, memory, storage) per namespace.
• Use Case: Prevent a single tenant from consuming all cluster resources.

1.3. IAM Roles for Service Accounts (IRSA)

• What: Assign AWS IAM roles to Kubernetes service accounts for fine-grained access control.
• Use Case: Restrict access to AWS resources (e.g., S3, DynamoDB) based on namespace.

1.4. Fargate

• What: Serverless compute for Kubernetes pods.
• Use Case: Isolate workloads at the pod level without managing nodes.

1.5. Karpenter

• What: Open-source autoscaler for Kubernetes.
• Use Case: Dynamically provision nodes based on workload requirements.

2. Architecture Design

The architecture consists of:

• Namespaces: Logical separation for tenants (tenant-a, tenant-b).
• Resource Quotas: Enforce limits per namespace.
• IRSA: Assign IAM roles to service accounts in each namespace.
• Fargate: Run pods in isolated environments.
• Karpenter: Autoscale nodes for cost efficiency.

3. Step-by-Step Implementation

3.1. Prerequisites

• AWS CLI: Installed and configured.
• kubectl: Installed and configured to access your EKS cluster.
• eksctl: Installed for managing EKS clusters.

# Install eksctl
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin

3.2. Create an EKS Cluster

AWS Console:

1. Navigate to EKS > Clusters > Create Cluster.
2. Choose Fargate as the compute type.
3. Enable IAM Roles for Service Accounts (IRSA).

CLI:

bash

Copy

eksctl create cluster \
  --name multi-tenant-cluster \
  --region us-east-1 \
  --fargate

3.3. Configure Namespaces

Create namespaces for each tenant (tenant-a, tenant-b).

CLI:

kubectl create namespace tenant-a
kubectl create namespace tenant-b

Verify:

kubectl get namespaces

3.4. Set Up Resource Quotas

Define resource quotas for each namespace.

tenant-a-quota.yaml:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "4"
    requests.memory: "8Gi"
    limits.cpu: "8"
    limits.memory: "16Gi"

Apply Quotas:

kubectl apply -f tenant-a-quota.yaml
kubectl apply -f tenant-b-quota.yaml

Verify:

kubectl describe resourcequota -n tenant-a

3.5. Configure IAM Roles for Service Accounts (IRSA)

1. Create IAM Policies:
   • Define policies for each tenant (e.g., tenant-a-s3-read-only).
2. Create IAM Roles:
   • Use eksctl to create roles and associate them with service accounts.

CLI:

eksctl create iamserviceaccount \
  --name tenant-a-sa \
  --namespace tenant-a \
  --cluster multi-tenant-cluster \
  --attach-policy-arn arn:aws:iam::123456789012:policy/tenant-a-s3-read-only \
  --approve

Verify:

kubectl describe sa tenant-a-sa -n tenant-a

3.6. Deploy Fargate Profile

Fargate profiles determine which pods run on Fargate.

AWS Console:

1. Navigate to EKS > Clusters > multi-tenant-cluster > Compute > Add Fargate Profile.
2. Specify namespaces (tenant-a, tenant-b).

CLI:

eksctl create fargateprofile \
  --cluster multi-tenant-cluster \
  --name tenant-a-profile \
  --namespace tenant-a

3.7. Install and Configure Karpenter

Karpenter dynamically provisions nodes based on workload requirements.

Install Karpenter:

helm repo add karpenter https://charts.karpenter.sh
helm install karpenter karpenter/karpenter --namespace karpenter --create-namespace

Configure Provisioner:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ["spot", "on-demand"]
  limits:
    resources:
      cpu: 100
      memory: 1000Gi
  provider:
    instanceProfile: KarpenterNodeInstanceProfile

Apply Provisioner:

kubectl apply -f provisioner.yaml

3.8. Deploy Sample Workloads

Deploy sample applications to test multi-tenancy.

tenant-a-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tenant-a-app
  namespace: tenant-a
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tenant-a-app
  template:
    metadata:
      labels:
        app: tenant-a-app
    spec:
      serviceAccountName: tenant-a-sa
      containers:
      - name: nginx
        image: nginx
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "1"
            memory: "1Gi"

Apply Deployment:

kubectl apply -f tenant-a-deployment.yaml

3.9. Monitor and Optimize

• CloudWatch: Monitor cluster metrics (CPU, memory).
• Karpenter: Review node provisioning logs.
• Cost Explorer: Analyze cost savings from Spot Instances and Fargate.

4. Best Practices

1. Namespace Isolation: Use network policies (e.g., Calico) to restrict pod communication.
2. Quota Management: Regularly review and adjust quotas based on usage.
3. IRSA: Use least-privilege IAM policies.
4. Fargate: Use for short-lived or bursty workloads.
5. Karpenter: Combine Spot and On-Demand instances for cost optimization.

5. Conclusion

By combining Kubernetes Namespaces, Resource Quotas, IRSA, Fargate, and Karpenter, you can build a secure, scalable, and cost-efficient multi-tenant EKS cluster. This setup ensures isolation, enforces resource limits, and optimizes costs while maintaining high availability.

Further Reading

• EKS Best Practices
• Karpenter Documentation
• Fargate Pricing
• IRSA Guide

This guide provides a robust foundation for advanced EKS multi-tenancy. Experiment with the configurations and adapt them to your specific use case!