AWS Storage Blog
Improve compute utilization with more Amazon EBS volume attachments on 7th generation Amazon EC2 instances
For many stateful containerized applications, such as those using Kubernetes orchestration, each stateful pod (the smallest deployable container object) may require dedicated persistent storage. A block storage solution is a good fit due to its high performance, low latency, and persistence attributes. If a compute instance has more compute resources to spare, you can only deploy as many stateful pods on an instance as the number of block storage attachments on that instance. You would launch new instances to host new pods beyond what existing instances could support. In other words, a higher number of volume attachments per instance would enable you to increase storage density per instance, reducing total compute costs by improving resource utilization.
On August 2, we introduced Amazon Elastic Compute Cloud (Amazon EC2) M7i instances powered by custom 4th Generation Intel Xeon Scalable processors available only on AWS. Then on August 15, we launched EC2 M7a instances, powered by the 4th Gen AMD EPYC (Genoa) processors. Since then, with the availability of C7i, C7a, R7i, R7iz, R7a, and U7i instances, and more instances in the future, you can attach up to 128 Amazon Elastic Block Store (Amazon EBS) volumes per instance. The new instance volume limit provides a baseline of 32 dedicated EBS volume attachments per virtual instance. The number of attachments scales based on instance size, increasing to 48 dedicated EBS attachments for 16xlarge and up to 128 dedicated EBS attachments for 48xlarge. In comparison, earlier generation Nitro instances have an attachment limit of 28, which was shared with other forms of attachments, such as elastic network interfaces and EC2 locally attached instance store volumes.
In this blog post, we walk through how you can deploy 127 pods on a 7th generation EC2 instance with an Intel or AMD processor using the Kubernetes native API by way of the Amazon EBS Container Storage Interface (CSI) driver. With this, you can host up to 4.5X the number of pods that require dedicated persistent storage per instance, compared to a previous generation instance. By hosting up to 127 containers per instance, you can scale your applications more cost-effectively without needing to provision more instances and only pay for the resources you need. The dedicated EBS volume attachment limit gives you a predictable number of stateful pods your instance can support as you continue to attach more network interfaces or other attachments. Higher storage density also enables you to quickly scale your applications with EBS volumes, reducing overhead to manage resources or redesign your applications.
Solution overview
With the higher number of volume attachments on the larger 7th Generation instances, you can run more containerized applications on these large instances, improving your resource utilization compared to an earlier generation Nitro instance. For example, you may use a powerful EC2 M7i.24xlarge instance with 96vCPUs to balance resources across performance-demanding applications. With M7i.24xlarge, you can deploy up to 63 pods (using 1 of the 64 EBS volumes as a root volume for the instance) with a single instance at $4.8384 per hour, compared to using 3 M6i.24xlarge totaling $13.824 per hour previously, a 65 percent compute cost reduction. As your workloads scale, you can scale by attaching more EBS volumes to host new data while using existing compute resources without having to over-provision compute to meet storage needs.
The new instance families provide dedicated EBS volume attachment limits that are not shared between EBS volumes, network interfaces, or EC2 locally attached instance store volumes. Dedicated volume attachment limits enable you to accurately calculate the remaining number of EBS volume attachments, regardless of your networking or locally attached storage needs. Customers, including Cisco, are excited about the ability to reliably deploy pods that need dedicated persistent storage support using Amazon EBS with Amazon Elastic Kubernetes Service (Amazon EKS), without having to manage dynamic limits shared with additional network interface attachments.
Furthermore, having to launch fewer instances to host the same number of pods also means you can speed up your application deployment. As you continue to scale, you can quickly scale the number of stateful pods by attaching more EBS volumes on demand. If you are migrating from on-premises data centers to AWS, the high number of volume attachments gives you greater flexibility on deployment sizes so that you can lift and shift applications with a broader range of deployment density, reducing migration time to AWS.
Prerequisite
Walkthrough
A dedicated EBS volume limit on the latest instances, including M7i instances, not shared with the network interface or LIS attachments will make sure spec.drivers.allocatable.count is always correct.
In this section, we will walk through two real-world examples of how to deploy stateful pods on the latest generation of instances and EBS CSI driver versions:
1. A new instance volume limit experience with automated adjustments in the EBS CSI driver.
2. A new instance volume limit experience requiring manual adjustments in the EBS CSI driver.
Example 1: A new instance volume limit experience with automated adjustments in the EBS CSI driver
Starting with Amazon EBS CSI driver version v1.25.0 and Amazon EKS add-on version v1.25.0 , volume attachment limits for M7i, M7a, C7i, C7a, R7i, R7iz, and R7a instance types are automatically calculated.
1. In our first example, we will walk through deploying 127 stateful pods on a single 7th generation instance with the new volume attachment limit that is automatically supported by the EBS CSI driver. We have created a small demo cluster using the eksctl utility with the following YAML configuration file. It’s a managed node group with just one M7i.48xlarge node and the Amazon EBS CSI driver as an EKS managed add-on.
# eksctl YAML configuration file
$ cat demo-cluster-ebs-csi-m7i.yaml
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: demo-cluster-ebs-csi-m7i
region: eu-west-1
version: "1.27"
cloudWatch:
clusterLogging:
enableTypes:
- "api"
- "controllerManager"
- "scheduler"
iam:
withOIDC: true
addons:
- name: vpc-cni
version: v1.14.1-eksbuild.1
attachPolicyARNs: #optional
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- name: aws-ebs-csi-driver
version: v1.25.0
attachPolicyARNs: #optional
- "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
managedNodeGroups:
- name: mnodegroup
instanceType: m7i.48xlarge
desiredCapacity: 1
minSize: 1
maxSize: 3
privateNetworking: true
$ eksctl create cluster -f demo-cluster-ebs-csi-m7i.yaml
2023-08-30 14:12:09 [ℹ] eksctl version 0.154.0-dev+f73436487.2023-08-24T11:48:09Z
2023-08-30 14:12:09 [ℹ] using region eu-west-1
2023-08-30 14:12:09 [ℹ] setting availability zones to [eu-west-1b eu-west-1c eu-west-1a]
2023-08-30 14:12:09 [ℹ] subnets for eu-west-1b - public:192.168.0.0/19 private:192.168.96.0/19
2023-08-30 14:12:09 [ℹ] subnets for eu-west-1c - public:192.168.32.0/19 private:192.168.128.0/19
2023-08-30 14:12:09 [ℹ] subnets for eu-west-1a - public:192.168.64.0/19 private:192.168.160.0/19
2023-08-30 14:12:09 [ℹ] nodegroup "mnodegroup" will use "" [AmazonLinux2/1.27]
2023-08-30 14:12:09 [ℹ] using Kubernetes version 1.27
2023-08-30 14:12:09 [ℹ] creating EKS cluster "demo-cluster-ebs-csi-m7i" in "eu-west-1" region with managed nodes
...
2023-08-30 14:29:44 [ℹ] creating addon
2023-08-30 14:30:38 [ℹ] addon "aws-ebs-csi-driver" active
2023-08-30 14:30:38 [✔] EKS cluster "demo-cluster-ebs-csi-m7i" in "eu-west-1" region is ready
# check EBS CSI add-on version
$ aws eks describe-addon --cluster-name demo-cluster-ebs-csi-m7i --addon-name aws-ebs-csi-driver
{
"addon": {
"addonName": "aws-ebs-csi-driver",
"clusterName": "demo-cluster-ebs-csi-m7i",
"status": "ACTIVE",
"addonVersion": "v1.25.0",
"health": {
"issues": []
},
"addonArn": "arn:aws:eks:eu-west-1:<redacted>:addon/demo-cluster-ebs-csi-m7i/aws-ebs-csi-driver/e2c5234e-d832-bce8-14ac-ece40e6516f2",
…
"serviceAccountRoleArn": "arn:aws:iam::<redacted>:role/eksctl-demo-cluster-ebs-csi-m7i-addon-aws-eb-Role1-1WK8LIZY4XUU1",
"tags": {}
}
}
# check the node
$ kubectl get no ip-<redacted>.eu-west-1.compute.internal -o yaml | grep node.kubernetes.io/instance-type
node.kubernetes.io/instance-type: m7i.48xlarge
2. The value for spec.drivers.allocatable in the CSInode object is properly calculated as 127 (128 possible EBS attachments for M7i.48xlarge minus 1 EBS root volume) and will always stay correct, even in case additional network interfaces are attached. Let’s check:
# check m7i.48xlarge CSInode object
$ kubectl get csinode ip-<redacted>.eu-west-1.compute.internal -o yaml
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
annotations:
storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-volume
...
name: ip-<redacted>.eu-west-1.compute.internal
ownerReferences:
- apiVersion: v1
kind: Node
...
spec:
drivers:
- allocatable:
count: 127
name: ebs.csi.aws.com
nodeID: i-<redacted>
topologyKeys:
- topology.ebs.csi.aws.com/zone
3. Next, let’s create an Amazon EBS CSI-based StorageClass to define configurations for new, dynamically provisioned volumes and make it the default one:
$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 10d
# remove the "default" annotation on StorageClass gp2
$ kubectl annotate sc gp2 storageclass.kubernetes.io/is-default-class-
storageclass.storage.k8s.io/gp2 annotated
$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 10d
$ cat aws-ebs-csi-gp3-sc.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp3
annotations:
storageclass.kubernetes.io/is-default-class: "true"
allowVolumeExpansion: true
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
type: gp3
$ kubectl apply -f aws-ebs-csi-gp3-sc.yaml
storageclass.storage.k8s.io/gp3 created
$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 10d
gp3 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 29s
4. As we continue to deploy stateful pods, the EBS CSI driver will dynamically provision volumes based on the storage class we defined, and pods claim their usage using PersistentVolumeClaim. Let’s create 127 PersistentVolumeClaim objects of minimum EBS gp3 size of 1 GiB. We will use a small shell script for this:
# create a script called "pvc.sh" with the following content
$ cat pvc.sh
#!/bin/bash
for i in {1..127}
do
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc$i
spec:
storageClassName: gp3
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF
done
# make the script executable for user
$ chmod u+x pvc.sh
# run the script (output shortened for brevity)
$ ./pvc.sh
persistentvolumeclaim/pvc1 created
persistentvolumeclaim/pvc2 created
persistentvolumeclaim/pvc3 created
...
persistentvolumeclaim/pvc126 created
persistentvolumeclaim/pvc127 created
5. The PVC will remain in Pending
state until a pod is using them.
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc1 Pending gp3 4m56s
...
pvc126 Pending gp3 4m36s
pvc127 Pending gp3 4m35s
$ kubectl describe pvc pvc1
Name: pvc1
Namespace: default
StorageClass: gp3
Status: Pending
Volume:
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: pod1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 2m29s (x82 over 22m) persistentvolume-controller waiting for first consumer to be created before binding
6. Now let us create 127 pods that use the PVC and assign all pods to the one node we have. Again, we are using a small shell script:
# create a script called "pvc.sh" with the following content
$ cat pod.sh
#!/bin/bash
for i in {1..127}
do
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: pod$i
spec:
containers:
- name: container
image: busybox
args: ["sleep","3600"]
volumeMounts:
- mountPath: "/data"
name: pvc-storage
nodeSelector:
eks.amazonaws.com/nodegroup: mnodegroup
volumes:
- name: pvc-storage
persistentVolumeClaim:
claimName: pvc$i
EOF
done
# make the script executable for user
$ chmod u+x pod.sh
# run the script (output shortened for brevity)
$ ./pod.sh
pod/pod1 created
pod/pod2 created
pod/pod3 created
...
pod/pod127 created
pod/pod128 created
# check thatPVC are created (output shortened for brevity)
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc1 Bound pvc-782be80e-4f78-47ce-b785-fcff56f1c208 1Gi RWO gp3 2m57s
...
pvc126 Bound pvc-9fdf5b11-7d32-4751-9653-4fff4cbef533 1Gi RWO gp3 116s
pvc127 Bound pvc-6884710a-834c-4048-9240-78f9a24f17de 1Gi RWO gp3 114s
# check that pods are running (output shortened for brevity)
$ k get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod1 1/1 Running 0 77s 192.168.103.181 ip-<redacted>.ec2.internal <none> <none>
...
pod126 1/1 Running 0 16s 192.168.122.239 ip-<redacted>.ec2.internal <none> <none>
pod127 1/1 Running 0 14s 192.168.120.10 ip-<redacted>.ec2.internal <none> <none>
7. Now all 127 pods are running with PVC attached on M7i.48xlarge node ip-<redacted>.ec2.internal.
Example 2: A new instance volume limit experience requiring manual adjustments in the EBS CSI driver
As future instance families are released with a higher number of attachments, Amazon EBS CSI driver through Helm chart and Amazon EKS-managed add-on may not yet support automatic adjustment of EBS attachment limits for the new instance family until the next CSI driver release cycle. For example, as of the publication of this blog, EBS CSI Helm chart v1.25.0 and EKS managed add-on v1.25.0 do not yet support automatically determining dedicated EBS attachments for the U7i instance family.
It is possible to manually override the maximum number of EBS volume attachments volumeAttachLimit (in the boundaries given by the instance type) either initially during cluster creation or later on by either modifying the Amazon EBS CSI Helm charts value.yaml or using the EKS managed add-on advanced configuration for aws-ebs-csi-driver.
In the second example, we will walk through deploying 31 stateful pods on a U7i instance with the new volume attachment limit that require manual adjustments in the Amazon EBS CSI driver. The example will set the value for the whole ebs-csi-node DaemonSet and will be applied to all nodes in the cluster irrespective of instance type. For full control in EKS clusters using different instance types, one can make use of the “Additional Node DaemonSets Feature”, an advanced configuration option available with EBS CSI driver version v1.23 and above.
1. Let’s start with the initial installation of EBS CSI driver through EKS managed add-on. For a brand-new cluster, it is possible to directly create the Amazon EBS CSI add-on with the proper volumeAttachmentLimit. Again, I have used eksctl with the following configuration YAML with an U7i instance. Amazon EBS CSI modification is applied using attribute configurationValues in the add ons section of the aws-ebs-csi-driver add-on:
# eksctl YAML configuration file
$ cat demo-cluster-ebs-csi-u7i.yaml
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: demo-cluster-ebs-csi
region: eu-west-1
version: "1.27"
cloudWatch:
clusterLogging:
enableTypes:
- "api"
- "controllerManager"
- "scheduler"
iam:
withOIDC: true
addons:
- name: vpc-cni
version: v1.14.1-eksbuild.1
attachPolicyARNs: #optional
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- name: aws-ebs-csi-driver
version: v1.25.0
attachPolicyARNs: #optional
- "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
# doc: https://eksctl.io/usage/addons/
configurationValues: |-
node:
volumeAttachLimit: 31
managedNodeGroups:
- name: mnodegroup
instanceType: u7i.4xlarge
desiredCapacity: 1
minSize: 1
maxSize: 3
privateNetworking: true
2. It is possible to update an already-provisioned Amazon EBS CSI driver to reflect the higher instance volume limit for U7i nodes. Let’s modify an already installed EBS CSI driver using Helm chart:
# update values.yaml with volumeAttachLimit for EBS CSI node DaemonSet
$ cat values.yaml
...
node:
# The "maximum number of attachable volumes" per node
volumeAttachLimit: 31
...
# apply change
$ helm upgrade --install aws-ebs-csi-driver --namespace kube-system \
aws-ebs-csi-driver/aws-ebs-csi-driver -f values.yaml
Besides, Amazon EKS managed add-ons support advanced configuration for aws-ebs-csi-driver. Let’s look at how to modify an already-installed CSI driver through the EKS managed add-on. We will utilize the jq utility to make the output more easily readable. For more information on jq and installation instructions, visit the jq link on GitHub.
# describe current add-on version and configuration
$ aws eks describe-addon --cluster-name demo-cluster-ebs-csi --addon-name aws-ebs-csi-driver{
"addon": {
"addonName": "aws-ebs-csi-driver",
"clusterName": "demo-cluster-ebs-csi",
"status": "ACTIVE",
"addonVersion": "v1.25.0",
...
}
# describe add-on configuration schema
$ aws eks describe-addon-configuration --addon-name aws-ebs-csi-driver --addon-version v1.25.0 --query configurationSchema --output text | jq
{
...
"$schema": "https://json-schema.org/draft/2019-09/schema",
"additionalProperties": false,
"description": "Configurable parameters of the AWS EBS CSI Driver",
"properties": {
..
"node": {
"additionalProperties": false,
"properties": {
"additionalArgs": {
"default": [],
"description": "Additional arguments passed to the node pod",
"items": {
"type": "string"
},
"type": "array"
},
...…
"volumeAttachLimit": {
"default": null,
"description": "Overrides the maximum number of volumes that can be attached per node (for all nodes)",
"minimum": 0,
"type": [
"integer",
"null"
]
}
...
# create a advanced configuration YAML file
$ cat node-attachments.yaml
"node":
"volumeAttachLimit": 31
# apply advanced config change and check
$ aws eks update-addon --cluster-name demo-cluster-ebs-csi --addon-name aws-ebs-csi-driver \
--addon-version v1.25.0 --configuration-values 'file://node-attachments.yaml'
{
"update": {
"id": "1a9622f7-6b57-373f-a98b-a59c110b7dfe",
"status": "InProgress",
"type": "AddonUpdate",
"params": [
{
"type": "AddonVersion",
"value": "v1.25.0"
},
{
"type": "ConfigurationValues",
"value": "\"node\":\n \"volumeAttachLimit\": 31"
}
],
...
}
}
$ aws eks describe-addon --cluster-name demo-cluster-ebs-csi --addon-name aws-ebs-csi-driver
{
"addon": {
"addonName": "aws-ebs-csi-driver",
"clusterName": "demo-cluster-ebs-csi",
"status": "ACTIVE",
"addonVersion": "v1.25.0",
...
"configurationValues": "\"node\":\n \"volumeAttachLimit\": 31"
}
}
3. Now that we have the updated CSI driver with the new instance volume limit, let’s check the Amazon EKS ebs-csi-node DaemonSet for the new volume-attach-limit:
$ kubectl get ds -n kube-system ebs-csi-node -o yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
…
name: ebs-csi-node
namespace: kube-system
…
containers:
- args:
- node
- —endpoint=$(CSI_ENDPOINT)
- —volume-attach-limit=31
- —logging-format=text
- —v=2
…
4. With the CSI driver ready, we will then begin to deploy stateful pods. The DaemonSet will start a new pod on each node and reregister the CSInode object with the proper spec.allocatable.count automatically.
$ kubectl get csinodes.storage.k8s.io
NAME DRIVERS AGE
ip-<redacted>.ec2.internal 1 12m
$ kubectl get csinodes.storage.k8s.io ip-<redacted>.ec2.internal -o yaml
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
annotations:
storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-volume
...
ownerReferences:
- apiVersion: v1
kind: Node
name: ip-<redacted>.ec2.internal
...
spec:
drivers:
- allocatable:
count: 31
name: ebs.csi.aws.com
nodeID: i-<redacted>
topologyKeys:
- topology.ebs.csi.aws.com/zone
5. We see here that creating PVC and attaching pods to the U7i node works similarly compared to the first example using M7i with the latest CSI driver v1.25.0.
Amazon EBS performance considerations
EBS volumes attached to the same instance share the instance’s resources, such as the maximum Amazon EBS bandwidth. Although larger-sized instances have more resources to share, you should still make sure each volume is getting enough resources to run your applications as you attach more to the instance.
You can use CloudWatch metrics at the EC2 instance level and at the EBS volume level to monitor performance. As a reference, follow Amazon EBS metrics for Nitro-based instances, which describe instance-level metrics like EBSReadBytes, EBSWriteBytes, and Amazon EBS metrics, which reference volume-level metrics like VolumeReadBytes and VolumeWriteBytes.
Cleaning up
To avoid future costs, delete all the resources, including the EKS cluster created for this exercise. This action, in addition to deleting the cluster, will also delete the node group. Here we will delete both the M7i and U7i clusters we created in the examples:
# in every cluster context delete all pods first, followed by pvc
# M7i based EKS cluster "demo-cluster-ebs-csi-m7i"
$ for i in {1..127}; do kubectl delete pod pod$i; done
$ for i in {1..127}; do kubectl delete pvc pvc$i; done
$ eksctl delete cluster -f demo-cluster-ebs-csi-m7i.yaml
# U7i based EKS cluster "demo-cluster-ebs-csi"
$ for i in {1..31}; do kubectl delete pod pod$i; done
$ for i in {1..31}; do kubectl delete pvc pvc$i; done
$ eksctl delete cluster -f demo-cluster-ebs-csi.yaml
Conclusion
In this blog post, we explored how the new instance volume limit on the 7th generation EC2 instances with Intel and AMD processors gives you predictability for compute resources needed while lowering total compute costs. We walked through how you can use the higher number of volume attachments on new instances to schedule more pods, with different tools to manage your Amazon EBS volumes. If you are running containerized applications that require dedicated persistent storage, you now know how to scale more cost-effectively on AWS by hosting up to 127 pods or tasks per EC2 instance. For those of you running databases on AWS using larger-sized EC2 instances with a higher number of EBS volume attachment limits, you can now fully utilize the memory and vCPU available on these powerful EC2 instances as your database storage footprints grow.
To get started, follow the Amazon EBS CSI User Guide and the open source project on Github. To learn more, you can visit the Amazon EC2 product page and the Amazon EBS product page.
If you have any comments or questions, then don’t hesitate to leave them in the comments section.