AWS Storage Blog

High-performance cloud storage comes of age with Amazon FSx for Lustre

The rapid maturation of cloud tools for high-performance workloads in the past several years has made it possible for household names like T-Mobile, Toyota, and Rivian to move their high-performance analytics and AI/ML environments to the cloud. These are hugely data-intensive workflows that many companies five years ago believed would never be able to be moved to the cloud.

AWS has steadily knocked down the big objections to cloud computing —economics, security, availability— one by one, proving that the cloud can deliver similar or superior results compared with most on-premises environments. As data repositories grow and the need to access more data faster grows, the debate continues on how the cloud can deliver the sheer data access performance needed.

Enter Amazon FSx for Lustre. One of four fully managed industry file systems currently offered under the FSx umbrella (along with FSx for NetApp ONTAP, FSx for Windows File Server, and FSx for ZFS), FSx for Lustre makes it simple and cost-effective to launch, run, and scale feature-rich, high-performance file systems in the cloud. It supports a wide range of workloads with its reliability, security, scalability, and broad set of capabilities. Each of these file systems has its place in helping customers migrate to the cloud, and FSx for Lustre’s place tends to be in high-performance analytics and ML model training.

Use cases

Based on the popular open-source Lustre file system that was originally developed to serve the huge data access appetites of some of the highest performance, on-premises compute clusters in the world, FSx for Lustre is used in traditional high-performance computing but can be found just as often in commercial applications across Analytics as a Service (AaaS), ML as a Service (MLaaS), Backup as a Service (BaaS), genomics, media rendering, customer analytics, and financial risk management.

Often used with Amazon Elastic Container Service, Amazon Elastic Kubernetes Service, AWS ParallelCluster, or Amazon SageMaker, FSx for Lustre file systems that deliver up to 100s of GB/s of bandwidth and millions of IOPs can be configured and deployed in just a few minutes. FSx for Lustre has the highest level of throughput of any cloud file storage service, and we just made it even faster. Per the announcement of FSx for Lustre Next Gen File systems at re:Invent 2021, FSx for Lustre just added:

  • up to 5x faster total performance
  • up to 60 percent improvement on price/performance

These new capabilities were launched alongside a number of enhancements to FSx for Lustre’s Amazon S3 integration, which build on FSx for Lustre’s unique integration with Amazon Simple Storage Service (Amazon S3)—automating the bidirectional propagation of changes between FSx for Lustre and any Amazon S3 bucket(s) it is linked to.

What does all this add up to for AWS customers? Workloads that were kept on premises due to bandwidth or latency concerns can now run in the cloud, often faster and less expensively than on-premises. For example, Qubole, who uses FSx for Lustre as the backing high-performance data store for analytics as a service, avoided up to 68 percent in cost and improved performance by 3x. Hive Visual Effects leverages S3-linked FSx for Lustre for fast, cost-effective data access across their studio in the cloud workloads. They’ve expressed appreciation for its ease of management, simple integration with S3, and set-up speed. “I can spin up an Amazon FSx for Lustre file system in five minutes, and it’s all managed by AWS,” says Bernie Kimbacher, founder of Hive VFX.

Learn more

See for yourself what FSx for Lustre can do by checking out the latest announcements and deep dives from re:Invent and learning from real-world customer experiences.

If you have any questions about Amazon FSx for Lustre, feel free to ask in the comments.