AWS re:Invent recap: On-premises access to cloud storage with AWS Storage Gateway

In December 2020, I presented a session at AWS re:Invent: “Get on-premises access to cloud storage in minutes with AWS Storage Gateway.” AWS Storage Gateway is a hybrid cloud storage service that provides on-premises access to virtually unlimited cloud storage. In my session, I talked about how you can use Storage Gateway to overcome on-premises storage challenges. I also cover how you can easily get started with using cloud storage in your data centers and edge locations without changing your applications.

In this blog, I provide a recap of my session detailing AWS Storage Gateway capabilities, benefits, and use cases, in addition to sharing real-world customer examples. My 30-minute session is now available to watch on-demand.

About AWS Storage Gateway

AWS Storage Gateway provides a fast and easy path to using cloud storage at your premises while minimizing changes to your current business workflows and storage experience. The service helps reduce your on-premises storage footprint by seamlessly integrating on-premises applications with AWS Storage. Storage Gateway consists of two parts. One is the gateway that typically runs on a virtual machine (VM) or a hardware appliance in your facilities. The other is the service running in the cloud to which the on-premises gateway connects to. The gateway sits in your environment and presents storage protocol interfaces such as NFS/SMB, iSCSI block, or iSCSI virtual tape. This means that your applications can connect to it without you having to rewrite or refactor the applications. The gateway in turn connects these on-premises applications and workloads to AWS Storage services. This enables you to store your data durably and securely in Amazon S3, Amazon S3 Glacier, S3 Glacier Deep Archive, or as Amazon EBS snapshots.

AWS Storage Gateway brings the manageability, durability, scalability, and security of AWS to existing on-premises environments through native integration with AWS Storage services and other AWS management services. For example, Storage Gateway integrates with Amazon CloudWatch for monitoring, AWS CloudTrail for logging and auditing, AWS Identity and Access Management (AWS IAM) for securely managing access to the service and resources, and AWS Key Management Service (AWS KMS) for managing encryption.

AWS Storage Gateway types

Within the AWS Storage Gateway service, there are three types of gateways. File Gateway enables you to store and retrieve objects in Amazon S3 using file protocols, such as NFS or SMB. Files written through File Gateway can be directly accessed in AWS as Amazon S3 objects or on premises as files. Tape Gateway provides your backup applications with an iSCSI virtual tape library (VTL) interface allowing you to store your backups and archives in S3. Volume Gateway provides cloud-backed block storage to your applications using the iSCSI protocol. Data on the volumes is stored in S3 and can be backed up as Amazon EBS snapshots in the cloud.

AWS Storage Gateway benefits

AWS Storage Gateway supports multiple storage protocols enabling you to seamlessly connect on-premises applications to AWS Storage without having to modify those applications. Storage Gateway also presents a local storage cache, which provides low latency access to frequently used data. So, your most often used data by applications can be served up from the local cache that’s next to your application servers with minimal latency.

AWS Storage Gateway optimizes and secures data transfers between the gateway and the cloud. The gateway compresses the data before synchronizing to the cloud and it also fetches only the data that’s needed when data requested is not in the cache. This helps speed up your data transfer and optimizes response time to application I/O. With Storage Gateway, you can present virtually unlimited storage to on-premises applications and only pay for what you consume. Finally, the gateways deployed at your premises are managed, monitored, and updated from the cloud enabling you to simplify management.

AWS Storage Gateway use cases

There are several common use cases customers focus on when moving storage to the cloud to simplify on-premises storage management.

The first use case is using cloud storage for backups of on-premises applications to reduce data center infrastructure and storage administration costs. With AWS Storage Gateway, you can back up your on-premises workloads to the cloud, move your tape-based archives to the cloud, and store data that you must retain long term in the cloud.

The second use case is to provide cloud-backed file shares to on-premises applications. Many customers have on-premises applications that need easy to use, cost-effective, and scalable file storage. Often, they run out of capacity on their on-premises storage arrays, and face expensive hardware replacement cycles every few years. Customers have also realized that many of their on-premises file workloads (for example, web servers, logging, and database backups) do not need expensive on-premises storage arrays, and can instead use AWS Storage Gateway.

The last use case is providing on-premises applications low latency access to data stored in AWS. Organizations have gathered large amounts of unstructured data that they want to store cost-effectively and centrally, so applications in distributed locations can access it for content management, analytics, or processing. However, many are finding it hard to do so with traditional storage arrays.

AWS Storage Gateway use case 1: Backup and archive on-premises applications to the cloud

All the three gateway types that I described earlier are suitable for your backup and archive needs. File Gateway provides NFS or SMB shares that you can use as a storage target to back up your on-premises Oracle or SQL Server databases to AWS. Tape Gateway enables you to archive your data as virtual tapes in AWS. Volume Gateway presents storage LUNs, so if your backup applications must write to a block storage interface, you can do that using this gateway. In this blog, I will cover use of Tape Gateway for backups. I explain how you can use other types of gateways for backups in my re:Invent session highlighting case studies from customers such as Ryanair, Kellogg’s, and TransferWise.

Tape Gateway provides a drop-in replacement for your physical tape infrastructure. It emulates a physical tape library and integrates with your backup apps allowing you to back up and archive data to virtual tapes without changing your workflows. Tape Gateway provides a managed storage service. This means that you don’t need to manage your own Amazon S3 buckets, configure lifecycle policies to transition data, or set IAM rules to access the S3 buckets. AWS Storage Gateway manages all of that and you simply manage the virtual tapes and the gateway. Tape Gateway compresses data in transit and at rest in cloud allowing you to optimize network bandwidth use in addition to lower storage costs. By virtue of storing long-term data in Amazon S3 Glacier or S3 Glacier Deep Archive, Tape Gateway provides a virtual air gapped solution, such that you can only restore this data using a two-step process. Tape Gateway also supports WORM virtual tapes helping protect your data from accidental or malicious deletion. The Tape Retention Lock feature enables you to lock the archived tapes and ensures that nobody can delete them. You can also configure it so that only users with the right permissions can delete them during the configured retention period. This can enable you to meet regulatory compliance needs.

AWS Storage Gateway use case 2: Provide on-premises file shares backed by cloud storage

If you need file storage for applications that must remain on premises, but want Amazon S3’s scalability in the backend, you can deploy File Gateway in your data centers and edge locations. File Gateway presents virtually unlimited storage through file shares using SMB and NFS storage protocols. You control access to File Gateway SMB file shares using your Active Directory domains or by using authenticated guest access. File Gateway caches your most recently used data locally, providing on-premises applications with low latency access to data. You can use File Gateway to automatically generate file-level upload notifications to Amazon CloudWatch. You can send the notifications to an event target, such as AWS Lambda, to automatically trigger downstream workflows required by your business processes. In addition, File Gateway provides audit logging of end user operations on files and folders for SMB file shares. Logging enables you to comply with internal security policies, meet external compliance requirements, and analyze usage trends of your data.

I provide more detail on how Gritstone Oncology uses File Gateway to meet their on-premises file share storage needs in my re:Invent session. I also recommend that you check out our blog on how to choose AWS Storage for on-premises SMB-based file workloads.

AWS Storage Gateway use case 3: Provide on-premises applications low latency access to data stored in the cloud

You may need to provide on-premises applications access to data generated or stored in the cloud using AWS data transfer services such as AWS DataSync and AWS Snowball. To do this, you can use File Gateway. File Gateway provides low latency access to data residing in the cloud to users in any of your office locations. You can think of this as a hub and spoke model. The cloud is the hub for your data, and the gateways deployed in each of your remote offices or edge locations facilitate low latency access to data generated or stored in S3. You may want to do this to share data generated in one factory or a lab with data scientists and analysts working in another location. You may want to provide on-premises users access to an entire dataset or just the results of data processed in the cloud. Regardless of what the case is, File Gateway provides on-premises users access to that data in the cloud using traditional storage protocols such as NFS and SMB. File Gateway’s automated cache refresh capability automatically refreshes the gateway metadata cache. This enables the cache to stay up to date with changes in your S3 buckets without having to manually invoke a cache refresh or manage a process to do so. Learn how Bristol Myers Squibb uses File Gateway for this use case in my session and by reading this blog.

Conclusion

If you are being asked to provide more on-premises storage by business stakeholders, but don’t have storage capacity on premises or sufficient budget, you should evaluate AWS Storage Gateway. Storage Gateway helps you get started with using cloud storage in your data centers, offices, campuses, and edge locations without changing your applications or workflows. Storage Gateway addresses your hybrid cloud storage needs regardless of whether you are just starting to use the cloud. You may have just started the process of migrating to the cloud, or have already moved to AWS, but no matter where you are in that journey, Storage Gateway provides an easy and efficient path to hybrid cloud management.

If you would like to get started with using AWS Storage Gateway today, visit the AWS Storage Gateway console. If you would like to catch up on all the features we launched in 2020, read the AWS Storage Gateway in 2020 – year in review blog. Thanks for reading this blog post as I recapped my AWS Storage Gateway re:Invent 2020-2021 session. Please don’t hesitate to leave comments or questions in the comments section