Add multiple storage buckets to your app using AWS Amplify – NEW

You can now configure and manage multiple buckets using AWS Amplify for your storage needs! Developers can now leverage Amplify storage to organize and manage their content across single or multiple storage buckets while applying granular access rules against individual paths within each bucket.

Earlier this year, we announced the new and improved Amplify storage experience, which integrates with Amazon Simple Storage Service (Amazon S3) and provides an intuitive approach to managing cloud-based file storage (link to blog post). Building on top of this, we are excited to announce the ability to configure and connect to multiple storage buckets using the backend configuration and the JavaScript storage APIs.

One common use case for having multiple storage buckets is to separate application data that requires optimizations like transfer acceleration from long-term back-office data. For example, you might have one bucket optimized for fast uploads and downloads of user-generated content (e.g., photos, videos) using Amazon S3 Transfer Acceleration, while another bucket is configured for storing long-term archival data with less frequent access patterns, such as reports, logs, or backups.

With Amplify storage, you can call APIs like upload, download, list, and more against:

Buckets configured with an Amplify backend: These are buckets defined and managed through your Amplify project’s backend configuration. You can specify settings like bucket name and access rules for these buckets within the amplify/storage/resource.ts file.
Existing S3 buckets: You can also interact with Amazon S3 buckets that were created directly in Amazon S3 console.

Connect app to storage configured in Amplify backend

1. Initialize a new Amplify project

In this example, we will create a new Next.js project:

npx create-next-app@latest multi-bucket-app

Follow the set up instructions and make sure to choose “Yes” when asked “Would you like to use TypeScript?”.

You have an app named “multi-bucket-app” in your current folder. Navigate to this new app and initialize AWS Amplify:

>cd multi-bucket-app
>npm create amplify@latest

This command will create an “amplify” folder in your project with the following structure:

Docs structure

2. Define storage buckets in the Amplify backend

First, we will create a bucket to store user-generated content like user profile photos. We want authenticated users to be able to read and write to the assets/photos/* path, while guest users will only be able to read from this path.

Create a new resource.ts file under amplify/storage/. In this file, you can define your first storage bucket configuration.

import { defineStorage } from '@aws-amplify/backend';

export const userDataBucket = defineStorage({
  name: 'user-data-bucket'
  access: (allow) => ({
    'assets/photos/*': [
      allow.guest.to(['read']),
      allow.authenticated.to(['read','write']),
    ]
  })
});

Next, add the storage bucket configuration to your Amplify backend definition in the amplify/backend.ts file.

import { defineBackend } from '@aws-amplify/backend';
import { auth } from './auth/resource';
import { userDataBucket } from './storage/resource';

defineBackend({
  auth,
  userDataBucket
});

In this use case, the app makes frequent calls to the user-generated content bucket to display relevant photos to the end users. To ensure these photos are delivered to the client quickly, we will enable transfer acceleration on this bucket.

Note: Amazon S3 Transfer Acceleration can speed up content transfers to and from Amazon S3 by as much as 50-500% for long-distance transfer of larger objects. Refer to S3 Transfer Acceleration to learn more about this.

import { defineBackend } from '@aws-amplify/backend';
import { auth } from './auth/resource';
import { userDataBucket } from './storage/resource';

defineBackend({
  auth,
  userDataBucket
});

const { cfnBucket } = backend.userDataBucket.resources.cfnResources;

cfnBucket.accelerateConfiguration = {
  accelerationStatus: "Enabled"
}

You can run npx ampx sandbox command, which will spin up a local sandbox environment for your Amplify backend so you can test your app locally.

Similarly, to define more storage buckets, you can use the same defineStorage method within the amplify/storage/resource.ts file.

import { defineStorage } from '@aws-amplify/backend';

export const userDataBucket = defineStorage({
  name: 'user-data-bucket',
  isDefault: true,
  access: (allow) => ({
    'assets/photos/*': [
      allow.guest.to(['read']),
      allow.authenticated.to(['read','write']),
    ]
  })
});

export const reportingDataBucket = defineStorage({
   name: 'reporting-data-bucket',
   access: (allow) => ({
    'reportingData/logs/*': [
      allow.groups(['admins']).to(['read','write']),
    ],
    'reportingData/performance/*': [
      allow.groups(['admins']).to(['read','write']),
    ]
  })
});

When working with more than one storage bucket, you need to assign one bucket as the default. To do this, set the isDefault property to true in one of the storage bucket definitions within the amplify/storage/resource.ts file. This default bucket will be used automatically by the Amplify Storage APIs when no specific bucket is provided.

Import this new resource to your your backend configuration in amplify/backend.ts.

import { defineBackend } from '@aws-amplify/backend';
import { auth } from './auth/resource';
import { userDataBucket, reportingDataBucket } from './storage/resource';

defineBackend({
  auth,
  userDataBucket,
  reportingDataBucket
});

const { cfnBucket } = backend.userDataBucket.resources.cfnResources;

cfnBucket.accelerateConfiguration = {
  accelerationStatus: "Enabled"
}

4. Upload files to specified bucket using Amplify storage library

To upload to your storage buckets, you can continue using the existing Amplify storage APIs. Pass in the bucket name defined in the amplify/storage/resource.ts file. If no bucket is provided, the file will be uploaded to the default bucket configured in the same resource.ts file.

import { uploadData } from 'aws-amplify/storage';

//Set the file variable to the file you want to upload
const file: File

const { result } = await uploadData({
  path: 'reportingData/logs/08222024.txt',
  data: file,
  options: {
    bucket: 'reporting-data-bucket'
  }
});

You can use the same bucket option to specify the bucket for other APIs like downloadData, list, copy, and more. Refer to the Amplify storage documentation for more Amplify storage APIs.

Connect app to existing S3 buckets

Alternatively, you can upload files directly to an existing Amazon S3 bucket by specifying the bucket name and region. This approach allows you to use custom S3 buckets with the Amplify storage library that are not defined within the Amplify backend configuration.

To upload files to an existing S3 bucket, pass the actual bucket name (as seen on the Amazon S3 console) and the corresponding AWS region as options to the Amplify storage APIs. Refer to the documentation on using custom Amazon S3 buckets with Amplify storage APIs.

S3 console showing bucket name

import { uploadData } from 'aws-amplify/storage';

//Set the file variable to the file you want to upload
const file: File

const { result } = await uploadData({
    path: 'reportingData/logs/08222024.txt',
    data: file,
    options: {
        bucket:{
            bucketName: 'bucket-name-from-s3-console',
            region: 'us-east-2'
        }
    }
});

Conclusion

You can now better organize and separate your app’s content by configuring and managing multiple storage buckets with Amplify. You can refer to the Amplify storage documentation for more information on how you can configure your storage buckets.

Amplify is an open source project and we are always looking for feedback from the community. We would love to hear from you on any of our channels – you can engage in discussion on our Discord server or open feature requests or bugs in our respective GitHub projects.

Front-End Web & Mobile