How Photobox optimizes storage costs for over 12 billion photos with Amazon S3 Glacier Instant Retrieval

albelli-Photobox Group is a leading player in the online European photo product and gifting market. We serve a pan-European customer base of over 7 million customers. At Photobox, we are focused on inspiring our customers to easily make beautiful photo products and bring their special moments to life. Whether it’s a birthday, holiday, or any day, we understand these aren’t just photos that we’re entrusted with, they’re treasured moments. That’s why every part of the Photobox experience, from website – to factory – to front door, is designed to delight. Amazon S3 is the main data storage solution that we rely on to host and manage over 12 billion photos totaling 10 petabytes of data. Our developers and Site Reliability Engineering teams ensure that our customers benefit from the highest standards of reliability and performance to store their images and create their very own products in the best possible conditions.

In this blog post, I share our experience using Amazon S3 and S3 Lifecycle policies to optimize our storage costs, while keeping the same level of performance by using Amazon S3 Glacier Instant Retrieval.

Assessing the Amazon S3 Glacier Instant Retrieval storage class according to our use case at Photobox

We know that the vast majority of our data will be stored for many years, and so it’s increasingly important that we optimally use a range of Amazon S3 storage classes to achieve the lowest cost of storage over the lifetime of our data. Prior to the launch of S3 Glacier Instant Retrieval at re:Invent 2021, we used the S3 Standard and S3 Standard-Infrequent Access (S3 Standard-IA) storage classes. Previously, we would first upload all photos to S3 Standard, a storage class designed for frequently accessed data. We then had an S3 Lifecycle policy that would move objects from S3 Standard to S3 Standard-IA after 30 days.

We believed that S3 Glacier Instant Retrieval could be a suitable solution to meet our needs and that implementing it could be relatively simple, since it would only be an additional storage class to configure in our lifecycle. Before using S3 Glacier Instant Retrieval in production, we first had some elements to analyze and manage:

• What ratio of photos are best suited for S3 Glacier Instant Retrieval and how do we estimate the potential savings
• How do we test transition and performance on such a large volume of photos
• How do we safely rollout S3 Glacier Instant Retrieval in production

To estimate the ratio of photos and what savings to expect, we had to consider a major factor in our S3 Glacier Instant Retrieval implementation. We needed to consider the unpredictable nature of content access, which is entirely driven by our customers. In other words, only our customers can decide at what point in time they will access their content to visualize, make, or finalize their creations. We knew S3 Glacier Instant Retrieval was most likely the perfect fit for our needs as it offers the lowest-cost archive storage with milliseconds retrieval for rarely accessed data. But we still had to determine the optimal time to start using this storage class, such as after a certain amount of days our data has been retained. Objects that are archived to S3 Glacier Instant Retrieval have a minimum of 90 days of storage, and objects deleted, overwritten, or transitioned before 90 days incur a pro-rated charge equal to the storage charge for the remaining days. Based on this, we had two options, continue to use S3 Standard-IA after 30 days and add S3 Glacier Instant Retrieval after 90 days, or directly use S3 Glacier Instant Retrieval after 30 days.

Figure 1: Initial design using S3 Standard-Infrequent Access after 30 days

We first considered adding an extra lifecycle from S3 Standard-IA to S3 Glacier Instant Retrieval to only store data older than 90 days. Since we had a large volume of small objects, we knew that there would be a transition cost from one storage class to another depending on the number of objects. We had to analyze the number of photos requested between their 30 and 90 day retention periods in order to determine the most cost effective solution.

AWS Cost Explorer and the AWS Billing console were valuable tools that helped our analysis. We determined that we had relatively limited early deletion (2.5% of our upload on last 30 days). With less than 5% of photos requested after 30 days, even considering the costs of requests and retrieval, it was advantageous for us to use S3 Glacier Instant Retrieval right away. We therefore decided to test a transition directly from S3 Standard to S3 Glacier Instant Retrieval.

Figure 2: Desired design using S3 Glacier Instant Retrieval after 30 days

Setting up performance comparison

To validate the performance of data access on S3 Glacier Instant Retrieval, we needed to select a sample of objects in our bucket. We used Amazon S3 Inventory to establish a list of files already in S3 Standard-IA.

On the bucket that contains the data, we completed the following steps to create an inventory, tag a high number of objects, and create a new S3 Lifecycle policy.

1. Access Amazon S3 Inventory
2. Configure inventory
3. Access Amazon S3 Batch Operations
4. Run the Job
5. Create and apply the new lifecycle policy

I hope that by sharing these detailed steps with you, you’ll benefit from our experiences and outcomes allowing you to start using Amazon S3 Glacier Instant Retrieval to optimize your storage costs to help transform your business. With that, let’s dive right in.

1. Access to Amazon S3 Inventory

The first step is to go to our bucket where the data to be listed is located. Log into the AWS Management Console using your account information. From the AWS console services search bar, enter ‘S3’. Under the services search results section, select S3.

Then, in the Amazon S3 menu on the left, choose Buckets, and then select your bucket in the list.

Next, navigate to the Management tab.

In the Inventory configuration section, select Create Inventory configuration.

2. Configure your inventory

In this next step, I will detail out how to configure your inventory. First, define an Inventory Configuration name. Choose if you want to apply a prefix, which object version to include, and select a destination bucket for the output file.

Next, in the Server-side encryption section, select if you want to enable any encryption, along with which fields to include in your inventory based on your needs. Then select the Create button.

This gives us a list in CSV format, with the path and the storage class in which the file is located.

We selected about 10,000 files in S3 Standard-IA from the inventory and then ran GET/PUT performance tests on them. Once we had the results, we tagged these 10,000 files to move them into S3 Glacier Instant Retrieval.

To manage tags on a high number of files, we use Amazon S3 Batch Operations. S3 Batch Operations allows us to add, remove, and modify tags on all desired objects.

3. Access Amazon S3 Batch Operations

Now we’ll dig into acessing Amazon S3 Batch Operations. While in the S3 console, in the menu on the left, select Batch Operations. Then select the Create job button.

In the Choose Region and manifest section, we choose the same Region as our bucket containing the files we want to tag. For our example, the selected CSV as the Manifest format, and we specify the bucket where the inventory is stored in the Manifest object section. Then, we select the Next button.

In the Choose operation section we select the Replace all object tags option.

Then, in the Replace all object tags section, we add a Key:Value pair to apply on all desired objects. Then, select the Next button.

In the Configure additional options section, we can configure the description name, priority, and reporting. We leave default values in Description and Priority and we select the destination bucket where we will send our completion report.

Then, in the Permissions section, we use a pre-defined IAM role. If you don’t have a pre-defined IAM role, the View IAM role policy template and IAM trust policy section shows you the necessary IAM rights to add. Then, select the Next button.

The final step of the job’s creation is to review its configuration, and then select the Create Job button.

4. Run the Job

Next, I dicsuss how to go about runing the job. In the Batch Operations section, you will see your job with a status set to Awaiting your confirmation to run. This means the input inventory is readable and ready to be launched. Next, select the job via the radio button, and the select the Run job button.

Depending on the number of objects you need to process, you may need to wait some time for the job to complete. Once the job completes, you will see a Completed status in the Status column.

Tags are now applied on our objects. We can now create a new lifecycle to move our sample to the desired storage class.

5. Create and apply the new lifecycle policy

Finally, I’ll walk through how to create and apply your new lifecycle policy to move our designated objects from S3 Standard-IA to S3 Glacier Instant Retrieval. In the Management tab we select the bucket and then we select the Create Lifecycle rule button.

Then, in the Create lifecycle rule section, we choose a name for our lifecycle, and select the Limit the scope of this rule using one or more filters radio button. We make this selection because we want to move only the tagged objects for our test. Next, in the Filter type section, we specify the tag Key:Value pair to filter on.

Finally, in the Lifecycle rule actions section, as we want to move only current versions of the files, we select the Move current versions of objects between storage classes option. Then, in the Transition current versions of objects between storage classes section, we set the value for transition as 30 days. We make these selections because our use case consists of comparing S3 Standard-IA to S3 Glacier Instant Retrieval. The last step is to validate your selections by selecting the Create rule button. After creation we can access our lifecycle and verify the content.

In the Lifecycle configuration section, we can click on the rule name to display the full configuration.

Alternatively, the lifecycle policy can be created using the AWS Command Line Interface (AWS CLI) with aws s3api. To setup the rule through the AWS CLI, you must type the following, where $bucket_name is to be replaced by your actual bucket.

aws s3api put-bucket-lifecycle-configuration --bucket $bucket_name --lifecycle-configuration file://lifecycle.json

{
  "Rules": [
    {
      "ID": "s3stdia-to-s3gir",
      "Status": "Enabled",
      "Filter": {
        "Tag": {
          "Key": "Gir",
          "Value": "yes"
        }
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "GLACIER_IR"
        }
      ]
    }
  ]
}

You can now verify that the rule has been correctly created, by using the following command:

aws s3api get-bucket-lifecycle --bucket $bucket_name 

{
    "Rules": [
        {
            "ID": "s3stdia-to-s3gir",
            "Status": "Enabled",
            "Transition": {
                "Days": 30,
                "StorageClass": "GLACIER_IR"
            }
        }
    ]
}

Performance comparison

Now that the objects have been moved, we want to understand the performance difference between S3 Standard-IA and S3 Glacier Instant Retrieval. To measure this, we ran the same GET/PUT latency test on S3 Glacier Instant Retrieval and compared it with our previous results on S3 Standard-IA.

We compared 10,000 objects, with an average object size of 1.3 MB.

This test showed us there was no difference in GET/PUT latency performance between the two storage classes based on the same testing machine. For our example, the test machine was on Amazon EC2 in a VPC with a private endpoint to S3. Given the performance parity and our estimated savings, we decided to use S3 Glacier Instant Retrieval. To do so, we had to change the lifecycle, which was extremely easy. We simply deactivated the old rule and created a new one. Of course, we could also have modified the current rule, but we wanted to activate the new rule at a specific time. So, we repeated the Create and apply the new lifecycle policy step from the above walkthrough, but this time on the production bucket. However, this time we didn’t filter on tags; we apply it on every object.

As a reminder, the lifecycle policy can be created directly with the aws s3api with the command below:

($bucket_name still to be replaced by your actual bucket)

The difference here is that there are no filters specified on tags, so we apply it on all objects and all versions in the bucket.

aws s3api put-bucket-lifecycle-configuration –bucket $bucket_name –lifecycle-configuration file://lifecycle.json

aws s3api put-bucket-lifecycle-configuration --bucket $bucket_name --lifecycle-configuration file://lifecycle.json


{
    "Rules": [
        {
          "ID": "After30DaysChangeStorageClassToS3GIR",
          "Status": "Enabled",
          "Filter": {
            "Prefix": ""
          },
          "Transitions": [
            {
              "Days": 30,
              "StorageClass": "GLACIER_IR"
            }
          ],
          "NoncurrentVersionTransitions": [
        {
          "NoncurrentDays": 30,
          "StorageClass": "GLACIER_IR"
        }
      ]
        }
    ]
}

It’s always worthwhile to validate that your rule has been set up correctly and returns the desired configuration. To validate, enter the command below:

aws s3api get-bucket-lifecycle --bucket $bucket_name 

{
    "Rules": [
        {
            "ID": "After30DaysChangeStorageClassToS3GIR",
            "Status": "Enabled",
            "Transition": {
                "Days": 30,
                "StorageClass": "GLACIER_IR"
            },
            "NoncurrentVersionTransition": {
                "NoncurrentDays": 30,
                "StorageClass": "GLACIER_IR"
            }
        }
    ]
}

This policy takes care of all transitions, both for future stored objects in S3 Standard and those already in S3 Standard-IA. However, we weren’t sure how long it would take to move a volume of data as large as 10 PB. We soon realized it took only about 24 hours to migrate all data from one class to another, without any further action or operational overhead on our end.

Figure 3: Volume of data in each storage class – objects being moved in 1 day between March 21 & 22

Cleaning up

We recommend you clean up the resources you created in this walk through. It is a best practice to delete resources that you are no longer using so that you do not incur unintended charges. For instance, you can now delete S3 Inventory configuration and the existing S3 Iventory reports to stop incurring costs, if they are needed for other tasks. Also note, S3 Standard-IA has a 30-day minimum storage duration, and S3 Glacier instant Retrieval has a 90-day minimum storage duration.

Conclusion

In this blog, we shared how we at Photobox evaluated and implemented S3 Glacier Instant Retrieval. Although the storage price of S3 Glacier Instant Retrieval is lower than S3 Standard-IA, the cost to access and retrieve data is slightly higher. This makes S3 Glacier Instant Retrieval ideal for long term data that is accessed roughly once per quarter. Something else to consider when implementing a storage class is the transition cost (billed per 1,000 objects), which could become significant when you manage a large number of objects, even if this is usually compensated by a lower storage cost.

The solution that we discussed in this blog represents an additional, particularly interesting, possibility for large volumes of data rarely accessed by customers. By using Amazon S3 Glacier Instant Retrieval, we realized a massive 50% reduction on our AWS storage bill without any compromise on access times, availability, or any other heavy additional processes.

We were able to achieve this storage class migration with minimal effort and in full confidence.

The ready-to-use tools provided by AWS allowed us to focus on planning and testing instead of complex operational tasks. At any time we were able to follow the progress via tools like AWS Cost Explorer, Amazon CloudWatch, and S3 Storage Lens, allowing us to safely achieve substantial savings for our business while continuing to provide the best possible service to our customers.

We also had excellent support from the AWS team to answer our questions and help our team along the implementation.

Thank you for reading this blog, any questions or suggestions are welcome, please leave them in the comment section below.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.