Category: AWS Step Functions

Organizations that store sensitive data need to answer a simple question: where did this data come from, and what happened to it? In healthcare, this isn’t optional. Regulations like HIPAA, GDPR, and SOX require organizations to produce this information on demand. When an auditor asks, for example, “Show me every transformation that touched patient record […]

As data volumes grow, organizations must move large datasets between storage locations to meet compliance requirements, optimize performance, and help meet data sovereignty requirements. However, migrating petabytes of data presents significant challenges: lengthy transfer times, complex coordination of parallel operations, data integrity verification, and substantial engineering overhead. Automation reduces resource consumption and operational complexity during […]

Many organizations operate thousands of Amazon S3 buckets in a single AWS Region, each with its own configuration accumulated over the years. Some were created manually in the AWS Management Console and others by scripts that are no longer actively maintained, provisioned by different business units with their own policies, lifecycle rules, encryption, and tags. […]

Every day, AWS customers process millions of compressed files in Amazon S3, from small ZIP archives to multi-gigabyte datasets. While decompressing a single file is straightforward, processing thousands of files efficiently requires complex orchestration, error handling, and infrastructure management. Consider this scenario: Your organization receives over 10,000 compressed files daily from partners, ranging from 5 […]

Automated testing in a pre-production environment is crucial for verifying the reliability and stability of software releases in any organization. However, for many applications, writing and executing these tests necessitates the use of data from production system. This production data is valuable for testing and development because it represents real-world scenarios, usage patterns, and edge […]

Editor’s note: AWS is not responsible for UCI’s public GitHub repo linked in this post, which has been provided so that interested parties can explore the solution described in this post in more detail. The University of California, Irvine (UCI) is a public land-grant research university with troves of research data stored on servers in […]

Data caching is a critical strategy for optimizing application performance in today’s data-intensive environments. By storing frequently accessed information in high-speed storage locations, organizations can dramatically reduce access times, optimize the use of compute resources, and improve overall system responsiveness. Effective caching strategies become particularly essential for workloads that require consistent low latency, such as […]

Customer Story | Amazon S3 Tables – Pendulum migrated from self-managed Apache Iceberg tables to fully managed Amazon S3 Tables and saw faster processing, fewer weekly maintenance hours, and reduced costs for their analytics platform. Pendulum is an AI-powered analytics platform that aggregates and analyzes real-time data from social media, news, and podcasts. Designed to […]

FICO powers decisions that help people and businesses around the world prosper. Using FICO solutions, businesses in more than 80 countries do everything from protecting four billion payment cards from fraud, to improving financial inclusion, and increasing supply chain resiliency. As a global leader in credit scoring and analytics, FICO processes massive volumes of sensitive […]

Update (5/29/2025): On May 29, 2025, AWS DataSync launched Enhanced mode support for cross-cloud transfers. Enhanced mode simplifies data transfers between AWS and other clouds by removing the need for a DataSync agent. It also provides higher performance and scalability when compared to Basic mode. For more details, see the What’s New announcement or review the documentation for guidance […]