Category: Announcements

Amazon Redshift now automatically resolves many concurrent writes to Apache Iceberg tables that don’t actually overlap, committing changes that previously failed and had to retry. Learn why these writes conflict, how the engine rebases non-overlapping ones across operation, partition, and file checks, and the throughput gains it delivers.

AWS Glue Data Quality is now available natively on Amazon EMR, bundled in AWS runtime for Apache Spark (emr-spark-8.1.0). Write the same DQDL rulesets you use in AWS Glue directly in your Amazon EMR Spark jobs. This post walks through two patterns: lightweight inline validation and an Apache Iceberg Write-Audit-Publish workflow with managed rulesets.

Amazon EMR 8.1.0 adds multi-catalog support through the RedirectingSessionCatalog, so you can query Iceberg, Delta Lake, Hudi, and Hive tables through a single catalog and join data across AWS accounts without copying it. This post shows how to put these capabilities into practice on Amazon EMR Serverless.

Amazon Redshift now supports Iceberg materialized views. Compute an aggregation once in Amazon Redshift and store the result as a standard Apache Iceberg table in Amazon S3, queryable by Amazon Athena, Apache Spark, Amazon SageMaker, and AWS Glue. This post covers use cases, incremental refresh, and a step-by-step getting-started guide.

Announcing Spark Connect on Amazon EMR on EKS: build, test, and debug Spark applications from VS Code, PyCharm, Jupyter notebooks, Amazon SageMaker Unified Studio, or dbt, while running full-scale Spark operations on your existing Amazon EKS clusters.

Amazon EMR on EC2 now supports Spark Connect, so you can develop and debug PySpark interactively from Amazon SageMaker Unified Studio Data Notebooks or your own IDE while Spark runs on your cluster. This post shows you how to get started from both a Data Notebook and a local IDE.

Connect Microsoft Power BI directly to governed data in Amazon SageMaker Unified Studio using new authentication modes in the Amazon Athena ODBC driver, with no third-party ODBC-JDBC bridge. Part 1 covers IAM Identity Center (IDC)-based domains with both DSN-based and DSN-less connection methods.

Connect Microsoft Power BI directly to governed data in Amazon SageMaker Unified Studio using the Amazon Athena ODBC driver. Part 2 covers IAM-based domains with SageMakerIam authentication, including AWS IAM Identity Center administrator setup, for both DSN-based and DSN-less connection methods.

Amazon Kinesis Data Streams now supports scaling down ingest capacity for on-demand Advantage streams with warm throughput. Learn how the scale-down works, how to monitor stream behavior with Amazon CloudWatch, and best practices for releasing excess capacity after transient traffic bursts.

AWS is announcing the AWS Data Analytics plugin for the new Data agent in ChatGPT Work. Teams can ask questions in natural language, analyze governed data across their Amazon Redshift data warehouse and data lakes, and build shareable dashboards, all from a conversation in ChatGPT Work.