Category: Analytics

AWS Glue Data Quality is now available natively on Amazon EMR, bundled in AWS runtime for Apache Spark (emr-spark-8.1.0). Write the same DQDL rulesets you use in AWS Glue directly in your Amazon EMR Spark jobs. This post walks through two patterns: lightweight inline validation and an Apache Iceberg Write-Audit-Publish workflow with managed rulesets.

Learn how to build an enterprise-scale cross-account streaming pipeline with Amazon Kinesis Data Streams using the Kinesis Producer Library (KPL) and Kinesis Client Library (KCL) across three separate AWS accounts, with resource-based policies, cross-account IAM, and a shared customer managed AWS KMS key for maximum security isolation.

Troubleshooting a multi-service, MWAA-orchestrated ETL pipeline often means hunting across scattered Amazon CloudWatch log groups. This post shows how to centralize your ETL logs in Amazon OpenSearch Service and query them in plain language through an MCP server on Amazon Bedrock AgentCore, so you find root causes faster.

Learn how to pair DuckDB with AWS Glue 6.0 to run SQL-centric ETL on a single worker, reading Parquet from Amazon S3 and writing Apache Iceberg tables to Amazon S3 Tables. This post walks through a complete, runnable example and compares measured cost and runtime against an equivalent Apache Spark job on the same Glue runtime.

Amazon EMR 8.1.0 adds multi-catalog support through the RedirectingSessionCatalog, so you can query Iceberg, Delta Lake, Hudi, and Hive tables through a single catalog and join data across AWS accounts without copying it. This post shows how to put these capabilities into practice on Amazon EMR Serverless.

The KIP-848 consumer protocol in Apache Kafka 4.0 redesigns consumer group rebalancing to eliminate stop-the-world pauses. This post explains how the consumer protocol works on Amazon MSK, how to enable it on MSK Standard and Express brokers, and how to diagnose and resolve slow rebalancing issues.

Announcing Spark Connect on Amazon EMR on EKS: build, test, and debug Spark applications from VS Code, PyCharm, Jupyter notebooks, Amazon SageMaker Unified Studio, or dbt, while running full-scale Spark operations on your existing Amazon EKS clusters.

Amazon SageMaker Unified Studio now supports custom permissions boundaries for the IAM roles its Tooling blueprint creates. Learn how to create a permissions boundary that restricts AI agent capabilities, configure it on the Tooling blueprint with the AWS CLI, and validate that every provisioned role carries the boundary.

Learn how Delivery Hero rebuilt its ad measurement pipeline from hourly batch processing to real time on Amazon Managed Service for Apache Flink, cutting the gap between an event and its recording from 61 minutes to 1.2 seconds and reducing monthly operational costs by about 57%.

Configuring VPC networking per project across a SageMaker Unified Studio domain creates inconsistent, hard-to-audit networks. This post shows administrators how to configure domain-level VPC networking once, so every new project automatically inherits consistent, private network isolation, then update existing projects and validate connectivity.