Deprecation of Lake Formation’s Governed Tables Feature

After careful consideration, we have made the decision to end support for Governed Tables, effective December 31, 2024, to focus on open source transactional table formats such as Apache Iceberg, Apache Hudi, and Linux Foundation Delta Lake. This decision stems from customer preference for these open source solutions, which offer ACID-compliant transactions, compaction, time travel, and other features previously provided by Governed Tables. Amazon Web Services (AWS) has enhanced our support for these formats across various analytics services, including Amazon Athena, Amazon EMR (Elastic MapReduce), AWS Glue, and Amazon Redshift, with features that include automatic compaction support for Apache Iceberg, retention and snapshot expiration and orphan file deletion for Apache Iceberg, enhancements to AWS Glue Data Catalog CreateTable API to create Apache Iceberg tables, and AWS Glue Crawlers schema detection support across Apache Iceberg, Apache Hudi, and Linux Foundation Delta Lake. Customers can now use these open source formats to achieve ACID-compliant transactions with Amazon Simple Storage Service (Amazon S3) backed data, benefiting from their rich features and wide compatibility.

After December 31, 2024, customers will no longer be able to create Governed Tables transactions (lakeformation:StartTransaction), write to Governed Tables (lakeformation:UpdateTableObjects), or query your Governed Tables using Amazon Athena. Customers will still be able to access their table state information by calling the lakeformation:GetTableObjects and transaction information by calling lakeformation:ListTransactions until February 17, 2025. After February 17, 2025, all Governed Table APIs will start to fail. Governed Tables metadata will continue to exist within the AWS Glue Data Catalog, and the Governed Tables data will remain in your S3 buckets. No other table types will be affected by this change, including Hive (Apache Parquet, CSV, ORC, and so on), Iceberg, Hudi, and Delta Lake tables.

Migrating your Governed Tables

Customers can migrate their tables from Governed Tables to one of the open source formats by copying their governed table data directly to Apache Iceberg using Amazon Athena. To migrate data to Iceberg, you can use the Amazon Athena CREATE TABLE AS (CTAS) statement, as shown in the following code example.

CREATE TABLE my_iceberg_table WITH ( 
    table_type = 'ICEBERG',
    is_external = false,
    location = 's3://mybucket/myicebergdata/'
) AS 
SELECT * FROM my_governed_table

You can specify additional table-level properties, which are listed in the Amazon Athena User Guide. If you specify partitions or buckets as part of the Apache Iceberg table definition, then you may run into the 100 partition per bucket limitation. In this case, refer to Use CTAS and INSERT INTO to work around the 100 partition limit.

If you require any assistance migrating your tables, or have any questions, reach out to us at governed-tables-support@amazon.com.

About the author

Mert Hocanin is a Principal Big Data Architect with AWS Lake Formation.