Listing Thumbnail

    Apache Iceberg Connector for AWS Glue

     Info
    Easily access Iceberg tables and operate DDLs, reads/writes, time-travels, streaming writes on the Iceberg tables.
    Listing Thumbnail

    Apache Iceberg Connector for AWS Glue

     Info

    Overview

    The Iceberg connector enables to access Iceberg tables on the Glue Data Catalog from your Glue ETL jobs. You can do operations supported by Apache Iceberg such as DDLs, read/write-data, time-travels and streaming writes for the Iceberg tables with Glue jobs.

    Highlights

    • Create Iceberg tables on the Glue Data Catalog via DDLs and DataFrame.
    • Operate queries, time-travels and streaming writes for Iceberg tables.

    Details

    Delivery method

    Delivery option
    Glue 4.0
    Glue 3.0
    Glue 1.0/2.0

    Latest version

    Operating system
    Linux

    Features and programs

    Financing for AWS Marketplace purchases

    AWS Marketplace now accepts line of credit payments through the PNC Vendor Finance program. This program is available to select AWS customers in the US, excluding NV, NC, ND, TN, & VT.
    Financing for AWS Marketplace purchases

    Pricing

    Apache Iceberg Connector for AWS Glue

     Info
    This product is free. Subscriptions have no end date and can be canceled anytime.

    Vendor refund policy

    No Refunds

    Legal

    Vendor terms and conditions

    Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA) .

    Content disclaimer

    Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.

    Usage information

     Info

    Delivery details

    Glue 4.0

    Supported services: Learn more 
    • Amazon ECS
    • Amazon EKS
    Container image

    Containers are lightweight, portable execution environments that wrap server application software in a filesystem that includes everything it needs to run. Container applications run on supported container runtimes and orchestration services, such as Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS). Both eliminate the need for you to install and operate your own container orchestration software by managing and scheduling containers on a scalable cluster of virtual machines.

    Version release notes

    New version of the container image

    Additional details

    Usage instructions

    Please subscribe to the product from AWS Marketplace and Activate the Glue connector from AWS Glue Studio .

    The Iceberg connector enables you to access iceberg tables from your Glue jobs. In particular, you can operate DDLs, read/write data, time-travels and streaming writes for the Iceberg sources. Further details about the Apache Iceberg, please see https://iceberg.apache.org/ .

    Using the Apache Iceberg Connector

    Here's the steps to set up the Iceberg connector. For more details about using Iceberg on AWS, please refer to https://iceberg.apache.org/aws/ .

    Required Spark configuration

    Before using this connector, you need to set the following Spark configuration for communication between this connector and the Glue Data Catalog.

    You can set the configuration on the job parameter in your Glue job or on the scripts in your Glue job. Please note that the configuration related to spark.sql.extensions is optional, however that's recommended for accessing full operations of Iceberg.

    Set on the job parameter

    • Key: --conf
    • Value: spark.sql.catalog.<catalog_name>=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.<catalog_name>.warehouse=<S3_PATH> --conf spark.sql.catalog.<catalog_name>.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.<catalog_name>.io-impl=org.apache.iceberg.aws.s3.S3FileIO --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

    Set on the Glue job script:

    from pyspark.sql import SparkSession from pyspark.context import SparkContext

    spark = SparkSession.builder \ .config(f"spark.sql.catalog.<catalog>", "org.apache.iceberg.spark.SparkCatalog") \ .config(f"spark.sql.catalog.<catalog>.warehouse", "s3://bucket/path/") \ .config(f"spark.sql.catalog.<catalog>.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \ .config(f"spark.sql.catalog.<catalog>.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \ .config(f"spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \ .getOrCreate()

    Connector options

    You can pass the following options to the connector.

    • path - An Iceberg table name such as <catalog_name>.<database>.<table>.
    • as-timestamp-of or snapshotid (optional, set only for read) - You can operate "Time travel" queries by setting this parameter. Please see the document  about the time-travel.

    NOTE: Glue natively supports Iceberg. Please refer to https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html .

    Support

    Vendor support

    Please allow 24 hours

    AWS infrastructure support

    AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.

    Similar products

    Customer reviews

    Ratings and reviews

     Info
    0 ratings
    5 star
    4 star
    3 star
    2 star
    1 star
    0%
    0%
    0%
    0%
    0%
    0 AWS reviews
    No customer reviews yet
    Be the first to write a review for this product.