Event-Driven Glue Catalog Registration for Snowflake-Managed Iceberg Tables

Eliminating the random suffix problem without a Glue crawler, in near-real-time, for pennies a day.

Our daily stand-up meeting commenced, and the customer jumped right in with, “I added the Iceberg tables you created yesterday to the Glue Catalog, and I wanted to know if you can create them without that random suffix on the end of the table name.”

The customer explained that instead of clean names, it resulted in names like customers_f0xgvx1a and orders_djctgenc. The names were cumbersome and frustrating to the initial users who were writing SQL to explore the data.

Snowflake introduced new write paths for Snowflake-managed Iceberg tables in the BCR bundle release 2025_01. This feature, which cannot be disabled, appends a random 8-character suffix to the base location so Snowflake can guarantee uniqueness when tables are dropped and recreated. The Glue crawler was simply picking up this name from the metadata and writing it to the catalog.

In my mind, I thought, “No problem.” There must be a setting to turn this off in Glue, or a rule we could apply to rename the table without the suffix.

Unfortunately, these features did not exist in Glue. With no approval to adopt a new catalog service and no appetite to re-architect access patterns that already worked for every other dataset, we had to get creative. After a few false starts and dead-end workarounds, I landed on directly registering the tables into the Glue catalog.

Our pipeline was working well. With Snowflake managing our Iceberg tables and metadata, we were 95% of the way to a successful delivery. We just needed to capture that last 5%, and event-driven registration was the answer.

Before You Build This: Consider Horizon Catalog

Enterprises are often limited in the tools and APIs they can use and are not just able to use new tools without approval from a variety of folks within their organization. If those options are not possible, then this article will help you.

The custom solution described in this article is meant for 1) organizations that have not adopted Horizon Catalog or another REST API Compliant Catalog. 2) You work in a regulated environment where adding new catalog services requires lengthy approval. 3) Your downstream consumers are strictly locked to AWS Glue for Lake Formation permissions or have existing Athena workloads, and/or 4) You need something working today without waiting on high-level platform decisions.

The Solution: Event-Driven Registration Without a Crawler

A crawler that scans table metadata may be a natural alternative, but this solution has a few benefits over using the crawler. One being that we register the tables directly in the Glue Data Catalog at the moment Snowflake writes the metadata file to S3. SQS sits between S3 and Lambda because it decouples the event source from processing, enables dead-letter queue support for failed invocations, and avoids hitting Lambda’s direct S3 concurrency limits when multiple tables write simultaneously. Key components of the architecture are below:

Key principles driving this architecture:

• No crawler. We use direct API registration in near-real-time.
• Clean names. We strictly control the table name that is registered in the Glue Catalog.
• Full schema extraction. We pull columns straight from the Iceberg metadata.json rather than hardcoding them.
• Automatic schema evolution. New columns appear in Glue on the very next write.
• Idempotency. Multiple events for the same table simply update the pointer.

Implementation

The Lambda function handles the core logic. It receives S3 events via SQS, parses the path to grab the clean table name, reads the schema from the Iceberg metadata, and registers it in Glue.

Path Parsing: The random suffix is exactly 8 characters of mixed-case alphanumerics.

PATH_PATTERN = re.compile(
    r'^(?P<prefix>.+)/(?P<base_loc>[^/]+)\.(?P<rand>[A-Za-z0-9]{8})'
    r'/metadata/(?P<file>.+\.metadata\.json)$')

For a path like iceberg/lseg_qa_iceberg/CUSTOMERS.f0XGVx1a/metadata/00042-uuid.metadata.json , the base location becomes CUSTOMERS. We keep that clean name and discard the random f0XGVx1a string.

def extract_columns_from_metadata(bucket, metadata_key):
    obj = s3.get_object(Bucket=bucket, Key=metadata_key)
    metadata = json.loads(obj['Body'].read())
    # Iceberg v2: find current schema by ID
    current_schema_id = metadata.get('current-schema-id', 0)
    schemas = metadata.get('schemas', [])
    schema = next(
        (s for s in schemas if s.get('schema-id') == current_schema_id),
        schemas[-1] if schemas else None
    )
    columns = []
    for field in schema.get('fields', []):
        columns.append({
            'Name': field['name'].lower(),
            'Type': iceberg_type_to_hive(field['type']),
        })
    return columns

The Iceberg metadata.json contains the complete table schema which we read directly from S3. We parse the file and then map the data types to Hive equivalents via iceberg_type_to_hive. Note that this function must handle primitive types (string, long, double, boolean, date, timestamptz) as well as complex types — structs, lists, and maps — which require recursive translation. Decimal types must preserve precision and scale (e.g., decimal(18,2)). Athena v3, Trino, and modern Spark all accept these Hive-style type strings. A reference implementation of iceberg_type_to_hive is available in the source code linked at the end of this article. Since these files are only a few kilobytes, we completely avoid scanning massive data files or manually maintaining column lists anywhere.

Glue Registration (with retry):

def register_table(table_name, table_location, metadata_location, columns):
    table_input = {
        'Name': table_name,
        'TableType': 'EXTERNAL_TABLE',
        'Parameters': {
            'table_type': 'ICEBERG',
            'metadata_location': metadata_location,
            'classification': 'iceberg',
        },
        'StorageDescriptor': {
            'Columns': columns,
            'Location': table_location,
            'InputFormat': 'org.apache.iceberg.mr.hive.HiveIcebergInputFormat',
            'OutputFormat': 'org.apache.iceberg.mr.hive.HiveIcebergOutputFormat',
            'SerdeInfo': {
                'SerializationLibrary': 'org.apache.iceberg.mr.hive.HiveIcebergSerDe'
            },
        }
    }
    max_retries = 3
    for attempt in range(max_retries):
        try:
            try:
                glue.get_table(DatabaseName=GLUE_DATABASE, Name=table_name)
                glue.update_table(DatabaseName=GLUE_DATABASE, TableInput=table_input)
            except glue.exceptions.EntityNotFoundException:
                glue.create_table(DatabaseName=GLUE_DATABASE, TableInput=table_input)
            return
        except ClientError as e:
            if e.response['Error']['Code'] == 'ConcurrentModificationException' \
            and attempt < max_retries - 1:
                time.sleep((attempt + 1) * 0.5)
            else:
                raise

Passing the metadata_location parameter makes this work. Query engines like Athena v3, Spark, and Trino read this field from the Glue table entry and go straight to the Iceberg metadata.json for schema and snapshot data. We populate the Columns list strictly for backward compatibility with older tools like Hive 3.x, older Presto configurations, or any tool that reads schema from Glue’s StorageDescriptor rather than from the Iceberg metadata directly. Athena v3, Spark with the Iceberg connector, and Trino all ignore the StorageDescriptor columns in favor of the native Iceberg schema. Note: this example uses the legacy Hive format classes (HiveIcebergInputFormat / HiveIcebergOutputFormat / HiveIcebergSerDe) for maximum compatibility with older tools. If you don’t need Hive 3.x or legacy Presto compatibility, AWS Glue now supports registering Iceberg tables natively via the OpenTableFormatInput / IcebergInput API, which skips the Hive SerDe entirely and is the more modern approach going forward.

S3 Event Configuration You must filter on the .metadata.json suffix. This is a critical detail. Snowflake writes Parquet data files, Avro manifests, and metadata.json files during every single refresh. Only the metadata.json represents a completed commit. Without the filter, your Lambda fires on every single file write and swamps your logs with warnings for non-matching paths.

Event name: iceberg-metadata-notify Prefix: iceberg/my_schema/ Suffix: .metadata.json Event type: s3:ObjectCreated:* Destination: SQS queue

IAM Policy Provide minimal permissions for the Lambda role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["glue:GetTable", "glue:CreateTable", "glue:UpdateTable"],
            "Resource": [
                "arn:aws:glue:REGION:ACCOUNT:catalog",
                "arn:aws:glue:REGION:ACCOUNT:database/my_db",
                "arn:aws:glue:REGION:ACCOUNT:table/my_db/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": ["s3:GetObject"],
            "Resource": ["arn:aws:s3:::my-bucket/*"]
        },
        {
            "Effect": "Allow",
            "Action": ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"],
            "Resource": ["arn:aws:sqs:REGION:ACCOUNT:my-queue"]
        }
    ]
}

What Was Verified

Certainly not an exhaustive list but these are the following validations and expected results that I ran to ensure the direct registration succeeds.

Cost and Performance

The financial cost of running this Lambda is negligible. You pay for one invocation per DML commit per table that reads just a few kilobytes of JSON. An example pipeline refreshing 20 tables every 5 minutes would trigger roughly 5,760 invocations a day and that costs about $0.01–$0.02 per day across Lambda, SQS, and S3 GetObject charges combined.

The real value comes from what you avoid paying for.

• No Glue crawler compute.
• No crawler scheduling/management overhead.
• No stale data window between crawler runs.
• Near-real-time availability, which was typically under a minute end-to-end (S3 event notification delivery to SQS plus Lambda polling adds roughly 30–60 seconds) rather than minutes.

When to Use Which Approach

If your organization allows Horizon Catalog, that is the right path to go down. Horizon Catalog is undeniably the platform-native answer but, if you are stuck in a Glue-centric environment today this event-driven Lambda pattern is highly effective. It will provide clean names, automatic schema sync, and near-real-time availability with little operational overhead.

Conclusion

By using native S3 event notifications to handle your external catalog updates and by treating metadata commits as triggers to read the schema from the source of truth, you can maintain the precise registration your downstream tools require.

This Python based Lambda solution offers fast deployment and maintains a small monthly operating cost. The primary benefit is the user experience: analysts can execute simple queries like SELECT * FROM customers without addressing suffix-related inconsistencies generated from Snowflake-managed Iceberg tables.

We maintain the benefits of Snowflake-managed Iceberg while creating a bridge for previously mentioned solutions, such as Horizon Catalog, to become available.

Detailed setup guides, CloudFormation templates, and the complete source code are available upon request.

Event-Driven Glue Catalog Registration for Snowflake-Managed Iceberg Tables was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.