返回

Samsung Ads如何用Cognito联邦Snowflake Iceberg表

RSS AWS Big Data Blog 原文 · 未翻译 发布 2026-11-12 03:48

有效阅读仅保存在当前浏览器,用于「为你」推荐;暂停或清除记录。

DataHot 速览

Samsung Ads的ML团队将特征数据以Snowflake管理的Apache Iceberg表存放在Amazon S3,通过Amazon Cognito签发OAuth2令牌,把Snowflake Horizon Catalog联邦进AWS Lake Formation,让SageMaker AI与Athena等引擎直接读取同一份数据。方案用AWS PrivateLink和NAT网关打通VPC内的Glue连接,省去定时拷贝到S3的第二个管线和第二份副本,并统一由Lake Formation管控读取权限。

为什么值得关注:这是一份第一方的跨云数据联邦实践,展示了湖仓表格式加统一权限治理如何消除数据复制与双管线运维,对做数据平台集成与治理的从业者有直接参考价值。

本文目录 15 节
  1. Solution overview
  2. Why Amazon Cognito
  3. Prerequisites
  4. Walkthrough
  5. Step 1: Deploy the Cognito user pool and token customization Lambda function
  6. Step 2: Verify the access token
  7. Step 3: Configure Snowflake
  8. Step 4: Authorize AWS PrivateLink in Snowflake
  9. Step 5: Set up VPC and PrivateLink connectivity
  10. Step 6: Register with Lake Formation and create the federated catalog
  11. Step 7: Query the federated tables from Amazon Athena
  12. Clean up
  13. AWS account
  14. Snowflake account
  15. Conclusion

原文

This post is co-written with Ajay Gunasekaran, ML Platform Engineer at Samsung Ads.

Samsung Ads, the advertising division of Samsung Electronics, keeps its feature data in Snowflake as Apache Iceberg tables. Its machine learning (ML) team trains large-scale models on Amazon SageMaker AI, serving programmatic buyers who reach Samsung Smart TV and connected TV audiences. Connecting the two usually means copying the data into Amazon Simple Storage Service (Amazon S3) on a schedule. That adds a second pipeline to operate, a second copy to store, and two sets of permissions to keep in agreement.

Instead of copying the data, a team can write the final pipeline stage as Snowflake-managed Iceberg tables in Amazon S3 and federate them into AWS Lake Formation. Amazon Cognito supplies the OAuth2 tokens that Snowflake trusts. Both applications then read one copy of the data in Amazon S3, with Snowflake managing the tables and the team using Lake Formation to control which principals can read them. Model engineers can choose the query engine that fits each training job, without duplicating data or maintaining a second pipeline.

In this post, we show you how to build the pattern Samsung Ads used. You federate Snowflake Iceberg tables into AWS Lake Formation with Amazon Cognito, then query them from Amazon Athena.

Solution overview

This solution federates the Snowflake Horizon Catalog into AWS Lake Formation, using Amazon Cognito to issue OAuth2 tokens for machine-to-machine authentication. AWS Glue connection runs inside an Amazon Virtual Private Cloud (Amazon VPC), reaching Snowflake through AWS PrivateLink and the Cognito token endpoint through a NAT gateway. AWS analytics engines then query Snowflake-managed Iceberg tables under Lake Formation governance, with no second copy of the tables to maintain. The following diagram shows the query path, from Amazon Athena through AWS Lake Formation and the AWS Glue connection to Snowflake and Amazon S3.

Figure 1: Architecture for Snowflake Horizon Catalog federation to AWS Lake Formation using Amazon Cognito

Here’s how the workflow runs:

  1. A user or application submits a SQL query in Amazon Athena that references a federated table in the Snowflake catalog.
  2. Athena delegates catalog resolution and permission checks to Lake Formation, which evaluates fine-grained access controls on the federated catalog entry.
  3. Lake Formation resolves the table reference through the AWS Glue Data Catalog, which holds the connection configuration for Snowflake and invokes the AWS Glue connection.
  4. AWS Glue connection runs in a private subnet. It sends a client_credentials request to the Amazon Cognito token endpoint with the client ID, client secret, and scope. The request leaves through the NAT gateway and internet gateway because the machine-to-machine token endpoint is public.
  5. Amazon Cognito invokes the Pre Token Generation Lambda trigger (V3_0), which adds the aud claim that Snowflake External OAuth validation requires. Amazon Cognito returns the access token to the connection.
  6. Using that token, AWS Glue connection calls the Snowflake Horizon Catalog Iceberg REST endpoint for table metadata. The response includes the schema and the location of the data files in Amazon S3. An Amazon Route 53 private hosted zone resolves the hostname, because the Snowflake endpoint service doesn’t provide private DNS. The request travels through the VPC endpoint over AWS PrivateLink.
  7. The connection returns the metadata to Lake Formation, which generates temporary, scoped Amazon S3 credentials and passes them to Athena.
  8. Athena uses those credentials to read the Iceberg metadata and Parquet data files from the registered Amazon S3 location, runs the query, and returns results.

Why Amazon Cognito

AWS Glue connection authenticates with the OAuth2 client_credentials grant, so providers Snowflake trusts that support that grant can issue the token. Samsung Ads chose Amazon Cognito for three reasons:

  • Each table lookup against Snowflake depends on a token from the provider, so the team wanted to own it alongside the rest of the solution.
  • An enterprise identity provider is typically administered by a central security team, so each new machine-to-machine client is a cross-team request. Owning the user pool makes it self-service.
  • Each app client is a distinct machine identity, and the security integration trusts the user pool rather than an individual client. You can therefore serve several workloads from one integration, each with its own read-only Snowflake role.

Prerequisites

Before you begin, verify that you have the following:

  • Snowflake-managed Iceberg tables created in Snowflake Horizon Catalog using CATALOG = 'SNOWFLAKE' with an external volume pointing to the S3 bucket. If you don’t have one, see Configure an external volume for Amazon S3 in the Snowflake documentation.

AWS account with an AWS Identity and Access Management (IAM) principal that has data lake administrator permissions in AWS Lake Formation. The principal also needs permissions to create IAM roles and policies, Amazon Cognito user pools, AWS Lambda functions, VPC resources, and AWS CloudShell. We use AWS Command Line Interface (AWS CLI) commands throughout this post. Amazon S3 bucket that holds your Snowflake-managed Iceberg data files. An AWS user with long-term access keys that you delete immediately after Step 4:

  • SYSTEM$AUTHORIZE_PRIVATELINK needs a federation token from the long-term keys of an IAM user, because sts:GetFederationToken rejects role-session credentials. Snowflake persists the authorization, so you can delete the key and the user afterwards and the federation keeps working.

Walkthrough

Two AWS CloudFormation stacks cover the setup that is identical in every deployment. The Snowflake and Lake Formation steps stay as explicit SQL and AWS CLI commands, so you can adapt them to your existing Snowflake account and VPC.

StepWhat you configureHow you run it
1 & 2Cognito user pool, machine-to-machine app client, and the Lambda function that adds the aud claimLaunch Stack, snowflake-federation-oauth
3Snowflake security integration, read-only role, service user, and table grantsSnowflake SQL worksheet
4PrivateLink authorization for your AWS accountAWS CLI and Snowflake SQL worksheet
5VPC, PrivateLink endpoint, Route 53 private hosted zone, and the Lake Formation data access roleLaunch Stack, snowflake-federation-network
6S3 location registration, AWS Glue connection, federated catalog, and Lake Formation permissionsAWS CLI
7Query validationAthena console

Step 1: Deploy the Cognito user pool and token customization Lambda function

Snowflake External OAuth validation requires an aud claim in each access token. The OAuth2 client_credentials grant doesn’t define that claim, so the Pre Token Generation trigger adds it. This stack creates the user pool, the machine-to-machine app client, and a Pre Token Generation Lambda function that adds the claim. The stack also writes the app client secret to AWS Secrets Manager in the format the AWS Glue connection expects. It then generates the Snowflake SQL statements you run in Step 3.

  1. Choose Launch Stack.
  1. In the Region selector, confirm the Region matches your Snowflake account Region, then choose Next.
  2. For Stack name, keep snowflake-federation-oauth. Leave all parameters at their default values, then choose Next.
  3. On Configure stack options, keep the defaults and choose Next.
  4. On Review and create, select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Submit.
  5. When the stack status is CREATE_COMPLETE, choose the Outputs tab.

You use six of the outputs in later steps.

OutputUsed in
VerifyAudClaimCommandStep 2
SnowflakeSecurityIntegrationSQLStep 3
OAuthClientSecretArnStep 5 and Step 6
CognitoTokenEndpoint, CognitoClientId and CognitoScopeStep 3 and Step 6

Figure 2: CloudFormation Outputs tab showing the values used in later steps

Note: After the connection validates in Step 6, AWS Glue caches the token it mints by writing it back into this secret. Treat read access to the secret as read access to the federated tables.

Step 2: Verify the access token

Confirm that the Lambda function is adding the claim before you configure Snowflake. A missing claim doesn’t fail here. It surfaces several steps later as an AWS Glue connection that can’t authenticate, which is harder to attribute.

  1. On the Outputs tab, copy the value of VerifyAudClaimCommand.
  2. Choose the AWS CloudShell icon on the console navigation bar, as shown in the following screenshot.
  3. Paste the command and run it. Confirm two values in the output. The aud claim matches your Cognito client ID, and scope is snowflake/catalog.read.

Figure 3: Decoded access token payload showing the injected aud claim and the catalog.read scope

Step 3: Configure Snowflake

In this step you configure Snowflake to trust Cognito tokens, create the service user that Lake Formation authenticates as, and grant it read access to your tables.

  1. Run the following statement in a Snowflake worksheet, replacing the three placeholders with the CognitoIssuerUri, CognitoJwksUrl, and CognitoClientId outputs from Step 1. The SnowflakeSecurityIntegrationSQL output contains the same statement with your values filled in. CREATE omits OR REPLACE, so it fails rather than overwriting an existing integration.
USE ROLE ACCOUNTADMIN;

-- Return a row if the integration exists. If so, use ALTER SECURITY INTEGRATION instead of CREATE
SHOW INTEGRATIONS LIKE 'lakeformation_cognito_oauth';

CREATE SECURITY INTEGRATION lakeformation_cognito_oauth
  TYPE = EXTERNAL_OAUTH
  ENABLED = TRUE
  EXTERNAL_OAUTH_TYPE = CUSTOM
  EXTERNAL_OAUTH_ISSUER = '<CognitoIssuerUri>'
  EXTERNAL_OAUTH_JWS_KEYS_URL = '<CognitoJwksUrl>'
  EXTERNAL_OAUTH_TOKEN_USER_MAPPING_CLAIM = 'sub'
  EXTERNAL_OAUTH_SNOWFLAKE_USER_MAPPING_ATTRIBUTE = 'login_name'
  EXTERNAL_OAUTH_AUDIENCE_LIST = ('<CognitoClientId>')
  EXTERNAL_OAUTH_ANY_ROLE_MODE = 'ENABLE'
  EXTERNAL_OAUTH_BLOCKED_ROLES_LIST = ('ACCOUNTADMIN', 'SECURITYADMIN', 'USERADMIN', 'SYSADMIN');

Snowflake normally takes the role from a session:role:<ROLE> scope, which Amazon Cognito custom scopes can’t carry because they use the form resourceServer/scope. The statement therefore sets EXTERNAL_OAUTH_ANY_ROLE_MODE to ENABLE, which lets the session use any role granted to the service user, so keep that user to a single role. The blocked roles list also covers SYSADMIN and USERADMIN, which the Snowflake default list doesn’t.

  1. Create the read-only role and the service user. Set LOGIN_NAME to the CognitoClientId output, because the token’s sub claim carries the client ID and the integration maps sub to login_name.
CREATE ROLE lakeformation_catalog_reader
  COMMENT = 'Read-only access for AWS Lake Formation catalog federation';

CREATE USER lakeformation_service_user_cognito
  LOGIN_NAME = '<CognitoClientId from Step 1>'
  DEFAULT_ROLE = lakeformation_catalog_reader
  TYPE = SERVICE;

GRANT ROLE lakeformation_catalog_reader TO USER lakeformation_service_user_cognito;
  1. Grant the role read access to the tables you want to federate. For the privileges the Iceberg REST Catalog API requires, see Configure access control in the Snowflake documentation.
GRANT USAGE ON DATABASE <database>
  TO ROLE lakeformation_catalog_reader;

GRANT USAGE ON SCHEMA <database>.<schema>
  TO ROLE lakeformation_catalog_reader;

GRANT SELECT ON ALL ICEBERG TABLES IN SCHEMA <database>.<schema>
  TO ROLE lakeformation_catalog_reader;

GRANT SELECT ON FUTURE ICEBERG TABLES IN SCHEMA <database>.<schema>
  TO ROLE lakeformation_catalog_reader;

Note: Federate a schema that contains only Snowflake-managed Iceberg tables. An externally managed table can fail the entire table listing. Converted tables, tables loaded from Iceberg-compatible Parquet files and clones whose source table the role can’t access aren’t supported. Tables with mixed-case quoted names aren’t federated, because the connection you create in Step 6 sets CATALOG_CASING_FILTER to UPPERCASE_ONLY.

Snowflake masking and row access policies don’t apply when Athena reads a federated table, because Lake Formation vends the Amazon S3 credentials and Athena reads the Parquet files directly. Keep tables that rely on them out of the federated schema or recreate the controls in Lake Formation and test them.

  1. Next, find the Amazon S3 bucket and prefix your external volume writes to. You need both in Step 5.
DESC EXTERNAL VOLUME <your_external_volume>;

STORAGE_BASE_URL holds the bucket and an optional prefix. For s3://my-bucket/iceberg/, the bucket is my-bucket and the prefix is iceberg. If nothing follows the bucket name, leave the prefix blank in Step 5. A prefix is better, because it limits the data access role to that path. If ENCRYPTION_TYPE is AWS_SSE_KMS, note the key in ENCRYPTION_KMS_KEY_ID for Step 5. If it is NONE, check the bucket’s default encryption in the Amazon S3 console. The Snowflake role that owns the Iceberg tables must keep the USAGE privilege on the external volume, or operations that request vended credentials fail.

If the bucket uses a customer managed KMS key, set KMS_KEY_ID on the external volume to the full key ARN. Then add the following statement to the key policy, so the session Snowflake uses for the role in STORAGE_AWS_ROLE_ARN can read the encrypted files. Without it, the AWS Glue connection fails with an error that says no session policy allows kms:Decrypt.

{
  "Sid": "AllowSnowflakeVolumeSession",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:sts::<account-id>:assumed-role/<external-volume-role>/snowflake"
  },
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey",
    "kms:GenerateDataKey"
  ],
  "Resource": "*"
}

Step 4: Authorize AWS PrivateLink in Snowflake

Snowflake requires proof that you own the AWS account before it accepts a PrivateLink connection. You provide that proof as an AWS Security Token Service (AWS STS) federation token. You must call get-federation-token with the long-term credentials of an IAM user, because AWS STS doesn’t support calling it with temporary credentials from an assumed role.

  1. Using an IAM user with long-term access keys, generate a federation token:
aws sts get-federation-token --name snowflake-privatelink --duration-seconds 900 --profile <iam-user-profile>

The token appears in the SQL statement, so Snowflake keeps it in query history. The 15-minute duration limits how long it stays valid.

  1. In your Snowflake worksheet, pass your AWS account ID and the full JSON output from the previous command:
USE ROLE ACCOUNTADMIN;
SELECT SYSTEM$AUTHORIZE_PRIVATELINK('<aws-account-id>', '<full-federation-token-json>');

The output confirms that Snowflake authorized PrivateLink for your account.

Figure 4: Snowflake output confirming PrivateLink authorization

  1. Retrieve the two values you need for the stack in Step 5:
WITH cfg AS (SELECT PARSE_JSON(SYSTEM$GET_PRIVATELINK_CONFIG()) AS j)
SELECT j:"regionless-privatelink-account-url"::string AS privatelink_hostname,
       j:"privatelink-vpce-id"::string AS vpce_service_name
FROM cfg;

Record both values. privatelink_hostname is the Snowflake PrivateLink hostname, and vpce_service_name is the endpoint service name.

After authorization succeeds, delete the temporary IAM user and its access key. Revoking PrivateLink access later requires a new federation token, so recreate a similar user at that point. For more information, see How to retrieve a Federation Token from AWS for PrivateLink Self-Service.

Step 5: Set up VPC and PrivateLink connectivity

This stack creates the interface VPC endpoint to the Snowflake PrivateLink service and the Amazon Route 53 private hosted zone that resolves your Snowflake hostname to it. It also creates the security group that the endpoint shares with the AWS Glue connection. Keep Existing VPC ID blank and it also builds a VPC with public and private subnets and a NAT gateway.

To use an existing VPC, supply Existing VPC ID and two or more private subnet IDs in different Availability Zones. The stack then creates no subnets and no NAT gateway, so confirm three things first:

  • You have enabled DNS resolution and DNS hostnames on the VPC.
  • The private subnets have a 0.0.0.0/0 route to a NAT gateway, which the AWS Glue connection uses to reach the Amazon Cognito token endpoint. Amazon Cognito doesn’t support AWS PrivateLink, but the traffic stays on the AWS network because both sit in the same Region. Without this route, the stack still succeeds and the connection fails in Step 6.
  • No Amazon Route 53 private hosted zone for privatelink.snowflakecomputing.com is already associated with the VPC. A second zone for the same domain makes name resolution unpredictable.

To deploy the stack, complete the following steps:

  1. Choose Launch Stack.
  1. Confirm the Region selector matches your Snowflake account Region, then choose Next.
  2. For Stack name, keep snowflake-federation-network, then enter the following parameters.
ParameterValue
Snowflake PrivateLink hostnameprivatelink_hostname from Step 4
Snowflake PrivateLink service namevpce_service_name from Step 4
Iceberg S3 bucket nameThe bucket in the STORAGE_BASE_URL output from Step 3
Iceberg data key prefixThe prefix in the STORAGE_BASE_URL output from Step 3. Leave blank if you have none
Iceberg bucket KMS key ARNKMS key ARN. Leave blank if the bucket does’t use KMS
OAuth client secret ARNOAuthClientSecretArn from Step 1

If the bucket uses SSE-KMS, the stack grants the data access role kms:Decrypt on the key you enter. The default key policy already allows this. If your key policy doesn’t delegate access to IAM, add the role as a key user.

  1. Choose Next, keep the defaults on Configure stack options, and choose Next.
  2. On Review and create, select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Submit.
  3. When the stack status is CREATE_COMPLETE, which takes about 5 minutes, choose the Outputs tab and record PrivateSubnetIds, GlueConnectionSecurityGroupId, LakeFormationDataRoleArn, and RegisteredS3Location. You use all four values in Step 6.

NAT gateway and the interface VPC endpoint carry hourly and data processing charges. For more information, see Amazon VPC pricing.

Step 6: Register with Lake Formation and create the federated catalog

The remaining setup is a sequence of AWS CLI commands. Run them in AWS CloudShell in the same Region as both stacks.

  1. Set the values used by the commands that follow. Fill in the Region, and the stack outputs from Step 1 and Step 5.
REGION='<your Region>'
ACCT=$(aws sts get-caller-identity --query Account --output text)
PREFIX=snowflake-federation
CONN=$PREFIX-iceberg-connection
CATALOG=$PREFIX-catalog
S3_LOCATION='<RegisteredS3Location from Step 5>'
LFROLE='<LakeFormationDataRoleArn from Step 5>'
SECRET_ARN='<OAuthClientSecretArn from Step 1>'
TOKEN_URL='<CognitoTokenEndpoint from Step 1>'
CLIENT_ID='<CognitoClientId from Step 1>'
SCOPE='<CognitoScope from Step 1>'
SUBNET='<one subnet ID from PrivateSubnetIds in Step 5>'
SG='<GlueConnectionSecurityGroupId from Step 5>'
WORKSPACE_URL='https://<privatelink_hostname from Step 4>/polaris/api/catalog'
REMOTE_DB='<Snowflake database name, uppercase>'
SCHEMA='<Snowflake schema name, lowercase>'
QUERY_PRINCIPAL='<IAM role or user ARN that will run Athena queries>'
  1. Register the Amazon S3 location that holds your Iceberg data. This is what lets Lake Formation vend scoped credentials to Athena for the underlying data files.
aws lakeformation register-resource \
  --resource-arn "$S3_LOCATION" \
  --role-arn "$LFROLE" --with-federation
  1. Create the AWS Glue connection. This writes the definition to a file, then creates it.
cat > conn.json <<JSON
{
  "Name": "$CONN",
  "Description": "Snowflake Iceberg REST catalog through AWS Cognito.",
  "ConnectionType": "ICEBERGRESTCATALOG",
  "ConnectionProperties": {
    "INSTANCE_URL": "$WORKSPACE_URL",
    "ROLE_ARN": "$LFROLE",
    "CATALOG_CASING_FILTER": "UPPERCASE_ONLY"
  },
  "AuthenticationConfiguration": {
    "AuthenticationType": "OAUTH2",
    "SecretArn": "$SECRET_ARN",
    "OAuth2Properties": {
      "OAuth2GrantType": "CLIENT_CREDENTIALS",
      "TokenUrl": "$TOKEN_URL",
      "OAuth2ClientApplication": { "UserManagedClientApplicationClientId": "$CLIENT_ID" },
      "TokenUrlParametersMap": { "scope": "$SCOPE" }
    }
  },
  "PhysicalConnectionRequirements": {
    "SubnetId": "$SUBNET",
    "SecurityGroupIdList": ["$SG"]
  }
}
JSON

aws glue create-connection --connection-input file://conn.json
  1. Check the connection status. AWS Glue validates asynchronously. It creates the network interface, completes the OAuth2 exchange with Amazon Cognito, and calls the Snowflake Iceberg REST API over AWS PrivateLink. Repeat this command until it returns READY, which takes approximately 5–10 minutes.
aws glue get-connection --name $CONN --query Connection.Status --output text

If it returns FAILED, read the reason and correct the Snowflake configuration from Step 3. A missing security integration, service user, or grant is the usual cause.

aws glue get-connection --name $CONN --query Connection.StatusReason --output text

Then delete the connection before you retry, because the name stays in use.

aws glue delete-connection --connection-name $CONN
  1. Register the connection with Lake Formation using privileged access. A federated catalog can only use a connection registered this way.
aws lakeformation register-resource \
  --resource-arn arn:aws:glue:$REGION:$ACCT:connection/$CONN \
  --role-arn "$LFROLE" --with-federation --with-privileged-access
  1. Create the federated catalog, then confirm the tables surface through it.
aws glue create-catalog --name $CATALOG --catalog-input '{
  "FederatedCatalog": {"Identifier": "'"$REMOTE_DB"'", "ConnectionName": "'"$CONN"'"},
  "CreateDatabaseDefaultPermissions": [],
  "CreateTableDefaultPermissions": []
}'

aws glue get-tables --catalog-id $ACCT:$CATALOG --database-name $SCHEMA --query 'TableList[].Name'
  1. Grant Lake Formation permissions to the data teams that will query the tables. The principal that created the catalog can already query it. Every other principal needs an explicit grant, including other Lake Formation administrators.
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier="$QUERY_PRINCIPAL" \
  --resource '{"Database":{"CatalogId":"'"$ACCT:$CATALOG"'","Name":"'"$SCHEMA"'"}}' \
  --permissions DESCRIBE

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier="$QUERY_PRINCIPAL" \
  --resource '{"Table":{"CatalogId":"'"$ACCT:$CATALOG"'","DatabaseName":"'"$SCHEMA"'","TableWildcard":{}}}' \
  --permissions SELECT DESCRIBE

Note: The catalog ID for a federated catalog is :, not the bare account ID. Using the bare account ID grants permissions on the local AWS Glue Data Catalog instead, which succeeds and then leaves Athena returning an access denied error.

Step 7: Query the federated tables from Amazon Athena

  1. Open the Amazon Athena console and choose Query editor. If this is the first time you use Athena in this Region, choose Settings, then Manage, and set an Amazon S3 location for query results.
  2. In the left panel, select the following.
FieldValue
Data sourceAwsDataCatalog
CatalogYour federated catalog, for example snowflake-federation-catalog
DatabaseYour Snowflake schema name, in lowercase
  1. Run the following query:
SELECT * FROM "<snowflake_schema>"."<snowflake_tablename>" LIMIT 10;

Rows in the results pane confirm the federation works end to end, as shown in the following screenshot.

Figure 5: Query results in Amazon Athena from a Snowflake-managed Iceberg table

To confirm that Athena reads current Snowflake data rather than a copy, insert a row in Snowflake and run the query again. The new row appears with no additional data movement into your AWS account.

Dropping a table in Snowflake doesn’t revoke the Lake Formation permissions granted on it. Revoke those permissions explicitly when you remove a federated table.

Clean up

To avoid incurring future charges, delete the resources you created while following this post. Complete the following steps in both your AWS account and your Snowflake account.

AWS account

  1. On the AWS Lake Formation console, choose Catalogs, select your federated catalog, then choose Actions and Delete catalog.
  2. Record the network interfaces the AWS Glue connection created, while the connection still exists. Filtering on the Step 5 security group and the ICEBERGRESTCATALOG description keeps the list to this deployment.
SG='<GlueConnectionSecurityGroupId from Step 5>'
aws ec2 describe-network-interfaces \
  --filters "Name=group-id,Values=$SG" "Name=description,Values=*ICEBERGRESTCATALOG" \
  --query 'NetworkInterfaces[].NetworkInterfaceId' --output text | tr '\t' '\n' > glue-enis.txt

cat glue-enis.txt
  1. Deregister and delete the AWS Glue connection. Do this before you delete the stacks, because the stack deletion fails while the connection’s network interfaces still hold the security group. Run the following in AWS CloudShell, replacing the Region, account ID, and connection name with your values.
aws lakeformation deregister-resource --resource-arn arn:aws:glue:<region>:<account-id>:connection/<glue-connection-name>
aws glue delete-connection --connection-name <glue-connection-name>
  1. Deleting the connection releases its network interfaces, but they stay in the in-use state for several minutes and AWS Glue doesn’t remove them. Wait for each one to reach available, because the Amazon Elastic Compute Cloud (Amazon EC2) console Detach action fails even with Force detachment selected. The following script checks each interface you recorded and deletes it when ready.
#!/bin/bash
while read -r eni; do
  [ -z "$eni" ] && continue
  for attempt in $(seq 1 20); do
    status=$(aws ec2 describe-network-interfaces \
      --network-interface-ids "$eni" \
      --query 'NetworkInterfaces[0].Status' \
      --output text 2>/dev/null || echo deleted)
    if [ "$status" = "available" ]; then
      aws ec2 delete-network-interface --network-interface-id "$eni"
      echo "$eni deleted"
      break
    fi
    if [ "$status" = "deleted" ]; then
      echo "$eni no longer exists"
      break
    fi
    echo "$eni is $status, checking again in 30 seconds"
    sleep 30
  done
done < glue-enis.txt
  1. On the Lake Formation console, under Administration, choose Data lake locations, select the registered Amazon S3 location, then choose Actions and Remove.
  1. On the AWS CloudFormation console, delete snowflake-federation-network first, then snowflake-federation-oauth. This removes the VPC endpoint, the Amazon Route 53 private hosted zone, the security group, and the IAM role. It also removes the Amazon Cognito user pool, the AWS Lambda function, and the client secret. If the stack created the VPC, subnets, and NAT gateway, it removes those too.

Snowflake account

  1. Open a SQL worksheet and run the following:
USE ROLE ACCOUNTADMIN;
DROP SECURITY INTEGRATION lakeformation_cognito_oauth;

USE ROLE USERADMIN;
DROP USER lakeformation_service_user_cognito;
DROP ROLE lakeformation_catalog_reader;
  1. Revoke the PrivateLink authorization for your AWS account. This requires a fresh federation token, generated the same way as during setup, from an IAM user with long-term access keys:
aws sts get-federation-token --name snowflake-privatelink --duration-seconds 900 --profile <iam-user-profile>
USE ROLE ACCOUNTADMIN;
SELECT SYSTEM$REVOKE_PRIVATELINK('<aws-account-id>', '<full-federation-token-json>');
  1. Confirm the account is no longer authorized:
SELECT SYSTEM$GET_PRIVATELINK_AUTHORIZED_ENDPOINTS();

Deleting the federated catalog and connection doesn’t delete data in Snowflake or Amazon S3. It only removes the federation configuration in your AWS account.

Conclusion

In this post, you learned how to federate Snowflake Horizon Catalog Iceberg tables into AWS Lake Formation and query them from Amazon Athena. Amazon Cognito handled the OAuth2 tokens for machine-to-machine authentication. To extend this pattern to additional workloads, create a new app client in your Cognito user pool and a corresponding read-only Snowflake role.

To get started, see Catalog federation to remote Iceberg catalogs and Federate to Snowflake Iceberg Catalog in the AWS Lake Formation Developer Guide, and External OAuth overview in the Snowflake documentation.

这篇内容对你有用吗?

反馈只用于改善内容筛选,不等同于收藏

分享这条资讯
分享海报
保存图片
iOS 也可以长按图片保存