Google Cloud 发布无边界湖仓跨云缓存与连接增强
DataHot 速览
Google Cloud 宣布对其“无边界湖仓”（borderless Lakehouse）进行增强，帮助数据工程师、数据科学家以及日益增多的 AI Agent 直接在数据所在位置查询受治理的数据。该方案通过采用 Apache Iceberg REST 目录规范，联邦接入 Databricks Unity Catalog、AWS Glue、Snowflake Horizon 等目录，并借助 Partner Cross-Cloud Interconnect 建立跨云私密高带宽链路。此次新增跨云缓存预览功能，可在 Google Cloud 本地缓存高频访问数据，将跨云查询的远程传输量降至处理数据量的 5% 以下，以降低总体拥有成本（TCO）。
为什么值得关注：该更新直面多云数据环境下跨云查询成本高、ETL 链路脆弱的痛点，Iceberg REST 目录联邦与跨云缓存是湖仓架构演进的重要方向，对构建跨云数据平台与 Data Agent 的团队有直接参考价值。
本文目录 9 节
原文
Will Ochandarena
Group Product Manager
Jason Ganetsky
Staff Software Engineer
Try Gemini Enterprise today
The front door to AI in the workplace
Today, we are excited to announce enhancements to the borderless Lakehouse, our answer to how data engineers, data scientists, and increasingly, AI agents, can query governed data directly where it lives.
To reason accurately and automate complex enterprise workflows, agents and data consumers of all types need fast, unified access to an organization's complete data estate, joining customer records, transaction logs, and operational telemetry across clouds. However, modern enterprise data is rarely confined to a single location; data estates often span Amazon S3, Azure Data Lake Storage (ADLS), Google Cloud Storage, operational databases, and SaaS platforms like Salesforce, SAP, and Workday. Historically, uniting these distributed datasets required brittle ETL pipelines, duplicated storage, and prohibitive cross-cloud data transfer costs.
We introduced the borderless Lakehouse earlier this year to let organizations query and activate data in place across clouds. By adopting the Apache Iceberg REST catalog specification, we federate directly to catalogs such as Databricks Unity Catalog, AWS Glue, and Snowflake Horizon. We also introduced Partner Cross-Cloud Interconnect to establish high-bandwidth, private links to other cloud providers, lowering per-gigabyte transfer costs compared to the public internet.
Today, we are taking multi-cloud efficiency a step further by optimizing how much data needs to be transferred across the wire in the first place.
We are excited to announce two new features to help further reduce costs of querying cross-cloud data. First, the preview of cross-cloud caching for Lakehouse transparently accelerates cross-cloud queries in BigQuery and cuts remote transfer costs by caching frequently accessed data locally in Google Cloud. Combining standard Iceberg columnar compression with cross-cloud caching means you often only need to transfer under 5% of the data you process across clouds, which helps lower the Total Cost of Ownership (TCO) to make cross-cloud analytics and AI viable at enterprise scale. In addition, BigQuery cross-cloud connections are also available in preview to query non-Iceberg data in other clouds and accelerate workloads.
How cross-cloud caching works
Cross-cloud caching meets enterprise performance and security requirements with no knobs to turn or storage to manage to accelerate your queries. Some of the mechanisms used under the hood are:
- Sub-file block granularity: Instead of transferring entire multi-gigabyte files across clouds when a query touches only a few columns, cross-cloud caching operates at the sub-file block level for columnar formats like Apache Parquet. BigQuery caches only the specific column chunks and dictionary pages projected by the query. On a cache miss, BigQuery fetches the needed data from the remote cloud to answer the query, and saves a local copy in the cache for future queries, drastically cutting network transfer and latency on repeated workloads.
- Default encryption at rest: Cached data blocks are encrypted at rest by default using Google-managed encryption keys (GMEK) so that temporary cache storage maintains the same enterprise-grade security posture as native BigQuery storage without extra overhead.
- Tenant and regional isolation: Cache entries are strictly partitioned by project and catalog boundaries to help prevent cross-tenant data exposure. Lakehouse anchors both the local cache and query execution strictly to the configured Google Cloud region (e.g.,
us-east4) to support compliance with regional data residency requirements when querying remote clouds. - Freshness checks: Multi-cloud caching often forces a trade-off between speed and freshness. To avoid stale reads, BigQuery fetches remote object metadata before using cached data to ensure the data hasn’t changed and the user still has access. Any upstream table modification prompts BigQuery to fetch new files, while unreferenced cached blocks expire automatically, delivering local query speed with single-source-of-truth accuracy.
For more details on caching mechanics, statistics counters, and regional considerations, see the Lakehouse intelligent caching documentation.
Cross-cloud caching in action
So how does this work in day-to-day operations? Consider an e-commerce team querying a 10 TiB Iceberg sales table (aws_lakehouse_catalog.sales.web_sales) in Amazon S3, federated into Lakehouse from Databricks Unity Catalog. During evening promotional drops (8:00–9:00 PM), analysts query historical transactions to identify which storefronts drive peak volume and revenue among high-intent demographics:
Loading...
Initial execution: Cold columnar retrieval
On this initial cold run, the local cache is empty (cacheBytesRead: "0"). BigQuery applies partition pruning and column projection to transfer only the required Parquet byte ranges from Amazon S3 over Partner Cross-Cloud Interconnect:
Loading...
- Logical data processed: BigQuery processes 214.5 GiB across the 10 TiB dataset.
- Standard Iceberg compression efficiency: BigQuery reads 24.1 GiB from S3 thanks to standard Iceberg columnar compression with Zstandard (
zstd) — an 8.9:1 compression ratio. As these sub-file Parquet blocks arrive in Google Cloud, BigQuery populates the regional cache.
Follow-on exploration: Adding a dimension
In practice, analysts and agents rarely run the exact same query twice in a row. To drill deeper into fulfillment methods, the analyst modifies the query by adding the shipping method dimension (sm.sm_type):
Loading...
Job statistics for this follow-on query show:
Loading...
- 94.8% cache hit rate: BigQuery serves 24.1 GiB of previously queried columns directly from local cache.
- Granular remote retrieval: BigQuery transfers only 1.33 GiB from S3 for the new
ws_ship_mode_skcolumn andship_modetable. - Sub-file flexibility: Modifying a query reuses cached column chunks and transfers only newly required bytes.
Compounding efficiency at enterprise scale
When thinking about TCO of cross-cloud queries, the top two factors to account for are:
- Compression ratio: when using default compression algorithms (Zstandard/zstd) on Iceberg, columnar data is highly compressible. If you assume that your data achieves a compression ratio of 8:1, it means every 1 TiB of logical data processed only requires ~128 GiB of data to move over the network.
- Cache hit rates: when data is retrieved from cache rather than across the network because it was recently accessed, a network transit is avoided. Assuming 80% of your data results in a cache hit it means for every 100 GiB of physical data accessed only 20 GiB moves over the network.
Taking both factors and assumptions into account, for every 1 TiB of data your organization processes, you only need to transfer ~26 GiB across the network (under 3% of total data processed). Combining this reduction with Partner Cross-Cloud Interconnect lowers TCO enough to make cross-cloud analytics and AI cost-effective at petabyte scale.
BigQuery cross-cloud connections now in preview
Alongside cross-cloud caching, the preview of BigQuery cross-cloud connections lets organizations connect BigQuery directly to open-format data in Amazon S3 and Azure Storage.
Understanding when to use catalog federation versus cross-cloud connections is straightforward:
- BigQuery cross-cloud connections (for raw files): For standalone files (CSV, JSON, ad-hoc Parquet) without an Iceberg catalog, cross-cloud connections let you create BigQuery external tables referencing remote bucket paths directly.
- Lakehouse catalog federation (for Iceberg): For Iceberg data managed by catalogs like Databricks Unity, AWS Glue, or Snowflake Horizon, Lakehouse automatically synchronizes schemas and table snapshots to simplify the user experience and ensure users are always querying the latest data.
Cross-cloud connections serve as the modern architectural evolution by using standard BigQuery compute workers in Google Cloud regions rather than compute workers in other clouds. This approach helps unlock global region availability and provides full BigQuery feature parity — including with BigQuery AI and Gemini on remote files.
The cross-cloud caching capabilities for Lakehouse applies to data queried from BigQuery cross-cloud connections as well as Lakehouse catalog federation. To learn how to create connections and query external bucket paths, see the BigQuery cross-cloud connections setup documentation.
Posted in
这篇内容对你有用吗？
反馈只用于改善内容筛选，不等同于收藏