DuckDB-Wasm 结合 OPFS 实现浏览器持久化数据库
DataHot 速览
DuckDB-Wasm 自 2021 年发布以来无法持久化数据库,数据只存在于 Wasm 堆内存中,关闭标签页即丢失,此前只能把表序列化为 Parquet 存入 IndexedDB 再重新注册。现代浏览器自 2023 年 3 月起支持 Origin Private File System(OPFS),提供按源隔离的沙箱文件系统与随机读写能力,DuckDB-Wasm 可将其作为存储后端。文章演示了以 opfs:// 路径打开数据库(如 opfs://analytics.duckdb,读写模式),生成带预写日志和检查点的标准 .duckdb 文件,可在页面重载与浏览器重启后存活。作者提醒 npm 上 latest 版本 1.33.1-dev57.0 会创建 OPFS 文件却不写入、路径被规范化为单斜杠导致不匹配,建议锁定 1.32.0 或使用 1.33.1-dev64.0 及以后的 next 标签。
为什么值得关注:浏览器端持久化是 DuckDB-Wasm 长期缺失的关键能力,OPFS 方案让纯前端分析应用无需自建应用层序列化即可保留数据,对做嵌入式分析与轻量数据产品的从业者有直接参考价值。
原文
Carlo Piovesan, Geertjan Wielenga 2026-09-18 | 9 min
TL;DR: DuckDB-Wasm can open a persistent database file in the browser's Origin Private File System (OPFS). This post shows how, and when data reaches disk.
When DuckDB-Wasm was launched in 2021, databases could not be persisted: everything lived in the Wasm heap and vanished when the tab closed. Keeping data meant serializing tables to Parquet, storing the bytes in IndexedDB, and re-registering them on the next page load. This was doable, but had to be handled at the application layer and was not offered out of the box by DuckDB-Wasm.
Modern browsers (since March 2023) now ship the Origin Private File System (OPFS), a per-origin, sandboxed file system with random-access reads and writes. DuckDB-Wasm (tested with versions 1.32.0 and 1.33.1-dev64.0) can use it as a storage backend, as described in the DuckDB documentation: a database opened at an opfs:// path survives reloads and browser restarts.
The following call opens a database file in OPFS:
awaitdb.open({path:'opfs://analytics.duckdb',accessMode:duckdb.DuckDBAccessMode.READ_WRITE,});The result is a regular .duckdb file with a write-ahead log and checkpoints that survives page reloads and browser restarts.
Note: at the time of writing, the build that npm serves aslatest(1.33.1-dev57.0) creates the OPFS files but never writes to them, so nothing persists. It canonicalizes the path toopfs:/analytics.duckdbwith a single slash, which no longer matches the OPFS handle. Pin 1.32.0 or use thenexttag (1.33.1-dev64.0 or later).
Opening a Database
The setup is the same as for any DuckDB-Wasm application: pick a bundle, start a worker, instantiate the database. The only new part is the open call, marked below. The import resolves to whichever version is installed, and getJsDelivrBundles() fetches the matching worker and .wasm files, so install a version that persists correctly: npm install @duckdb/[email protected] or @next.
import*asduckdbfrom'@duckdb/duckdb-wasm';constbundles=duckdb.getJsDelivrBundles();constbundle=awaitduckdb.selectBundle(bundles);// Worker scripts must be same-origin, so wrap the CDN worker URL in a BlobconstworkerUrl=URL.createObjectURL(newBlob([`importScripts("${bundle.mainWorker}");`],{type:'text/javascript'}));constworker=newWorker(workerUrl);constdb=newduckdb.AsyncDuckDB(newduckdb.ConsoleLogger(),worker);awaitdb.instantiate(bundle.mainModule,bundle.pthreadWorker);URL.revokeObjectURL(workerUrl);// NEW: open a persistent database in OPFS instead of the default :memory:awaitdb.open({path:'opfs://analytics.duckdb',accessMode:duckdb.DuckDBAccessMode.READ_WRITE,});constconn=awaitdb.connect();awaitconn.query(`
CREATE TABLE IF NOT EXISTS transactions (
id BIGINT,
ts TIMESTAMP,
merchant VARCHAR,
category VARCHAR,
amount DECIMAL(10, 2)
);
`);awaitconn.query(`INSERT INTO transactions VALUES (1, now(), 'Coolblue', 'electronics', 49.95)`);awaitconn.query('CHECKPOINT');constresult=awaitconn.query('SELECT count(*) AS n FROM transactions');console.log(result.toArray()[0].n);Reload the page and run the same code. The CREATE TABLE IF NOT EXISTS statement finds the existing table and does nothing, the insert adds a second row, and the count prints 2. There is no sync step, no export, no localStorage key to remember. The opfs:// prefix tells DuckDB-Wasm's file system layer to resolve the path against the origin's private file system instead of the in-memory Emscripten file system.
Opening the database creates the database file and its .wal in OPFS. Builds from 1.33.1-dev64.0 onward also create two empty helper files, .wal.checkpoint and .wal.recovery, that DuckDB uses during checkpointing. The .duckdb file is a regular DuckDB database file. If you pull it out of OPFS (shown below) and open it with the CLI or the Python client, it works.
Data Files
The same prefix works for data files. A common pattern is to load a remote dataset once, keep it in the persistent database, and cache derived results as Parquet files in OPFS. The example below uses the TPC-H orders table (scale factor 0.01, about 1,500 rows) that the DuckDB web shell serves:
awaitconn.query(`
CREATE TABLE IF NOT EXISTS orders AS
SELECT * FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet';
`);awaitconn.query('CHECKPOINT');DuckDB-Wasm reads the remote file with HTTP range requests. Because the table is created with IF NOT EXISTS, the file is fetched only on the first page load; on later loads the table comes from OPFS and no request goes to shell.duckdb.org. You can see this in the browser's Network tab, which lists the range requests on the first load and stays quiet afterwards, or in DuckDB-Wasm's own logs: the ConsoleLogger passed to AsyncDuckDB records each HTTP read, so the absence of those log lines on a reload confirms the data is served entirely from OPFS.
With the data local, an aggregation can be written to a Parquet file in OPFS and read back later:
COPY(SELECTo_orderpriorityASpriority,date_trunc('month',o_orderdate)ASmonth,sum(o_totalprice)AStotalFROMordersGROUPBYALL)TO'opfs://cache/monthly_totals.parquet';SELECT*FROM'opfs://cache/monthly_totals.parquet';Nested directories such as cache/ are created on demand. OPFS files are ordinary DuckDB file paths, so globbing, read_csv and the other readers work as usual. Reading and writing opfs:// paths from SQL needs one extra option on open(), described next.
File Handling Modes
With opfs: { fileHandling: 'auto' }, DuckDB-Wasm scans each statement for single-quoted 'opfs://...' literals, registers those files before execution (creating them and any missing directories if needed) and drops the handles afterwards. The option only takes effect when the database itself was opened from an opfs:// path. Without it, every file other than the database has to be registered by hand:
// Option 1: automatic registration of opfs:// paths found in SQLawaitdb.open({path:'opfs://analytics.duckdb',accessMode:duckdb.DuckDBAccessMode.READ_WRITE,opfs:{fileHandling:'auto'},});// Option 2: manual registration (the default)awaitdb.open({path:'opfs://analytics.duckdb',accessMode:duckdb.DuckDBAccessMode.READ_WRITE,});awaitdb.registerOPFSFileName('opfs://cache/monthly_totals.parquet');// ... run queries against it ...awaitdb.dropFile('opfs://cache/monthly_totals.parquet');Automatic mode is convenient for one-off reads. Manual mode requires more code but avoids re-acquiring an OPFS access handle on every statement, which adds up for applications that run many small queries. A file can be held by only one handle at a time, so the DuckDB documentation recommends dropping registered files with db.dropFile() before another connection or database instance opens them.
Durability
DuckDB-Wasm writes to OPFS the same way native DuckDB writes to a local disk: through a write-ahead log and periodic checkpoints. What differs is that a browser tab is rarely closed cleanly, so the defaults that work on a desktop can leave you with a slow reopen.
DuckDB uses a write-ahead log. Committed transactions are appended to analytics.duckdb.wal first. The main file is updated at checkpoint time. A checkpoint happens automatically when the WAL grows past checkpoint_threshold (16 MB by default), when the database is closed cleanly, or when you run CHECKPOINT yourself.
In a desktop process, "closed cleanly" is the common case. In a browser tab, it is not: the user closes the tab, the phone kills the background page, the laptop lid goes down. None of these run your shutdown code reliably. Two rules follow from that.
Call CHECKPOINT after writes you cannot afford to lose. The DuckDB documentation is explicit about this: writes are flushed to OPFS by CHECKPOINT. Committed transactions are appended to the WAL, and DuckDB replays the WAL on the next open, but a browser tab can be terminated at any point, so a checkpoint is the only way to be certain that the data is in the main file.
Checkpoint per batch, not per statement. A large WAL also makes the next open slower, because replay has to happen before the first query. For an interactive app, checkpointing after each batch of user edits keeps both the data safe and the reopen fast:
awaitconn.query('INSERT INTO transactions VALUES (...)');awaitconn.query('CHECKPOINT');If you would rather not track batches, set the checkpoint threshold to zero once after connecting. DuckDB then checkpoints after every statement, which costs some write throughput but removes the question entirely:
awaitconn.query(`SET checkpoint_threshold = '0KB'`);A clean shutdown looks like this:
awaitconn.query('CHECKPOINT');awaitconn.close();awaitdb.terminate();What happens when a tab is killed mid-transaction, and how to share one database between tabs, are covered in a follow-up post.
There is a second kind of durability to keep in mind, one that sits below DuckDB. OPFS is browser storage, not a hard guarantee. The browser can evict it when disk space runs low or when the origin has not been visited for a long time, and the user can clear it from the site's settings. Treat OPFS as a fast local cache for accelerating startup and persisting working state, not as your only copy of data you cannot lose. For durable storage, keep the source of truth somewhere stable and sync back to it: a DuckLake catalog, or plain files on object storage through s3:// paths.
Export
Users will want to move their data to another device, back it up, or open it with a different tool. DuckDB-Wasm itself cannot move files into or out of OPFS yet, but the database is a plain DuckDB file and the browser's OPFS API lets you read it back as bytes:
awaitconn.query('CHECKPOINT');constroot=awaitnavigator.storage.getDirectory();consthandle=awaitroot.getFileHandle('analytics.duckdb');constfile=awaithandle.getFile();// Offer as a download, upload to your backend, etc.consturl=URL.createObjectURL(file);Or export from SQL to Parquet:
COPYtransactionsTO'opfs://export/transactions.parquet'(FORMATparquet,COMPRESSIONzstd);Combined with DuckDB's Parquet support, this allows preparing and cleaning data in the browser before uploading it to a server. And because the on-disk format is standard, the reverse works too: ship a pre-built .duckdb file with your app, copy it into OPFS on first launch, and open it. Users get a local dataset without an import step.
Conclusion
Lack of persistence was the main limitation of DuckDB-Wasm for a long time. With OPFS, DuckDB-Wasm can open a database file in the browser, commit transactions to a WAL, checkpoint, and reopen the same database after a reload. Three things make it work well: run CHECKPOINT after each batch of writes rather than after every statement, give users a way to download the database file, and read the limitations listed in the DuckDB documentation before shipping: one handle per file, and renames from SQL only work between two already-registered OPFS files.
With this, a local-first application no longer needs a server, IndexedDB wrapper, or custom serialization to keep analytical data between sessions. Try it in your own application, and share what you build on GitHub or Discord.
这篇内容对你有用吗?
反馈只用于改善内容筛选,不等同于收藏