Improving AI_EXTRACT Accuracy on Dense, Multi-Section Financial Documents

POWERED BY SNOWFLAKE AI_EXTRACT

Author: Nikhil Singh (Kipi.ai)

Co-Author: Vivek Kanna Jayaprakash (Kipi.ai)

Co-Author: Ripu Jain (Snowflake)

The Business Challenge

Wealth management firms receive thousands of client financial documents in image format every month — account opening packages, consolidated statements, retirement plan summaries, and trust documents. Each contains multiple dense sections: account summaries, beneficiary designations, trade confirmations, fee schedules, and portfolio breakdowns.

Today, advisors and analysts manually review each image, identify sections, and transcribe key data into planning and CRM systems. The cost is significant: onboarding takes days instead of hours, advisors spend time on data entry instead of relationships, transcription errors create compliance exposure, and operations become the growth bottleneck.

Snowflake AI_EXTRACT was built precisely to solve this class of problem — and on straightforward, single-section documents it delivers exactly the accuracy firms need. The question this paper explores is how to get that same best-in-class performance on the dense, multi-section documents that dominate wealth management workflows.

AI_EXTRACT excels on simple, single-section documents. But wealth management documents are rarely simple — a typical consolidated statement contains six to ten distinct sections on a single page. When an entire multi-section document is processed in one pass, similar-looking fields — dollar amounts, percentages, dates, account numbers — compete across sections for the model’s attention. This creates a particularly subtle failure mode: the model may correctly recognize a field but associate it with the wrong section. For example, when multiple sections contain an “Account Number,” “Balance,” or similar field, a one-shot extraction can return a valid-looking value from a different section. The result on whole-page, single-pass extraction: 70–80% accuracy, with the remaining 20–30% of missed or misattributed fields triggering the same manual review firms are trying to eliminate.

The fix isn’t a different engine — it’s giving AI_EXTRACT a narrower, cleaner problem to solve, one section at a time.

Snowflake AI_EXTRACT Primer

At the center of this solution is Snowflake’s AI_EXTRACT function — a production-grade, in-platform extraction engine that converts unstructured document content directly into structured JSON. It is, quite simply, the most capable extraction engine available natively inside the data platform where wealth management firms already govern their most sensitive client data.

• Schema-driven output: define exactly which fields you need — names, dates, percentages, line items — and receive clean, typed JSON every time, with no brittle regex or custom parsers to maintain.
• Confidence scoring built in: every extracted value carries a confidence signal, enabling automated pass/fail thresholds without human review.
• Runs entirely inside Snowflake: no data leaves the platform, no external API calls, no credential management. Governance and compliance are native, not bolted on.
• Enterprise-grade accuracy: state-of-the-art extraction quality on the kinds of documents financial services firms handle every day, backed by Snowflake’s continuous model improvements.

The Solution: A Three-Step Framework Built on AI_EXTRACT

A reusable pattern that orchestrates AI vision, image processing, and Snowflake AI_EXTRACT — applicable to any multi-section financial document.

Step 1: Section Detection

AI identifies logical section boundaries and locations across the full document before any data extraction begins.

Step 2: Section Isolation

Each section is isolated into its own clean sub-image. Precise, repeatable, zero hallucination risk.

As an example of the input-output flow: every input image, named <AccountNumber>.PNG, is analyzed to identify its individual sections. Each detected section is then cropped out and saved as a separate image following the naming convention <AccountNumber>_<SectionAbbreviation>.PNG.

Input Stage

Output Stage

Phase 1 — LLM Vision (Section Identification)

What it does: This is the “AI brain” of the pipeline — it sends the document image to Snowflake’s AI_COMPLETE function with a carefully engineered prompt, and gets back a structured map of where each section lives on the page.

We evaluated multiple Claude models, and Claude Opus 4.7 performed the best, with a 100% success rate and the highest average confidence score of 95.33%, compared to Opus 4.6 (93.44%) and Sonnet 4 (90.36%).

How it works:

1. Constructs a prompt instructing the LLM to act as a document layout analyzer
2. Uses PROMPT() + TO_FILE() to pass the image directly to the model
3. The LLM returns a JSON array of detected sections, each with:

• title — the section heading text
• top_frac / bottom_frac — vertical position as a fraction of page height
• confidence — the model’s self-reported confidence score

4. On failure, it falls back to an equal-thirds split — graceful degradation instead of a hard crash

def _get_llm_section_positions(session, stage_name: str, file_path: str, model_name: str) -> list:
    """Use Cortex AI_COMPLETE with PROMPT() + TO_FILE() to identify section positions."""
    try:
        prompt_text = """
You are a precise document layout analyzer. Your task is to segment a financial document image {0} into its logical sections with exact horizontal boundaries.
INSTRUCTIONS:
1. Scan the image top-to-bottom and identify section boundaries using visual cues: bold/colored headings, horizontal rules that indicate a new section start.
2. A section STARTS at the top edge of its heading/title bar and ENDS where the next section heading/title bar begins (or at the bottom of the page for the last section).
3. Express all positions as fractions of total page height (0.0 = top pixel, 1.0 = bottom pixel).
4. Only report sections that are visually present in the image. Do NOT hallucinate sections that are not visible.

Below are the Sections:
 - ACCOUNT SUMMARY
 - BENEFICIARY DESIGNATION
 - PERIODIC TRADES SUMMARY
 - RETIREMENT PLAN OVERVIEW
 - FEE SCHEDULE & ADVISORY CHARGES

These are the Sections-Heading (For Sub Sections-headings are mentioned in Brackets)
Sub Headings Sections should be within the Section itself.

BOUNDARY RULES:
- If a section heading has a colored/shaded background bar, the top_frac starts at the top edge of that bar.
- The first sections top_frac should align with where the first section heading appears (skip any navigation/header chrome above it).

OUTPUT FORMAT:
Return ONLY a valid JSON array. No markdown, no explanation, no extra text. Each element must have exactly these four keys: title, top_frac, bottom_frac, confidence.
Example: [{{"title":"Account Summary","top_frac":0.05,"bottom_frac":0.25,"confidence":"confidence score - float value"}}]
VALIDATION CHECKS (apply before responding):
- Every top_frac < its own bottom_frac.
- Sections are ordered by top_frac ascending.
- No two sections overlap.
- Fractions are rounded to 3 decimal places.
- Only sections visually confirmed in the image are included."""
        # Escape single quotes in prompt for SQL string literal
        escaped_prompt = prompt_text.replace("'", "'")
        result = session.sql(f"""
            SELECT AI_COMPLETE(
                '{model_name}',
                PROMPT('{escaped_prompt}',
                       TO_FILE('@{stage_name}', '{file_path}'))
            ) AS result
        """).collect()
        if not result or not result[0]["RESULT"]:
            return []
        response = str(result[0]["RESULT"])
        # Strip markdown code fences if present (LLMs sometimes wrap JSON in ```)
        response = response.replace("```json", "").replace("```", "").strip()
        sections = json.loads(response)
        # Handle double-encoding (rare case where LLM returns a JSON string of JSON)
        if isinstance(sections, str):
            sections = json.loads(sections)
        # Validate: must be a list of dicts
        if isinstance(sections, list) and len(sections) > 0:
            if isinstance(sections[0], str):
                return []
            return sections

        return []

    except Exception as e:
        # Fallback: equal thirds so cropping still produces something usable
        return [
            {"title": "Top_Section", "top_frac": 0.0, "bottom_frac": 0.33, "description": "Top third"},
            {"title": "Middle_Section", "top_frac": 0.33, "bottom_frac": 0.66, "description": "Middle third"},
            {"title": "Bottom_Section", "top_frac": 0.66, "bottom_frac": 1.0, "description": "Bottom third"},
        ]

Phase 2 — Boundary Refinement

What it does: The LLM from Phase 1 gives approximate positions for each section — but these can be off by some pixels (px). This component uses pure-NumPy image processing to “snap” those rough boundaries to the nearest clean whitespace gap, so a cut never slices through actual text or content.

Two sub-components:

1. Sobel Filter — detects edges in the image (high values indicate content boundaries)
2. Boundary Refiner — scans near the LLM’s estimate to locate a clean whitespace row to cut along

def _refine_boundary(gray, edge_map, approx_y: int, boundary_type: str, search_range: int = 50) -> int:
    """
    Refine a boundary position using image analysis.

    For "top" boundaries: scans upward from the LLM estimate to find the last
    whitespace row before content begins (so we include the full heading).

    For "bottom" boundaries: scans downward to find the first whitespace row
    after content ends (so we don't cut off trailing text).

    A row is considered "whitespace" when:
      - Mean pixel intensity > 245 (nearly white)
      - Edge density < 0.02 (no significant content)
    """
    h = gray.shape[0]

    if boundary_type == "top":
        start = max(0, approx_y - search_range)
        end = min(h, approx_y + 10)
    else:
        start = max(0, approx_y - 10)
        end = min(h, approx_y + search_range)

    if start >= end:
        return approx_y

    # Row-wise analysis in search region
    region_means = np.mean(gray[start:end], axis=1)
    region_edge_density = np.mean(edge_map[start:end] > 0, axis=1)

    if boundary_type == "top":
        # Scan from bottom to top - find the last whitespace row before content
        for i in range(len(region_means) - 1, -1, -1):
            if region_means[i] > 245 and region_edge_density[i] < 0.02:
                return start + i
    else:
        # Scan from top to bottom - find the first whitespace row after content
        for i in range(len(region_means)):
            if region_means[i] > 245 and region_edge_density[i] < 0.02:
                return start + i

    return approx_y


def _sobel_filter(img, axis=0):
    """
    Pure-numpy Sobel filter - detects edges.

    axis=0: Detects horizontal edges (changes in vertical direction)
            Used here to find horizontal section dividers/boundaries.
    axis=1: Detects vertical edges (changes in horizontal direction)

    Implementation: Applies a 3x3 convolution kernel via array slicing,
    which is significantly faster than nested pixel loops.
    """
    if axis == 0:
        kernel = np.array([[-1, -2, -1],
                           [ 0,  0,  0],
                           [ 1,  2,  1]], dtype=float)
    else:
        kernel = np.array([[-1, 0, 1],
                           [-2, 0, 2],
                           [-1, 0, 1]], dtype=float)

    h, w = img.shape
    output = np.zeros((h, w), dtype=float)
    padded = np.pad(img.astype(float), 1, mode="edge")

    # Vectorized convolution using slicing (much faster than pixel loops)
    for ki in range(3):
        for kj in range(3):
            if kernel[ki, kj] != 0:
                output += kernel[ki, kj] * padded[ki:ki+h, kj:kj+w]

    return output

Step 3: Focused Extraction with AI_EXTRACT

Each isolated section is handed to Snowflake AI_EXTRACT independently, with targeted, section-specific field definitions. No cross-section confusion. Full model attention on a single task — this is where AI_EXTRACT does what it does best.

Why This Works

When AI_EXTRACT sees only a beneficiary table and is asked for names, relationships, and percentages, it cannot confuse a beneficiary percentage with a portfolio allocation from another section. The extraction becomes precise because the context is precise — a simpler problem yields better results, and it lets AI_EXTRACT operate at the accuracy it was built to deliver.

Utilities & Observability

Behind the scenes, there’s also a layer of code that handles all the “plumbing” — the parts that talk to Snowflake but don’t actually do any AI work. This includes:

• Reading images from stages — pulling input files
• Writing cropped images to stages — saving output back to the stage.
• Generating presigned URLs — for downstream/external access to results
• File operations — cleaning filenames, extracting account numbers, matching section titles by pattern
• Logging every extraction attempt — for observability and auditing

Business Impact

• Client Onboarding: Accuracy jumps from 70% to 95%+ (whole-image processing vs section-based processing). Onboarding compresses from days to hours.
• Retirement Plan Analysis: Contribution limits, vesting schedules, and distribution options extracted correctly the first time — no follow-up calls to verify missing data.
• Beneficiary Processing: Names, relationships, and percentage allocations extracted with precision. A misread beneficiary percentage isn’t just a data issue — it’s a fiduciary liability.
• Trade Reconciliation: Complete transaction data for automated portfolio reconciliation. Operations teams handle exceptions, not routine data entry.
• Financial Strategy Development: Complete, accurate client financial pictures enable faster, higher-quality personalized advice.

Comparison: Single-Pass vs. Intelligent Section Extraction

Accelerating Development with CoCo Skills

Building and operationalizing this pipeline is faster with two purpose-built skills available in Snowflake CoCo:

• Document Intelligence: Invoke AI_EXTRACT, AI_PARSE_DOCUMENT, AI_CLASSIFY, or AI_COMPLETE on any file or batch already on a Snowflake stage in a single guided interaction — no boilerplate SQL required, just describe what you need extracted and the skill handles syntax, validation, and execution.
• AI Functions Pipeline Builder: Turn a plain-language description into a fully incremental, Snowflake-native pipeline (stream → task → dynamic tables) that automatically processes new documents as they land on a stage — keeping AI_EXTRACT outputs perpetually fresh without manual reruns.

Together, these skills collapse what would be days of pipeline engineering into a conversational workflow — from prototype to production in minutes, all on top of AI_EXTRACT.

Improving AI_EXTRACT Accuracy on Dense, Multi-Section Financial Documents was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.