The Chunking Problem

Metadata is half the chunk

The text of a chunk is the part everyone thinks about. The fields attached to it decide whether the chunk can be filtered, dated, cited, or updated — and whether, when it comes back, anyone can tell where it came from.

Metadata is also the cheapest quality improvement available in a retrieval pipeline, because most of it is already sitting in the document you’re splitting and gets thrown away at split time.

The chunk that can’t defend itself

Here is a chunk from a real-shaped corpus, as most pipelines store it:

{
  "text": "This does not apply to accounts on the legacy plan.
           Those accounts follow the terms in effect at the
           time of signup."
}

Everything is wrong with this and none of it is the text’s fault.

What doesn’t apply? Which document is this? Is this the current policy or a version from three years ago that’s still in the corpus? If a user asks about legacy plans and gets this back, what does the system cite? If the source document is updated, which chunk does the pipeline replace?

Now the same text with its context preserved:

{
  "text": "This does not apply to accounts on the legacy plan.
           Those accounts follow the terms in effect at the
           time of signup.",
  "doc_id": "billing-policy",
  "doc_title": "Billing and Refund Policy",
  "heading_path": ["Refunds", "Exceptions"],
  "source_url": "/policies/billing/",
  "effective_date": "2026-04-01",
  "doc_version": 7,
  "position": 14,
  "prev_chunk_id": "billing-policy:13",
  "next_chunk_id": "billing-policy:15",
  "audience": "customer",
  "language": "en"
}

Same text. Now it can be filtered by date, restricted by audience, cited with a link, expanded to its neighbours, and replaced when version 8 lands.

The fields worth having

Identity and lineage — doc_id, chunk_id, position, doc_version. Without a stable document identifier you cannot delete a document’s chunks when it’s removed, and stale content accumulates silently. Without position you cannot reassemble or expand. These are not optional; a pipeline lacking them has no update story.

Provenance — source_url, doc_title, author, system_of_origin. Needed for citation, and needed for the moment when someone asks “where did the model get that” and the honest answer must not be “somewhere in the corpus.”

Structural context — heading_path, section_title, page_number, part_of (table, code block, footnote). The heading path is the highest-value field on this list, for reasons below.

Time — created_at, updated_at, effective_date, expires_at. Corpora rot. A retriever with no concept of time will happily return the 2021 policy alongside the 2026 one, both scoring well, and the model has no way to prefer the current one unless the date is visible.

Access and scope — tenant_id, audience, classification, region. If your system serves more than one group of users, this is a security boundary, and it belongs on the chunk rather than being inferred at query time.

Type — doc_type (policy / reference / tutorial / transcript), content_type (prose / table / code). Enables type-specific handling downstream, and lets you diagnose whether one document class is retrieving badly.

Embedded versus attached

The distinction that most affects quality: some metadata should be inside the text you embed, and some should only be a filterable field.

Embed it when it changes what the chunk is about:

[Billing and Refund Policy → Refunds → Exceptions]

This does not apply to accounts on the legacy plan. Those
accounts follow the terms in effect at the time of signup.

That prefix costs a handful of tokens and transforms the vector. The chunk now encodes “refunds” and “exceptions” and “billing policy” — none of which the bare text contains. A query about refund exceptions for legacy accounts can now find it. Without the prefix it’s a floating sentence about something unspecified.

Rules of thumb for what to embed:

  • Embed: document title, heading path, section title, document type where it’s semantically meaningful (“API reference”, “meeting transcript”).
  • Attach only: IDs, versions, URLs, timestamps, tenant identifiers, access levels.

Why keep the second group out: they’re noise in the vector space. A UUID or an ISO timestamp contributes nothing a query will ever match against, fragments badly into tokens, and dilutes the signal from the actual content. Do not embed your chunk_id. It happens more often than you’d expect, usually by serialising the whole record to JSON and embedding that.

Dates are the interesting edge case. effective_date: 2026-04-01 as a field enables filtering and sorting. The phrase “effective April 2026” in the text lets a query mentioning 2026 match it. If dates matter to your users’ questions, do both — but write the embedded one in natural language, not ISO format.

Filtering is where it pays off

Metadata’s second job: narrowing the candidate set before or during search.

query:  "what's the current refund window?"
filter: doc_type = "policy"
    AND effective_date <= today
    AND (expires_at IS NULL OR expires_at > today)
    AND audience IN ("customer", "public")

This removes superseded policies, internal-only documents, and everything that isn’t a policy, before similarity is ever considered. On a corpus with several years of accumulated versions, that filter does more for answer quality than any amount of chunk-size tuning — because the failure it prevents isn’t “retrieved something irrelevant,” it’s “retrieved something that used to be true.”

Two practical notes. Filtering interacts with how your index is built, and aggressive filters can degrade approximate search or return fewer results than requested; that’s an index behaviour question rather than a chunking one, but it’s worth knowing before you design a filter-heavy schema. And a filter on a field that’s missing from half your chunks will silently exclude them — which is an argument for making required fields actually required at ingest.

Generated metadata

Beyond what the document already carries, you can produce more at index time. Diminishing returns, in this order:

A generated summary or title for the chunk, embedded with it. Genuinely useful for long or poorly-structured chunks. Costs a model call per chunk at index time.

Extracted entities — people, products, dates, identifiers mentioned in the chunk. Useful when your queries are entity-shaped (“what did we agree with Contoso”). Store as a filterable list.

Hypothetical questions the chunk answers, embedded alongside or instead of the text. The reasoning is that queries are questions and chunks are statements, and matching question-to-question can work better than question-to-statement. Real technique, real cost, and it makes your index dependent on a generation step you’ll need to re-run.

All three share a hazard: generated metadata can be wrong, and a wrong generated summary is worse than none, because it distorts the vector confidently. Sample the output before trusting it across a corpus, and keep the generation model and prompt versioned so you know what produced what.

What this means for updates

The field nobody adds until they need it: a content hash per chunk.

When a document changes, the naive approach deletes all its chunks and re-indexes. That re-embeds everything, including the 95% that didn’t change, and it churns your index. With a hash you can compare new chunks against stored ones and only touch what differs.

The catch: if your splitter is position-dependent, inserting one sentence near the top shifts every subsequent boundary and every hash changes. Another argument for structural splitting — a chunk that’s “the section titled Exceptions” is stable across edits elsewhere in the document in a way that “characters 4000–5000” is not.

Minimum viable set

If you’re adding metadata to an existing pipeline and want the order that pays fastest:

  1. heading_path, embedded as a prefix. Biggest single quality gain, smallest effort.
  2. doc_id and position. Makes updates and neighbour-expansion possible at all.
  3. source_url and doc_title. Citation becomes real.
  4. Dates. Stops the corpus from answering with its own history.
  5. Access fields, if you have more than one audience — and treat these as correctness, not enrichment.

Everything after that is refinement. The first item alone is worth more than most chunk-size tuning, and it’s a string concatenation.

metadatastrategy