Split on structure, not on characters

A fixed-size splitter has exactly one idea: count characters, cut, repeat. It doesn’t know it’s in the middle of a sentence. It doesn’t know the table it just bisected was a table. It cuts where the counter runs out, and where the counter runs out is unrelated to where meaning ends.

Your documents already contain boundaries that somebody chose deliberately. Headings, paragraphs, list items, clause numbers, function definitions, speaker turns. Those are semantic boundaries — they mark where one idea stops. A splitter that respects them produces chunks that are about something, and a splitter that ignores them produces chunks that end mid-clause.

What character splitting produces

The same markdown, cut at a fixed character count:

--- chunk 1 (chars 0–420) ---
## Configuring retries

The client retries failed requests automatically. Retry
behaviour is controlled by three settings.

### max_attempts

Total number of attempts including the first. Setting this to
1 disables retrying. The default is documented in the client
reference.

### backoff_
--- chunk 2 (chars 420–840) ---
factor

Multiplier applied to the delay between attempts. A factor of
2 doubles the wait each time.

### retry_on

List of status codes that trigger a retry. By default this
includes 429 and 5xx responses but not 4xx responses other
than 429, because those indicate a client error that retrying
will not fix.

Two things went wrong, and only one of them is obvious.

The obvious one: backoff_factor is split across the boundary. Chunk 1 ends with a dangling identifier fragment; chunk 2 opens with factor on its own line. Neither piece is searchable for “backoff factor” in any sensible way.

The subtle one, which costs more: chunk 2 has lost its heading. ## Configuring retries is in chunk 1. Retrieved on its own, chunk 2 is a paragraph about status codes with no indication that it concerns retry configuration. A query about “which errors does the client retry” has to match against text that never says the word “retry” in a way that establishes context.

That second failure is the general case. Character splitting doesn’t just cut through sentences; it severs content from the headings that give it meaning.

The same document, split structurally

--- chunk 1 ---
## Configuring retries

The client retries failed requests automatically. Retry
behaviour is controlled by three settings.

--- chunk 2 ---
## Configuring retries → max_attempts

Total number of attempts including the first. Setting this to
1 disables retrying. The default is documented in the client
reference.

--- chunk 3 ---
## Configuring retries → backoff_factor

Multiplier applied to the delay between attempts. A factor of
2 doubles the wait each time.

--- chunk 4 ---
## Configuring retries → retry_on

List of status codes that trigger a retry. By default this
includes 429 and 5xx responses but not 4xx responses other
than 429, because those indicate a client error that retrying
will not fix.

Every chunk is one complete idea. Every chunk carries its heading path, so each one is self-describing — chunk 4’s vector now encodes both “retry configuration” and “status codes,” which is what the query is looking for.

That heading prefix is doing a lot of work and it costs a handful of tokens. It’s part of a broader point about what has to travel with a chunk.

A boundary hierarchy

Not every document is markdown, but most have a hierarchy you can walk. The general procedure:

1. Identify the document’s natural units, largest to smallest. For markdown: heading levels, then paragraphs, then sentences. For HTML: sections and articles, then block elements. For code: modules, classes, functions. For contracts: articles, clauses, subclauses. For transcripts: speaker turns.
2. Split at the largest unit first. Take a whole section as a chunk.
3. If a unit exceeds your maximum, descend one level and split it into its children.
4. Repeat until every chunk fits. If you reach the bottom of the hierarchy and something still doesn’t fit — a single enormous paragraph — fall back to character splitting within that unit only.
5. Optionally merge upward. Consecutive units well under the minimum can be combined, provided they share a parent. Two adjacent one-sentence subsections belong together; a subsection from the end of one section and the start of the next do not.

Step 4 is the important one. Character splitting isn’t wrong — it’s the fallback, used on the smallest possible scope, rather than the primary mechanism applied to the whole document.

Step 5 is what stops structural splitting from producing a hundred two-line chunks out of a document with many short subsections, which is its most common failure.

Recursive character splitting is not this

The common recursive splitter — try to split on \n\n, then \n, then . , then characters — is frequently described as structural splitting. It isn’t quite, and the difference matters.

What it gets right: it prefers larger separators, so it usually avoids cutting mid-sentence. That’s a real improvement over naive counting and it’s why it’s the sensible default when you know nothing about your documents.

What it doesn’t do:

It doesn’t know what the separators mean. \n\n before a heading and \n\n between two paragraphs of the same subsection are the same character sequence to it, and it will cheerfully split a heading away from the text it introduces.

It doesn’t propagate context. No heading path, no section title, no parent reference. Each chunk is a bare span of text.

It’s still driven by the size target. Separators only determine where near the target to cut. If the target lands in the middle of a table, it’ll cut at the nearest newline inside the table.

So: recursive character splitting is a reasonable default for unknown input, and a poor choice for input whose format you control or can parse. If your corpus is markdown, parse the markdown.

What structural splitting breaks

Every strategy mangles something.

Wildly uneven chunk sizes. Real documents have a two-line introduction and a nine-page appendix under sibling headings. Structural splitting produces both as chunks. Uneven sizes make retrieval comparisons less meaningful — a long chunk has a diluted vector and a short one has a sharp one, and they’re competing on the same scale.

Documents with no structure. A scanned PDF converted to text, a plain-text export, a transcript with no speaker labels. There’s nothing to split on and you’re back to counting.

Structure that lies. Documents where headings were used for visual styling rather than hierarchy — a heading applied to a single bold line mid-paragraph, or a level-3 heading directly under a level-1 with nothing between. Parsing these faithfully produces nonsense. Worth sampling your corpus before trusting its structure.

Cross-boundary ideas. Some arguments genuinely span sections: a definition in section 2 that section 7 depends on. Structural splitting cuts them apart just as cleanly as character splitting does, because the split is real. This is not solvable at the chunking layer.

Cost. You now need a parser per format, and format-specific handling for tables, code and transcripts. Character splitting works on anything, which is the entirety of its appeal.

How to tell if it helped

Look at the chunks, first. This sounds too simple to mention, and almost nobody does it. Dump a random sample of fifty chunks and read them. Ask of each: could I tell what this is about, if I had never seen the document? If a chunk starts with “It should be noted that this also applies” or consists of three table rows, you have your answer, and you didn’t need a metric to get it.

Then measure. Build a query set with known-correct passages, index with both strategies, and compare whether the right passage comes back and whether it’s complete. Structural splitting usually helps most on documents with strong hierarchy and short subsections, and helps least on flowing prose — which is worth knowing before you build a parser.

The cheap first move, if you build nothing else: prepend the heading path to every chunk. It works with the splitter you already have, costs almost nothing, and fixes the severed-context failure that does most of the damage.