Ingestion, Chunking, and Metadata

Key jargon

Term	Plain-language meaning
Ingestion	Converting source material into a normalized, indexable representation.
Chunk	A bounded unit of content stored and retrieved as a candidate.
Overlap	Repeated content across adjacent chunks used to preserve boundary context.
Metadata	Structured attributes such as source, title, date, access rule, and section.

Key concepts

Chunking is an information-design decision tied to questions users ask, not merely a character count.
Keep stable source and location metadata so every retrieved passage can be traced and filtered.

Concept map

flowchart LR
    A["Acquire source with provenance"] --> B["Parse and normalize"]
    B --> C["Chunk with metadata"]
    C --> D["Validate then index"]

Ingestion contract

Capture source URI/path, owner, version/date, content type, parser, checksum, access policy, sensitivity, language, and extraction warnings.

Chunk boundaries should follow information structure where possible: headings, paragraphs, tables, code blocks, or semantic units. Fixed-size overlap is a baseline, not a universal best practice.

Failure modes

OCR silently changes numbers or names.
Tables lose row/column relationships.
Chunk overlap creates duplicate evidence.
A chunk loses title, date, or access policy.
Deleted or superseded documents remain retrievable.

Exercise

Chunk one Markdown guide three ways: fixed characters, paragraphs, and headings. Test five questions and compare whether each returned chunk contains a complete answer and provenance.

Checklist

Keep the original artifact.
Version parser and chunk strategy.
Attach stable document and chunk IDs.
Carry authorization metadata into retrieval.
Re-index deletions and updates deterministically.