Ingestion, Chunking, and Metadata
Ingestion, Chunking, and Metadata
Fact-check status
Key jargon
| Term | Plain-language meaning |
|---|---|
| Ingestion | Converting source material into a normalized, indexable representation. S1 |
| Chunk | A bounded unit of content stored and retrieved as a candidate. S1 |
| Overlap | Repeated content across adjacent chunks used to preserve boundary context. S1 |
| Metadata | Structured attributes such as source, title, date, access rule, and section. S1 |
Key concepts
- Chunking is an information-design decision tied to questions users ask, not merely a character count. S1
- Keep stable source and location metadata so every retrieved passage can be traced and filtered. S1
Concept map
flowchart LR
A["Acquire source with provenance"] --> B["Parse and normalize"]
B --> C["Chunk with metadata"]
C --> D["Validate then index"]Diagram semantics verified against S1 and S2 on 2026-09-02.
Ingestion contract
Capture source URI/path, owner, version/date, content type, parser, checksum, access policy, sensitivity, language, and extraction warnings. S1
Chunk boundaries should follow information structure where possible: headings, paragraphs, tables, code blocks, or semantic units. Fixed-size overlap is a baseline, not a universal best practice. S1
Failure modes
- OCR silently changes numbers or names. S1
- Tables lose row/column relationships. S1
- Chunk overlap creates duplicate evidence. S1
- A chunk loses title, date, or access policy. S1
- Deleted or superseded documents remain retrievable. S1
Exercise
Chunk one Markdown guide three ways: fixed characters, paragraphs, and headings. Test five questions and compare whether each returned chunk contains a complete answer and provenance.