Chunking is product design, not a utility call
Default 512-token splits destroy tables, break FAQ pairs, and separate headings from body text. Users ask "What is the refund window for annual plans?" — if the chunking split policy header from table rows, retrieval returns garbage and the LLM fills gaps with fiction.
Start from document types: contracts need clause-aware splits, menus need item-level rows, support macros need one article per chunk. Inspect failures visually in a chunk viewer before tuning embeddings.
Tactics that move retrieval metrics
Use structure-aware parsers for PDFs (headings, lists, tables). Add overlap only where continuity matters — long policies, not isolated Q&A. Attach metadata: source file, page, section, effective date, outlet ID for multi-location restaurants.
Hybrid search (dense + keyword) rescues exact SKU codes, drug names, and legal citations that pure embeddings miss. Re-rank top-k with a cross-encoder or lightweight scorer before passing context to the LLM.
Evaluate chunk changes like code changes
When you re-chunk, rerun the golden question set. Track MRR@k and human graded faithfulness — not just latency. Version your ingestion pipeline so you can roll back a bad splitter without re-embedding everything blindly.
Sabrixa tunes chunking per vertical during consulting engagements and applies sensible defaults in the WhatsApp product so SMB uploads work without a data engineering team.