Choose a chunking method

The chunking_method parameter on create_ingestion_job controls how SeekrFlow segments your content into chunks before embedding. Choose the method that best fits your documents.

Markdown chunking (default)

Intelligent structure-aware chunking that automatically detects logical content breaks:

Respects document structure and heading hierarchies
Keeps related content together (headings with their content)
Preserves tables with their headers
Groups small sections to optimize chunk sizes

Semantic chunking

Call with chunking_method="semantic" to enable meaning-aware segmentation. This method detects sentence boundaries, embeds paragraphs, and groups them by meaning using hierarchical clustering, rather than relying on document position:

Searches for topic shifts instead of raw heading boundaries to keep tightly related sentences together
Automatically merges short paragraphs or bullets when they express the same idea
Honors document structure when it provides strong signals but can span across headings if the semantics match
Applies token_count and overlap_tokens as safety caps, splitting only when a semantic chunk would exceed those limits

When to use it:

Long narrative content (wikis, blogs, requirements) where sections do not follow strict Markdown hierarchy
Mixed-format documents where context spans multiple small headings or callouts

No additional markup is required. Tune token_count and overlap_tokens to control chunk granularity.

Sliding window chunking

Call with chunking_method="sliding" to split content into fixed-size overlapping windows. This method has no structural awareness, which makes it fast and predictable. Use it for plain-text or homogeneous content that lacks clear document structure. To force chunk boundaries at specific points, insert ---DOCUMENT_BREAK--- markers in your Markdown:

# Section 1

This content will be in one chunk...

---DOCUMENT_BREAK---

# Section 2

This content will be in a separate chunk...

---DOCUMENT_BREAK---

# Section 3

This starts another chunk...

Set token_count (for example, 1000) and overlap_tokens (for example, 100) in the same create_ingestion_job call to control chunk size and overlap. Document break markers apply only to sliding window chunking. With markdown or semantic chunking, a marker does not create a boundary, and the literal marker text stays in the chunk. Remove the markers from your Markdown before you ingest it with either of those methods.

If the content between two document break markers exceeds token_count, the sliding window splits it into multiple chunks.

Add per-chunk metadata

With sliding window chunking, you can attach different metadata to different chunks by embedding a metadata block inside a section. Place the block between ---CHUNK_META_START--- and ---CHUNK_META_END---, with a single line of JSON in between. The block is scoped to the section it appears in and is removed from the text before indexing and never becomes part of the chunk content.

# Introduction

---CHUNK_META_START---
{"author": "Jane Smith", "source": "chapter_1", "classification": "public", "year": 2024}
---CHUNK_META_END---

This is the first chunk of the document.

---DOCUMENT_BREAK---

# Conclusion

This chunk has no metadata block and falls back to the job-level metadata from the ingestion request.

Per-chunk metadata follows the same metadata rules as job-level metadata, but a block that breaks them is handled differently. An ingestion request with invalid metadata is rejected outright, while inside a block an invalid field is dropped with a warning and the rest of the chunk still ingests. Per-chunk metadata behaves as follows:

It is supported only with sliding window chunking.
A section’s block replaces the job-level metadata for that chunk. The two are not merged.
A block applies to chunks only. It does not change the metadata on a file record, which keeps the job-level values.
A section with no block inherits the job-level metadata from the ingestion request, or no metadata if the request supplied none.
When a section is longer than token_count and splits into several chunks, every chunk from that section carries the section’s metadata.
When a section contains more than one block, the last block applies. Use a single block per section.

Ingestion does not reject metadata blocks that appear with markdown or semantic chunking. The block is not applied, and its marker lines and JSON remain in the chunk text, where they become part of the embedded content. Use these markers only with chunking_method="sliding".

​Markdown chunking (default)

​Semantic chunking

​Sliding window chunking

​Add per-chunk metadata

Markdown chunking (default)

Semantic chunking

Sliding window chunking

Add per-chunk metadata