Skip to main content
The data engine transforms raw content into structured, AI-ready data. It manages the complete data lifecycle from file ingestion through preparation for training and retrieval workflows.

Data engine UI guide

Upload files, generate training datasets, and build vector stores through the SeekrFlow web interface.

Data engine SDK guide

Upload files, run ingestion, and create data jobs programmatically with the Python SDK.

Core capabilities

The data engine provides two primary functions:

Storage

Storage manages raw content through file ingestion and vector database creation, with ingestion insights that surface real-time processing status and diagnostics. Vector databases transform these files into searchable knowledge bases through chunking and embedding, with support for user-defined chunk metadata.

AI-ready data

AI-ready data generates and transforms datasets, converting raw content into training-ready formats. These jobs produce structured outputs optimized for model fine-tuning and alignment, including standard instruction datasets and context-grounded datasets.

Data workflow

The typical data engine workflow:
1
Upload raw content files to storage.
2
Create vector stores for retrieval applications.
3
Generate AI-ready datasets from selected files.
4
Use outputs for fine-tuning, agent knowledge bases, or evaluations.

Integration points

Data engine outputs integrate across SeekrFlow:
  • Fine-tuning – AI-ready datasets feed model training pipelines
  • Agents – Vector stores power FileSearch tool for knowledge retrieval
  • Evaluations – Structured datasets support model testing and validation
Last modified on November 19, 2026