Data engine UI guide
Upload files, generate training datasets, and build vector stores through the SeekrFlow web interface.
Data engine SDK guide
Upload files, run ingestion, and create data jobs programmatically with the Python SDK.
Core capabilities
The data engine provides two primary functions:Storage
Storage manages raw content through file ingestion and vector database creation, with ingestion insights that surface real-time processing status and diagnostics. Vector databases transform these files into searchable knowledge bases through chunking and embedding, with support for user-defined chunk metadata.
AI-ready data
AI-ready data generates and transforms datasets, converting raw content into training-ready formats. These jobs produce structured outputs optimized for model fine-tuning and alignment, including standard instruction datasets and context-grounded datasets.
Data workflow
The typical data engine workflow:1
Upload raw content files to storage.
2
Create vector stores for retrieval applications.
3
Generate AI-ready datasets from selected files.
4
Use outputs for fine-tuning, agent knowledge bases, or evaluations.
Integration points
Data engine outputs integrate across SeekrFlow:- Fine-tuning – AI-ready datasets feed model training pipelines
- Agents – Vector stores power FileSearch tool for knowledge retrieval
- Evaluations – Structured datasets support model testing and validation