Data Pipeline

Data Pipeline Overview

The autonomous data pipeline — collect, index, curate, and quality-score data for labeling.

Overview

The data pipeline service automates data collection and preparation. Configure a source, run a crawl job to collect content, then index it with embeddings and entity linking. A curation pipeline applies quality scoring so only high-value records flow downstream. All endpoints live under /api/v1/data-pipeline.

Sources & crawl jobs

GET/api/v1/data-pipeline/sourcesList configured data sources.
POST/api/v1/data-pipeline/sourcesConfigure a source (Web, API, S3, GCS, Database, …).
POST/api/v1/data-pipeline/crawl-jobsStart a crawl job.
GET/api/v1/data-pipeline/crawl-jobs/{id}Monitor a crawl job.

Indexing & search

GET/api/v1/data-pipeline/index-configsList indexing configurations.
GET/api/v1/data-pipeline/indexed-recordsList indexed records.
POST/api/v1/data-pipeline/vector-searchRun vector similarity search.
POST/api/v1/data-pipeline/keyword-searchRun keyword search with quality signals.

Curation & analytics

GET/api/v1/data-pipeline/curation-pipelinesList quality-based curation pipelines.
POST/api/v1/data-pipeline/curation-pipelinesCreate a curation pipeline.
GET/api/v1/data-pipeline/analyticsRead quality, usage, and flywheel metrics.

See every endpoint and schema — including the Firecrawl-compatible scrape API — in the Data Pipeline API reference.