Datasets | everyn

Datasets are the input layer for Everyn work. A dataset is not just a file name; it is a durable source snapshot with stable row identity, column metadata, sample/profile context, and a link to the runs that use it.

CSV intake uses direct uploads. Everyn accepts the upload, processes it asynchronously, validates and normalizes the source, and exposes a completed dataset only after the input is ready to use for runs.

Intake and identity

Intake lifecycle

Step	What happens	What to preserve
Create upload	Client sends a CSV with `sourceName`, `sourceType`, metadata, and an idempotency key.	`uploadId`, `sourceName`, request ID.
Processing	Everyn validates stored upload metadata and parses the CSV.	Upload status and any failure details.
Completed	A durable dataset is registered from the normalized snapshot.	`datasetId`, row count, columns, sample/profile.
Failed	The upload remains inspectable with public-safe failure metadata.	`failure.code`, `failure.docsUrl`, field errors.
Retry	Retryable failed uploads can be reprocessed while the source exists.	Original `uploadId` and retry outcome.

Dataset identity

Important identifiers and fields:

Field	Why it matters
`datasetId`	Identifies the durable dataset used by future runs.
`datasetRowId`	Stable row identity across scopes, retries, outputs, failures, and exports.
`uploadId`	Identifies the intake attempt and its failure or retry state.
`sourceName`	Connects an error to the original file or source system.
`columnMap`	Connects source columns to normalized snapshot columns.
`failure.code`	Tells the client whether to fix input, retry, or create a new upload.
`failure.docsUrl`	Links directly to recovery guidance.

CSV expectations

CSV uploads use direct multipart intake:

The direct multipart file limit is 25 MiB.
Dataset creation is asynchronous after upload acceptance.
Upload progress is expressed through resource state and byte fields, not progressPercent.
Failed uploads are still readable resources.

For endpoint mechanics, use Dataset uploads and Dataset uploads API.

Before creating a job spec

Inspect enough of the dataset to avoid writing a generic prompt:

source name and source type
row count
column names and example values
missing or malformed values
whether the intended output should add columns or create child rows
any fields that should never be sent to external systems or exported

Recovery paths

Error	Meaning	Next step
`csv_parse_failed`	The CSV could not be parsed into a valid dataset.	Fix the file and create or retry the upload as directed.
`transient_intake_failure`	Intake failed for a retryable system reason.	Retry if the upload source still exists.
`upload_source_expired`	The stored source needed for retry is gone.	Create a new upload.

Common mistakes

Treating upload acceptance as completed dataset creation.
Starting a run before checking that datasetId is present.
Replacing parse details with a generic "the upload failed" message.
Assuming row indexes are enough identity across retry lineage.
Editing the original local CSV and expecting an existing dataset snapshot to change.