Data pipelines reference

Configuration fields, execution behavior, and limits for data pipelines. For an overview of how pipelines work, see Data pipelines overview.

Pipeline configuration fields

These are the underlying fields on the pipeline resource. The CLI flags and SDK parameters you use to create a pipeline each set one of these fields.

Cron schedule

The schedule field uses standard five-field cron syntax: minute hour day-of-month month day-of-week. All times are UTC.

The schedule determines both when the pipeline runs and the time range it queries. Each run processes the time window between the previous two schedule ticks.

For example, a pipeline with schedule 0 * * * * that triggers at 03:00 PM UTC processes data from 02:00 PM to 03:00 PM UTC. The time window is [start, end) (start inclusive, end exclusive).

Choose a schedule that matches how frequently you need updated summaries. Shorter intervals produce more granular summaries but create more pipeline sink documents.

Data source types

Pipelines support three data source types. You use them in two contexts: when creating a pipeline (the pipeline reads from this source) and when querying pipeline results (the query targets this source). Not every type is valid in both contexts.

The SDK constants for each type:

To query the output of another pipeline, use pipeline_sink in your query call alongside the source pipeline’s ID. See Query pipeline results.

Run statuses

SDK methods return the enum Status values. The viam datapipelines describe CLI command prints the friendly “CLI label” form.

If a run stays in STARTED for more than 10 minutes, it is automatically marked as FAILED and a new run is created for that time window.

Run fields

Each pipeline run record contains:

Backfill behavior

When enable_backfill is true:

• On pipeline creation, Viam processes historical time windows backward from the creation time to the earliest available data.
• When data syncs with a delay (machine was offline), the pipeline automatically reruns affected time windows to include the late-arriving data.
• Backfill processes in batches of up to 10 concurrent time windows with a 2-minute delay between batches.
• For standard data source, backfill may provision an Atlas Data Federation instance for faster historical queries.
• Backfill results replace any existing results for the same time window.

When enable_backfill is false:

• Each time window is processed exactly once.
• Late-arriving data is not incorporated into past summaries.

Backfill does not apply to windows missed while a pipeline was disabled. If you disable a pipeline for 3 hours and re-enable it, those 3 hours are not backfilled.

Pipeline sink

Each pipeline stores its output in a dedicated sink collection named sink-<pipeline-id>. Each result document includes metadata:

{
  "_viam_pipeline_run": {
    "id": "run-id",
    "interval": {
      "start": "2025-03-15T14:00:00.000Z",
      "end": "2025-03-15T15:00:00.000Z"
    },
    "organization_id": "org-id"
  },
  "location": "warehouse-a",
  "avg_temp": 23.5,
  "count": 3600
}

The _viam_pipeline_run field is added automatically. Your pipeline’s $project output fields appear alongside it.

To query the sink, use data source type pipeline_sink with the pipeline’s ID. See Query pipeline results.

Execution limits

Permissions

Only organization owners can create, modify, and delete data pipelines. Query access to pipeline results follows the same permissions as other data queries. See Permissions.