Data Pipelines

Typed, run-scoped DAGs that combine deterministic code, LLM jobs, agents, decisions, maps and sub-pipelines over a validated information graph.

Pipelines are opt-in

Add a pipelines: block to agents.yaml to define one or more pipelines. A config without that block behaves exactly as before — no registry, graph, tools, or validation overhead.

A data pipeline is a named, typed DAG of jobs that runs over a run-scoped information graph. It is the structured counterpart to a chat turn: instead of asking a supervisor to choose agents turn by turn, you declare the steps, the data each step produces, and the shape of the final result.

Pipelines are built for repeatable work that mixes deterministic code, LLM calls and agents over data that must be validated. The same definition can be invoked from Python, HTTP, the CLI, an agent tool, or a Pollen/Bloom trigger. Pipeline runs never create chat rows and never use a LangGraph checkpointer.

When to use a pipeline#

Use a pipeline when you need one or more of these:

  • A typed contract — input, intermediate objects and output are validated against named shapes, not passed around as prose.
  • A repeatable sequence — deterministic code, an LLM call, a decision, a fan-out and an agent run in a declared order.
  • An inspectable artefact — job statuses, durations, object/relation counts and the final typed output are returned with every run.
  • A schedulable or event-driven unit — the same pipeline can be triggered by a webhook, a cron schedule or a chat tool.
  • Isolation from chat — a pipeline does not write chat history, does not read a chat checkpoint and does not need a chat session at all.

Keep using the chat supervisor for open-ended dialogue and routing. Use a pipeline when the work has a stable shape.

A pipeline in YAML#

The definition lives beside the agents in agents.yaml:

pipelines:
  enabled: true
  defaults:
    model: ollama/llama3.2

  shapes:
    Request:
      fields:
        customer_id:
          type: string
          required: true
    Report:
      fields:
        headline:
          type: string
          required: true
        risk:
          type: string

  definitions:
    customer_health:
      description: "Assess a customer and produce a report."
      input:
        shape: Request
      jobs:
        load:
          kind: deterministic
          uses: examples.data_pipeline.steps.load_account
        assess:
          kind: llm
          prompt: |
            Assess customer {{ input.customer_id }}.
            Account: {{ jobs.load.output }}
          output:
            shape: Report
          depends_on: [load]
      output:
        shape: Report
        from: "jobs.assess.output"
      interpretation:
        mode: template
        template: "{{ output.headline }}"

The example above has two jobs. The first is trusted operator code; the second renders a prompt from the mapping context, asks the configured chat model for a structured Report, and the pipeline projects that report as its typed output.

The run model#

input ──► validate shape ──► run session ──► job 1 ──► job 2 ──► output
                               │             │          │
                               ├─ information graph (objects + relations)
                               ├─ job results + usage
                               └─ limits, deadline, events

Five concepts explain almost everything:

ConceptMeaning
PipelineA named DAG declared under pipelines.definitions.
JobOne node. Its kind selects the step implementation.
RunOne execution with an immutable run_id, validated input and an isolated graph.
ShapeA named, versioned schema used for input, output and graph writes.
Object / relationTyped nodes and edges in the run's information graph.

A job declares what it may write. Writes are applied atomically after the job returns, so a failed job cannot leave half-written objects behind. Objects and relations carry revisions and provenance; the graph is serialized only when the caller asks for it and stays within a size cap.

Shapes and typed writes#

Shapes can be declared with fields or imported from a Pydantic model or dataclass:

shapes:
  Account:
    fields:
      id: { type: string, required: true }
      tier: { type: string }
  Score:
    class: my_pkg.shapes.Score

relations:
  has_usage:
    from: [Account]
    to: [Usage]

A job writes an object with a declaration such as:

writes:
  - id: "account:{{ input.customer_id }}"
    shape: Account
    as: account
links:
  - type: has_usage
    from: "account:{{ input.customer_id }}"
    to: "usage:{{ input.customer_id }}:north"

The write id and relation endpoints are templates. The written value is validated against the shape; a job may only write relation types declared in its own links allowlist.

Job kinds#

KindWhat it does
deterministicRuns trusted operator code resolved from a dotted path. Sync functions run off the event loop; async functions are awaited.
llmRenders a prompt, optionally retrieves RAG context and built-in tools, and returns a shape-validated structured output with one repair attempt.
agentRuns one top-level leaf agent through the same executor as chat: guardrails, model routing, MCP skills and mini-agents are preserved.
decisionAsks a configured decision model to choose one of the declared labels, records confidence, and applies the low-confidence policy.
mapFans out over a list, runs a template job per mapping item with bounded concurrency, and joins the results.
subpipelineRuns another pipeline as a child run with bounded depth and optional graph merge.
emitRenders an identity-only signal envelope and hands it to the configured emitter; it never persists the envelope or includes the bearer token.
customLoads an OrchidPipelineStep subclass from an operator-configured dotted path.

Every job shares the same policy surface: depends_on, when, writes, links, retry, timeout_seconds, on_error and metadata.

v1 scheduling note

depends_on wires scheduling edges, not an all-predecessor barrier. In v1 a job with multiple independent predecessors can start when the first predecessor triggers it. For a strict sequence, chain jobs with one predecessor each. join: any is the supported convergence point for mutually exclusive decision branches.

Mapping context#

Prompts, gates, write ids and output projections evaluate against one documented context:

KeyContents
inputThe validated run input.
runRun identity: run.id, pipeline, mode, timestamps.
jobsPer-job results: jobs.job_id.output, .status, .kind, .error.
objectsEvery graph object keyed by id.
typesObjects grouped by type.
aliasesThe latest object written under each alias.
summaryThe interpretation text, when one already exists.

{{ ... }} renders a value; a bare string documented as JMESPath is evaluated directly. Missing paths fail the job instead of silently rendering an empty string. Map items add their own binding plus item, index and parent_job.

Results and interpretation#

Every run returns an OrchidPipelineResult:

FieldMeaning
run_id, pipelineRun identity and definition name.
statussucceeded, partial, failed or cancelled.
outputThe typed output, or null when projection failed.
summary, summary_sourceHuman-readable summary and how it was produced.
jobsPer-job status, output, duration, LLM/token usage, attempts and errors.
graph_statsObject/relation counts, type counts, bytes, LLM calls and errors.
graphOptional serialized snapshot, capped and opt-in.

Summary modes are none, template, llm and caller. caller is the deterministic digest used by chat-mode runs; template is useful when a pipeline has a natural headline; llm adds one bounded summarisation call.

Limits#

Defaults live under pipelines.limits; a definition may tighten them.

LimitDefaultProtects
max_jobs100Declared jobs
max_objects10,000Graph objects
max_relations50,000Graph relations
max_object_bytes1,000,000One serialized object
max_graph_bytes20,000,000Graph snapshot
max_llm_calls50LLM calls per run
max_prompt_chars32,000Rendered prompt
max_context_chars8,000Retrieved context
max_map_items100Items in one map
max_parallel8Concurrent jobs/items
max_subpipeline_depth3Nested child runs

A run deadline covers the whole graph; per-job timeouts apply per attempt. Cancellation is terminal for the remaining jobs.

Invoking a pipeline#

Python#

from orchid_ai import Orchid

async with Orchid.from_config_path("orchid.yml") as orchid:
    result = await orchid.run_pipeline(
        "customer_health",
        {"customer_id": "C-100"},
        user_id="alice",
        tenant_id="acme",
        include_graph=True,
    )
    print(result.status, result.output)

stream_pipeline(...) yields pipeline.started, pipeline.job.* and pipeline.finished events; the final event carries the result.

HTTP#

The API exposes a stable pipeline surface:

MethodPathPurpose
GET/pipelinesList configured pipelines
GET/pipelines/{name}Sanitized definition detail
POST/pipelines/{name}/validateValidate input without running
POST/pipelines/{name}/runsSync run (mode: sync) or Bloom launch (mode: async)
POST/pipelines/{name}/runs/streamSSE event stream for a sync run
curl -sS -X POST http://localhost:8000/pipelines/customer_health/runs \
  -H 'Authorization: Bearer dev-token' \
  -H 'Content-Type: application/json' \
  -d '{"input": {"customer_id": "C-100"}, "include_graph": true}'

The response carries status, output, summary, jobs and graph_stats. Invalid input is a 422 with pipeline_input_schema_invalid; an unknown pipeline is 404; a graph over the cap is 413. Async runs return 202 with a status_url under /jobs/{trigger_id}/runs.

CLI#

orchid pipeline list --config orchid.yml
orchid pipeline show customer_health --config orchid.yml
orchid pipeline validate customer_health --config orchid.yml \
  --input-json '{"customer_id":"C-100"}'
orchid pipeline run customer_health --config orchid.yml \
  --input-json '{"customer_id":"C-100"}' --output json

list, show and validate are offline: they do not start models, MCP servers or tools. run uses the normal facade bootstrap and per-user MCP warm-up. Exit codes are 0 success, 1 internal error, 2 validation, 3 failed/partial, 4 timeout and 5 not found.

Agent tool#

A pipeline with expose.as_tool: true becomes a built-in tool. Registration alone does not grant access: the operator must list the tool in the agent's tools:. The tool returns a bounded structured result plus the summary, and the agent writes the user-facing answer.

Pollen + Bloom#

A trigger can target a pipeline with emits.pipeline:

events:
  triggers:
    - id: customer-health-request
      "on":
        signal: pipeline.customer_health.requested
      emits:
        pipeline: customer_health
        input_from: payload
        identity: { mode: act_as_user, user_id_from: signal.user_id }
        respect_chat_binding: true

The dispatcher creates a pipeline JobRun; the runner materialises auth, executes the pipeline and stores the result. With respect_chat_binding: true and a writable chat, the summary is appended to the bound chat with metadata.origin = "bloom". Schedules produce synthetic cron signals on the same path. Read Pollen + Bloom for the event layer.

Extending pipelines#

The extension surface is intentionally operator-only:

  • Deterministic functions via uses: my_pkg.steps.fn.
  • Custom step classes via kind: custom + class: my_pkg.steps.MyStep.
  • Registered kinds via register_pipeline_step(kind, factory) for programmatic/plugin pipelines.
  • Class-based shapes via shapes.MyShape.class: my_pkg.shapes.MyShape.

A custom step is trusted code. It runs with the run's auth, model, reader and tool caller; never build uses/class paths from user input, and never put a bearer token in an output, log or signal. The full extension contract is in the Pipeline extension guide.

Runnable examples#

  • Data Pipeline — the direct example: deterministic, LLM, agent, decision, map, sub-pipeline and emit jobs with a typed customer-health report.
  • Event-Driven Data Pipeline — a bearer-authenticated webhook and a cron schedule start the same pipeline; a chat binding posts the summary back to the chat.

Both examples run with a deterministic model in their e2e suites — no API key, no network and no Ollama required for tests.