Event-Driven Data Pipeline
Bloom-triggered typed pipeline: bearer-authenticated webhook, cron schedule, SQLite event storage and chat-bound results.
What this demonstrates#
This example runs the same customer-health pipeline as the direct example, but starts it from Pollen signals instead of a caller:
- a bearer-authenticated webhook starts a per-user Bloom run and, when the signal carries a chat binding, posts the pipeline summary back to that chat;
- a cron schedule starts a service-account run every morning.
It uses SQLite for chat and event storage, the HTTP ingestion producer, the
scheduler producer, act_as_user identity minting, actor/tenant visibility
and a deterministic decision provider. No frontend, MCP gateway or Qdrant is
required.
Run it#
From workspace-py:
make demo-data_pipeline_events # via script/start_data_pipeline_events.sh
make run-data_pipeline_events # docker compose up --buildOr run the example's own compose file:
cd examples/data_pipeline_events
cp .env.example .env
docker compose up --buildThe API listens on http://localhost:8080.
Send a request signal#
curl -sS -X POST http://localhost:8080/signals \
-H 'X-Orchid-Source: customer-health-webhook' \
-H 'Authorization: Bearer dev-ingest-token' \
-H 'Content-Type: application/json' \
-d '{
"type": "pipeline.customer_health.requested",
"tenant_key": "data-pipeline-demo",
"user_id": "dev-user",
"payload": {"customer_id": "C-100"}
}'The trigger copies payload into the pipeline input (input_from: payload),
runs as the signal's user (act_as_user) and records actor visibility. To
post the summary to a chat, create a chat and add a binding:
curl -sS -X POST http://localhost:8080/chats \
-H 'Authorization: Bearer dev-token' \
-H 'Content-Type: application/json' -d '{}'Then include "chat_binding": {"chat_id": "CHAT_ID", "mode": "append_final_message"}
in the signal body. The binding is honored only when the signal user owns the
chat (or an admin writes it); a foreign chat fails the Bloom run before the
pipeline starts. The assistant message carries metadata.origin = "bloom", the
trigger id, the pipeline name and the pipeline run id.
Schedule#
agents.yaml declares a daily cron schedule and a matching cron trigger:
schedules:
- id: customer-health-daily
cron: "0 6 * * *"
trigger_id: customer-health-scheduled
identity: { mode: service_account, name: scheduler }The scheduler producer emits a synthetic cron signal; the trigger runs the
pipeline with customer_id: C-SCHEDULED under the scheduler service account.
Change the cron expression in both the schedule and the trigger to test a
different cadence.
Inspect runs#
curl -sS http://localhost:8080/runs -H 'Authorization: Bearer dev-token'
curl -sS http://localhost:8080/runs/<run-id> -H 'Authorization: Bearer dev-token'Request runs are visible to their actor in data-pipeline-demo; the scheduled
run lives in tenant default under the service account, so the demo webhook
identity does not see it. /schedules is admin-only in v1.
Tests#
cd workspace-py
../.venv/bin/python -m pytest examples/data_pipeline_events \
-m "not integration" --import-mode=importlib -xThe suite covers trigger parsing, HTTP signal ingest, the real pipeline
JobRun, result shape, chat binding and rejection, the schedule producer,
visibility, the emit sink and the production API adapter. It uses SQLite event
storage in a temp directory, an in-memory chat store and a deterministic model.
See the pipelines guide and Pollen + Bloom for the underlying contracts.