---
title: "Best Practices"
description: "Recommended patterns for production-grade Orchid deployments."
package: cross-cutting
section: meta
html_url: https://orchestratorindex.com/best-practices
markdown_url: https://orchestratorindex.com/best-practices.md
---

# Best Practices

This page distils the most important decisions you will face when deploying Orchid in production. Each section is anchored to a concept or package page for deeper reading.

## Choosing Scopes

Orchid's RAG system uses a five-level hierarchy: **root → tenant → user → chat → agent**. Choosing the right scope for each collection determines both retrieval precision and operational cost.

Use **tenant scope** for shared knowledge that every user in an organisation should access — product documentation, company policies, public FAQs. Indexing once at tenant level avoids duplicating vectors per user.

Use **user scope** for personal context that must not leak across accounts — uploaded CVs, private notes, user-specific preferences. Construct it as `OrchidRAGScope(tenant_id=..., user_id=...)` — the scope filter enforces this boundary automatically.

Use **chat scope** sparingly — only when a document is relevant only to a single conversation thread. Chat-scoped collections accumulate quickly and should be pruned on chat deletion.

Use **agent scope** when a specialised agent (e.g. a legal-review agent) needs its own curated knowledge base that other agents in the same graph must not access.

Never pass raw `tenant_id` strings as vector store filters. Always construct scopes via `OrchidRAGScope` — this is the only way to guarantee the hierarchy is respected and future scope changes are backward-compatible.

See [Hierarchical RAG](/concepts/rag) for the full scope API.

## Sliding-Window Summarization

The supervisor keeps a rolling conversation window in graph state. By default it retains the last 20 turns verbatim (`supervisor.history_max_turns`) and **sliding-window summarization is enabled** (`supervisor.history_summary_enabled` defaults to `true`): older turns are compressed into a running summary via a cheap LLM call, while the most recent `history_summary_recent_turns` (default 10) exchanges stay verbatim.

**Keep it enabled** when:
- Sessions routinely exceed 15 turns.
- You are using a paid API provider and cost is a concern.
- The conversation topic shifts over time (summaries compress stale context well).

**Disable** it (`supervisor.history_summary_enabled: false`) when:
- Sessions are short-form (customer support, single-question Q&A).
- Exact verbatim recall of earlier turns is required (legal, compliance).
- You are running a local Ollama model where token cost is irrelevant.

Tune `supervisor.history_summary_recent_turns` (default 10) to control how many recent exchanges are kept verbatim before the older portion is compressed. A value of 5–8 is a good starting point for cost-sensitive deployments.

See [Supervisor](/concepts/supervisor) for the full configuration reference.

## Embedding Model Selection

Orchid supports any embedding model exposed via LiteLLM. The three most common choices differ significantly in dimension and cost:

| Model | Dimensions | Notes |
|---|---|---|
| `nomic-embed-text` (Ollama) | 768 | Free, local, good for dev/demo |
| `text-embedding-3-small` (OpenAI) | 1536 | Low cost, strong multilingual |
| `gemini-embedding-001` (Google) | 3072 | Highest quality, highest cost |

**Switching embedding models requires a full re-index.** Vector store collections are created with a fixed vector dimension. If you change `rag.embedding_model` in `orchid.yml` after data has been indexed, you must drop and recreate every collection. There is no automatic migration — this applies equally to Qdrant and ChromaDB.

Establish your embedding model choice before ingesting production data. If you anticipate switching, keep a record of which model was used for each collection in your operational runbook.

See [Embeddings](/concepts/embeddings) for dimension details and provider configuration.

## MCP Auth Decision Tree

Every MCP server entry in `agents.yaml` carries an `auth.mode` field. Choose the right mode using this table:

| Scenario | Mode | Notes |
|---|---|---|
| Local dev server, no user identity needed | `none` | Default. No auth headers sent. Server warms at process startup. |
| Server trusts the same IdP as your API | `passthrough` | Forwards the user's bearer token unchanged. Warm once per user session. |
| Server has its own OAuth AS | `oauth` | Framework runs RFC 9728 → RFC 8414 → RFC 7591 discovery automatically. |
| Internal service, mTLS or network-level auth | `none` + network policy | Use `none` and rely on infrastructure-level controls. |

Never put `client_id`, `client_secret`, or issuer URLs in `agents.yaml`. The `oauth` mode discovers and registers dynamically — YAML carries only `mode: oauth`.

See [MCP Integration](/concepts/mcp) and [OAuth & Auth](/concepts/oauth) for the full auth flow.

## Tool Strategies

The `tools.strategy` field on each agent controls how `GenericAgent` invokes tools during skill execution.

| Strategy | When to use |
|---|---|
| `all` | Tools are independent and cheap; run them in parallel for lowest latency. |
| `sequential` | Tools have side effects or depend on each other's output. |
| `llm_decides` | The set of relevant tools varies per query; let the LLM pick. Adds one extra LLM call. |

`llm_decides` is the most flexible but also the most expensive. Prefer `all` for read-only, idempotent tools (stats lookups, search). Use `sequential` when tool B consumes tool A's output (e.g. fetch → summarise).

Register custom strategies via `register_strategy(name, cls)` — never modify `STRATEGY_REGISTRY` directly.

See [Tool Strategies](/concepts/tool-strategies) for implementation details.

## Persistence

The `orchid-ai` library ships a dependency-free **in-memory** store as its zero-configuration default — ideal for tests and embedded runs, but non-durable. For durable persistence install **`orchid-storage-sqlite`** (`pip install orchid-storage-sqlite`) for single-process deployments, or **`orchid-storage-postgres`** for multi-replica deployments. `orchid-api` and `orchid-cli` bundle the SQLite plugin, so both persist to `~/.orchid/chats.db` out of the box.

**Write a custom `OrchidChatStorage`** when:
- You need a different database engine (DynamoDB for serverless, Redis for ephemeral sessions).
- Your platform has an existing chat/session store you must integrate with.
- You need custom retention policies or encryption at rest beyond what the plugin backends provide.

**Use the `orchid-storage-postgres` plugin** when:
- You already run PostgreSQL (or can add it).
- You want zero-maintenance schema migrations (the plugin ships its own versioned migrations).
- Multi-tenant isolation via `tenant_key` is sufficient.
- You need multi-replica API deployments sharing state.

**Use the `orchid-storage-sqlite` plugin** when:
- Single-process deployments (development, demos, CLI).
- No external database dependency desired.

A configured `storage.class` is resolved strictly: if the dotted path cannot be imported, startup fails with the `pip install` command that provides it — there is no silent fallback to in-memory.

The `OrchidChatStorage` ABC defines the full contract. Implement all methods — partial implementations will fail at runtime when the supervisor calls an unimplemented method.

See [Persistence](/concepts/persistence) for the ABC reference and migration guide.

## Observability

**LangSmith tracing** is enabled by setting `LANGCHAIN_TRACING_V2=true` and `LANGCHAIN_API_KEY` in your environment. The `orchid-api` lifespan calls `configure_tracing()` automatically when these variables are present. Every graph run is traced end-to-end with agent names, tool calls, and token counts.

**Logging** in `orchid-api` uses Python's standard `logging` module at INFO level with a timestamped plain-text format, and a redaction filter that scrubs Bearer tokens from every handler. `LANGCHAIN_DEBUG=true` adds verbose LLM/tool traces, and `ORCHID_ENABLE_PERF_LOGS=true` turns on `[PERF]` profiling lines for latency investigations.

**Correlation IDs** in `orchid-mcp` flow via `AsyncLocalStorage` — every log line emitted during an MCP tool call carries the same `correlationId` as the originating HTTP request. This makes cross-service tracing possible without a full distributed tracing stack.

For production deployments, wire `orchid-mcp`'s OpenTelemetry exporter to your preferred backend (Jaeger, Honeycomb, Datadog). The OTEL instrumentation is opt-in via environment variables.

## Anti-Patterns

These patterns appear frequently in early Orchid integrations and cause subtle bugs in production.

**Persisting augmented prompts.** When a user uploads a file, the framework prepends the file content to the prompt before sending it to the LLM. Save the *original* user message to chat history — not the augmented version. Storing augmented prompts bloats the history, leaks file content into future turns, and makes conversation summaries incoherent.

**Hardcoding vendor names in framework code.** The supervisor's `assistant_name` and all user-facing strings must come from `agents.yaml` (`supervisor.assistant_name`). Never hardcode a platform or company name in `orchid/`, `orchid-api/`, `orchid-cli/`, `orchid-mcp/`, or `orchid-frontend/`. Use the `supervisor:` YAML section.

**Bypassing `register_parser` / `register_strategy`.** Do not modify `PARSER_REGISTRY` or `STRATEGY_REGISTRY` directly from consumer code. Use the public registration functions. Direct mutation breaks the Open/Closed principle and will conflict with future framework versions that may replace the registry implementation.

**Importing `litellm` at module level in consumer agents.** Use `self.summarise()` (backed by the injected `self._chat_model`) for simple completions. Reserve direct `litellm` imports (inside methods, with a comment) for agentic tool-calling loops where you need the raw `tool_calls` response object.

**Calling `extract_text()` more than once per document.** PDF and image parsing is expensive. Call `extract_text()` once, pass the result to both the prompt builder and `ingest_document(pre_extracted_text=...)`. Parsing twice doubles latency and cost for no benefit.

See [Configuration](/configuration) for the full YAML reference and [Quickstart](/quickstart) for a working example that avoids all of the above.

## Multi-LLM Provider Comparison

Orchid routes all LLM calls through LiteLLM, so switching providers is a single YAML change. The table below summarises the practical trade-offs. **All numbers are approximate and sourced from provider documentation — verify current pricing before committing to a provider.**

| Provider | Latency band | Cost ballpark (input/output per 1M tokens) | Tool-calling reliability | Context window |
|---|---|---|---|---|
| OpenAI GPT-4o | Low–medium | ~$2.50 / $10.00 ¹ | Excellent — native function calling | 128k tokens |
| Anthropic Claude 3.5 Sonnet | Low–medium | ~$3.00 / $15.00 ² | Excellent — tool use API | 200k tokens |
| Google Gemini 1.5 Pro | Medium | ~$1.25 / $5.00 ³ | Good — function calling | 1M tokens |
| Ollama (local, llama3.2) | High (hardware-dependent) | Free | Good for 8B+; weaker on complex schemas | 128k tokens |
| Groq (llama3-70b) | Very low | ~$0.59 / $0.79 ⁴ | Good | 8k tokens |

¹ [OpenAI pricing](https://openai.com/api/pricing/) — verify current rates.  
² [Anthropic pricing](https://www.anthropic.com/pricing) — verify current rates.  
³ [Google AI pricing](https://ai.google.dev/pricing) — verify current rates.  
⁴ [Groq pricing](https://groq.com/pricing/) — verify current rates.

For production workloads, start with OpenAI or Anthropic for reliability, then benchmark Groq for latency-sensitive paths. Use Ollama for local development to avoid API costs during iteration.
