Preview / closed beta. Interested? Email info@indexify.dev.

Developer Guide

Agents & MCP

A complete, hands-on guide for developers building agents: start from scratch, get a working integration shipped, then level it up with advanced retrieval, structured document access, and the operational details you’ll want in production.

Agents & MCP

Building Agent Workflows

Treat Indexify as a retrieval and evidence service: your agent (or chat backend) decides when to search, what to fetch next, and how to cite results. The same operations exist as REST for production traffic and MCP for agent hosts that already speak the Model Context Protocol—see MCP Integration. For end-to-end scenario walkthroughs (levels 1–5), read Building agents with Indexify. Start from Retrieval fundamentals and Structured knowledge; map product ideas to flows using Start here.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Mental model
    • Discover — Resolve kbId / documentId (list KBs or store ids from your own catalog).
    • Retrieve — Run search (semantic, hybrid, optional rerank) with citation metadata when users need provenance.
    • Drill down — If snippets are insufficient, call parsed document, chunks, elements, sections tools or the equivalent REST routes.

Single-shot RAG vs multi-step agents

Start with the smallest retrieval that answers the question; add steps only when metrics show ambiguity or thin evidence. Every extra tool call adds latency and failure modes—record traces to prove the depth is worth it.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Single-shot
    • One POST …/search or MCP indexify_search with includeSourceMetadata: true so the model can cite filenames, chunk indices, or offsets. Best when queries are short and the KB is on-topic.
  • Inspect-then-answer
    • If search hits look truncated or conflicting, fetch indexify_get_document_chunks or indexify_get_parsed_document (GET …/chunks, GET …/parsed) for the strongest documentId before generating.
  • Structured inspection
    • For tables, figures, or outline navigation, use indexify_list_document_elements / indexify_get_document_element (and related section/table/figure routes). Tune multimodal search surfaces on the knowledge base (search.multimodal.retrievalModes); combine search elementTypes / filters with those routes where the API reference allows them.
  • Plan / refine loops
    • Your orchestration layer (state machine, workflow engine, or custom loop) can rewrite queries, merge structured reads and vector hits, or ask the user to disambiguate—always cap iterations, tool calls, and wall-clock time.
  • REST vs MCP

REST in your product vs MCP in the loop

The host that runs the agent (your API, a desktop product, or an MCP-capable IDE) decides where tokens live and how tools are exposed. Indexify remains a stateless HTTP API behind both paths.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Backend agent / chat API
    • Mint and cache project access tokens on the server; call POST …/search and structured routes directly; log documentId, chunkId, and correlation ids—never log Bearer tokens.

Citations, traces, budgets, and refusals

Observable retrieval fields (scores, metadata, empty results) let you build refusal and escalation rules. You are responsible for prompt and policy; Indexify supplies evidence and ids for audit.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Evidence & traces
    • Turn on source metadata in search when answers must cite real documents (Search).
    • Persist tool name, kbId, documentId, and pagination cursors in your traces; redact secrets.
  • Budgets & refusal
    • Bound retrieveTopK, rerank.topK, search limit, and chunk page sizes so tail latency stays predictable (Advanced retrieval).
    • Define product behavior for low scores, empty search, or conflicting passages (second query, ask user, escalate to human).

Landing scenarios → Indexify agent workflows

Maps the scenario cards in Start here to concrete REST and MCP sequences. Replace https://api.indexify.dev with your account’s base URL from the API reference when different.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Ask-anything knowledge chatbot
    • Pick KB(s) via indexify_list_kbs or a stored kbId; each turn indexify_search with includeSourceMetadata: true; enable hybrid and rerank in request settings when the KB and eval set warrant it (Advanced retrieval).
    • Multi-KB: run parallel search per KB, then merge or rank hits in your orchestration before generation.
  • KB-aware tool-calling agent
    • Retrieve-first: indexify_search (+ elements/sections); pass documentId, snippet, metadata into custom tools (tickets, CRM).
    • Tool schema: include kbId + evidence refs. Narrow MCP to search:run + docs:parsed:read if ingest/delete must be blocked.
  • Document navigation copilot
    • Per document: indexify_get_document_chunks or indexify_list_document_elements (sectionId, type).
    • Corpus-scale: indexify_search then expand with elements/chunks; cite headings, tables, figures for UI highlight.
  • Workflow automation (REST or MCP)
    • Async: webhooks → your backend → search/structured for classify/summarize/route.
    • Interactive: MCP in the IDE for live KB context; idempotent upload + job polling when ingesting; read-only tokens for retrieval-only services.
  • Compliance and policy Q&A
    • Isolate policy corpora in dedicated KBs; use hybrid + rerank and groupBy: document (or documented filters) so distinct policies are not blended in one context window.
    • Log evidence ids and timestamps for audit; when two passages disagree, present both with citations rather than merging silently.
  • IDE knowledge agent (MCP)
    • Configure MCP with a read/search-scoped project token (Connecting MCP clients).
    • Loop: indexify_list_kbsindexify_search (includeSourceMetadata: true) → chunk/section/element tools for drill-down (MCP agent retrieval loop).
    • Use REST for ingest and admin; MCP only for interactive IDE retrieval.

MCP Integration and Tooling

Model Context Protocol (MCP) lets an agent host (IDE, desktop assistant, or custom runtime) discover and invoke tools that map to the same public REST surface you would call with curl. Point the host at the MCP base URL from your product documentation, path /api/mcp, Streamable HTTP transport, and send an Indexify project access token as Authorization: Bearer on every HTTP request. Tool definitions in your client should match the API reference for the underlying routes.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Coverage
    • The shipped tool set covers KB listing, document listing, search, parsed reads, chunks, elements, section-scoped element lists, artifacts, relationships, sections, tables, and figures—see Tool catalog.
    • REST-only today (not exposed as MCP tools): job listing/detail for arbitrary automation and other routes listed in the catalog—call them with curl or your HTTP client.
    • Typical agent loop: indexify_list_kbsindexify_search (or indexify_search_enriched / indexify_search_with_neighbors) → optional drill-down tools matching your evidence needs. Step-by-step: MCP agent retrieval loop.
  • When to use MCP vs REST
    • MCP — interactive development, IDE copilots, and any host that already manages MCP servers and secrets for you.
    • REST — production backends, webhooks, batch pipelines, and anywhere you need full route coverage, custom retries, and centralized token custody.

Authentication and scopes

MCP is a transport; authorization is still OAuth2 client credentials for the target project. Every Streamable HTTP request to /api/mcp must include Authorization: Bearer <project_access_token>. The same token authorizes equivalent REST calls to your public API base URL. Minting: Quickstart, Appendix — token matrix.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Header & minting
    • Authorization: Bearer <access_token> on every MCP HTTP call (configure in the client).
    • Mint via POST /oauth/token, grant_type=client_credentials, with client_id and client_secret in the form body (Authentication).
    • RFC 9728 discovery — clients can GET /.well-known/oauth-protected-resource on the MCP host for resource (protected MCP endpoint id) and authorization_servers (token issuer URLs). Unauthenticated 401 responses from /api/mcp include a resource_metadata hint pointing at that metadata URL.
  • Typical scope needs
    • List + search minimum: kb:read, search:run.
    • Parsed doc + chunks: docs:parsed:read (+ API-enforced doc read scopes).
    • Elements, artifacts, sections, tables, figures, relationships: document routes — include docs:read / docs:parsed:read as required.

Connecting MCP clients

Use the Streamable HTTP transport to the URL shown in your Indexify product documentation. Every request must send your project access token as Authorization: Bearer ….

Tip: the snippets are meant to be copy/paste friendly—start with the happy path, then intentionally poke at scopes, missing fields, and bad inputs so you know exactly what your app will see in production.

  • Endpoint
    • https://mcp.indexify.dev/api/mcp (Indexify MCP server).
  • Client patterns
    • IDE / agent product — configure MCP server URL + static or injected Bearer header per that product’s MCP documentation.
    • Custom integration — your HTTP client calls /api/mcp with Streamable HTTP framing and the Bearer header on each request.
    • stdio-only MCP clients — use a small HTTP bridge process that forwards requests to /api/mcp and attaches the Bearer token from your secret store.

Example MCP client config shape (JSON; exact keys depend on your MCP host application)

{
  "mcpServers": {
    "indexify": {
      "url": "https://mcp.indexify.dev/api/mcp",
      "headers": {
        "Authorization": "Bearer <indexify_project_access_token>"
      }
    }
  }
}

MCP agent retrieval loop (think → retrieve → evaluate → refine → answer)

Map an iterative agentic RAG loop to Indexify MCP tools. The same behavior is available over REST (Retrieval fundamentals, Advanced retrieval, Search diagnostics); MCP is for hosts that already run tool-calling agents.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Think
    • The host LLM plans sub-questions, chooses kbId, and decides whether to start with vector search or structure-first navigation.
  • Retrieve
    • List / resolveindexify_list_kbs (or use a configured kbId).
    • Searchindexify_search with includeSourceMetadata: true (and includeGrounding when you need page/section grounding). Use indexify_search_enriched for MCP-side refinementHints and citations, or indexify_search_with_neighbors for adjacent chunk snippets in one call.
    • Structure-firstindexify_list_document_sections / indexify_list_document_elements, or indexify_search with groupBy: section / elementTypes; use indexify_get_document_chunks / section tools after the user picks a branch.
    • Diagnostics — set diagnostics: true on indexify_search (or enriched/neighbors) to receive API diagnostics for the evaluate step (Search diagnostics).
  • Evaluate
    • Use hit score / rerankScore, refinementHints from indexify_search_enriched, and optional API diagnostics to decide if evidence is sufficient, empty, or dominated by one document or modality.
  • Refine
    • Adjust searchMode, rerank (see OpenAPI SearchRequestRerank), retrieveTopK, groupBy, tableQuery, or filters; tune which surfaces search uses via KB search.multimodal.retrievalModes; or drill down with indexify_get_document_chunks, indexify_get_document_section, indexify_list_section_elements, indexify_list_document_elements, indexify_get_parsed_document.
  • Retrieve again → answer
    • Re-run search or fetch tighter passages, then generate an answer that cites documentId / chunkId / metadata from includeSourceMetadata. Cap tool iterations and wall-clock time in your orchestrator.

MCP tool catalog (all tools)

Each tool is a thin wrapper around a documented GET or POST on the public API. Parameter names and types follow the OpenAPI schemas—your MCP client may show JSON Schema derived from the server; treat the API reference as the contract of record. Auth: Authorization: Bearer with a project access token on every MCP HTTP request. For connection details see Connecting MCP clients.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

Tools: knowledge bases & document metadata

Resolve kbId and documentId before search or structured reads. Related guide: Document ingestion.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • indexify_list_kbs
    • Parameters: optional limit (1–100), cursor (opaque pagination), order (asc | desc). If you omit limit, the tool may default to a larger page—pass limit explicitly to match your REST pagination tests.
    • Returns: JSON with KB rows and nextCursor (same semantics as GET …/kbs).
    • REST: GET …/kbs
  • indexify_list_documents
    • Required: kbId (UUID).
    • Optional: limit (integer 1–100), cursor (UUID nextCursor), sort (createdAt | updatedAt), order (asc | desc).
    • Returns: upstream JSON (items, cursor fields).
    • REST: GET …/documents
  • indexify_get_document

Tools: indexify_search_enriched & indexify_search_with_neighbors

MCP-only composition on top of POST …/search plus optional chunk reads. Same auth and search:run scope as indexify_search. See MCP agent retrieval loop.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • indexify_search_enriched
    • Parameters: same flattened search fields as indexify_search, plus optional includeRefinementHints, includeCompactCitations, maxPerDocument, maxPerSection, diagnostics.
    • Returns: results, nextCursor, format, optional diagnostics, optional refinementHints (heuristic strings + stats), optional citations (compact provenance objects). Post-filters apply after the API returns (ordering preserved within each group).
  • indexify_search_with_neighbors
    • Behavior: calls search with includeSourceMetadata: true, then loads up to maxChunksPerDocument chunks per document for the top neighborTopHits hits and attaches neighbor snippet arrays (span neighborSpan, preview length neighborPreviewChars; all capped).
    • Returns: results, neighbors, neighborNotes (truncation hints), plus optional diagnostics, citations, refinementHints when those flags are set.

Tools: elements & artifacts

Typed layout elements and parser artifacts for one documentId. Concepts: Structured document access.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • indexify_list_document_elements
    • Required: kbId, documentId (UUIDs).
    • Optional: type (string), sectionId (UUID), page (int ≥ 1), limit (1–100), cursor, include (comma-separated API flags, e.g. payload,grounding,relationships).
    • Returns: JSON string.
    • REST: GET …/elements
  • indexify_get_document_element
  • indexify_list_document_artifacts
    • Required: kbId, documentId (UUIDs).
    • Optional: kind, limit (1–200), cursor.
    • Returns: JSON string.
    • REST: GET …/artifacts
  • indexify_get_document_artifact

Tools: parsed document & chunks

Parsed body (per Accept / format) vs chunk listing for retrieval windows. Use when search snippets are insufficient (Agent patterns).

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • indexify_get_parsed_document
    • Required: kbId, documentId (UUIDs).
    • Optional: formatmarkdown | html | text | json | doctags (default markdown; maps to API Accept).
    • Returns: plain text in MCP text (not JSON-wrapped).
    • REST: GET …/parsed
  • indexify_get_document_chunks
    • Required: kbId, documentId (UUIDs).
    • Optional: limit (1–100, default 20), cursor.
    • Returns: JSON with items, nextCursor (or a short “no chunks” message).
    • REST: GET …/chunks

Tools: document & element relationships

Document-wide and element-scoped relationship lists (typed edges between parsed nodes).

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • indexify_list_document_relationships
    • Required: kbId, documentId (UUIDs).
    • Optional: type, limit (1–200), cursor.
    • Returns: JSON string.
    • REST: GET …/relationships
  • indexify_list_element_relationships
  • indexify_list_related_elements

Tools: sections, tables & figures

First-class sections, tables, and figures for one document—useful for navigation UIs and multimodal answers.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • indexify_list_document_sections
    • Required: kbId, documentId (UUIDs).
    • Optional: limit (1–100), cursor.
    • Returns: JSON string.
    • REST: GET …/sections
  • indexify_get_document_section
    • Required: kbId, documentId, sectionId (UUIDs).
    • Optional: depth (0–10, nested descendants).
    • Returns: JSON string.
    • REST: GET …/sections/{sectionId}
  • indexify_list_section_elements
    • Required: kbId, documentId, sectionId (UUIDs).
    • Optional: type (comma-separated element types), limit (1–100), cursor (UUID pagination).
    • Returns: JSON string (items, nextCursor).
    • REST: GET …/sections/{sectionId}/elements
  • indexify_list_document_tables
    • Required: kbId, documentId (UUIDs).
    • Optional: limit (1–100), cursor.
    • Returns: JSON string.
    • REST: GET …/tables
  • indexify_get_document_table
  • indexify_list_document_figures
    • Required: kbId, documentId (UUIDs).
    • Optional: limit (1–100), cursor.
    • Returns: JSON string.
    • REST: GET …/figures
  • indexify_get_document_figure
    • Required: kbId, documentId, figureId (UUIDs).
    • Optional: include (query string for extra figure fields).
    • Returns: JSON string.
    • REST: GET …/figures/{figureId}

API Runner and Developer Productivity

The API Runner runs authenticated HTTP requests from your browser against the same paths described in the API reference. Most cURL examples in this guide that target https://api.indexify.dev show Open endpoint in api-docs and Open in API Runner under the block—use those links to jump straight to the right operation, paste a project token, and fill path placeholders.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Workflow
    • Walk the Quickstart sequence in the Runner first (token → KB → upload → search).