indexify.dev - Document Processing for AI Agents

Building agents with Indexify

This chapter walks through six product scenarios for agent and RAG backends using only the public HTTP API on the hosted Indexify service (OpenAPI: paths under /projects/{projectId} and /projects/{projectId}/kbs/{kbId}). MCP is an optional integration surface for agent runtimes: it exposes a subset of the same operations as tools and is not part of the OpenAPI document. Progress deliberately from minimal HTTP to richer retrieval, then agent-hosted MCP, structure-aware navigation, webhook-driven automation, and compliance-style Q&A.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

Actors
- End user — asks questions or navigates your UI.
- Your application / backend — holds project access tokens, calls Indexify over HTTPS, enforces policy, aggregates multi-KB results.
- Agent runtime (optional) — LLM + tool loop; may call MCP tools that map to the same REST routes.
- Indexify API — creates KBs, ingests documents, runs jobs, answers search requests, delivers webhooks.
Progression (simple → advanced)
- Level 1 — HTTP only
- Level 2 — stronger answers
  - settings.includeSourceMetadata on search.
  - Hybrid mode and rerank when your eval set warrants it.
  - Multiple KBs: aggregate in your app—each request still uses one kbId in the path.
- Level 3 — agentic (MCP)
  - Typical tools: indexify_list_kbs, indexify_search / indexify_search_enriched / indexify_search_with_neighbors, then drill-down reads as needed.
  - Step-by-step loop: MCP agent retrieval loop. Optional diagnostics: true on search mirrors REST settings.diagnostics (Advanced retrieval — Search diagnostics).
  - MCP is optional and not part of the OpenAPI document.
- Level 4 — structure-aware
  - POST …/search
  - GET …/sections
  - GET …/elements
  - GET …/tables · GET …/figures · … (API reference)
- Level 5 — automation
  - PUT …/webhook
  - Verify X-Indexify-Signature on the raw body.
  - Follow-up: POST …/search, GET …/parsed, section/element routes as events imply.
Cross-links
- Knowledge Base Design · Ingestion · Jobs & webhooks · Search · Structured knowledge · MCP · Authentication
Best endpoint combinations (by scenario)
- Ask-anything knowledge chatbot
  - POST …/search with includeSourceMetadata: true (optional includeGrounding).
  - Optional GET …/chunks to widen context around a hit.
  - GET …/parsed when the model needs full-document coherence.
- KB-aware tool-calling agent
  - POST …/search with filters; set search.multimodal.retrievalModes on the KB when tables/figures should appear in hits.
  - GET …/tables/{tableId} · GET …/elements/{elementId} · GET …/elements/{elementId}/related for structured tool args.
  - Run external tools using citation metadata from search.
- Document navigation copilot
  - GET …/sections · GET …/sections/{sectionId}/elements · GET …/elements (filters).
  - GET …/figures · GET …/figures/{figureId} for diagrams.
  - Add POST …/search for cross-document retrieval when you need ranked passages.
- Workflow automation via HTTP
  - POST …/documents (or POST …/reprocess / bulk POST …/documents/reprocess).
  - Jobs — GET …/jobs/{jobId}, or drive flow from webhooks.
  - GET …/parsed · GET …/chunks for downstream text consumers.
  - GET …/sections and GET …/sections/{sectionId} to navigate document outlines.
  - POST …/search for event-driven Q&A or tagging.
- Compliance and policy Q&A
  - POST …/search with includeSourceMetadata: true for citation ids + source attributes (add includeGrounding when the UI needs page/bbox/sectionPath — see api-docs).
  - GET …/chunks or GET …/elements/{elementId} for audit-friendly evidence.
  - GET …/parsed for clause-level review when one document is in scope.
- IDE knowledge agent (MCP)
  - Configure MCP host with project access token (Connecting MCP clients).
  - indexify_list_kbs → indexify_search with includeSourceMetadata: true; drill down with chunk/section/element tools.
  - Read-only scopes (kb:read, search:run, docs:parsed:read) for IDE use; REST for ingest and admin.

Ask-anything knowledge chatbot

What this is — A conversational product where each user message is grounded in your corpus via search, then an LLM (outside Indexify) turns retrieved passages into an answer. Why Indexify — managed ingest, chunking, indexing, and search with optional citation metadata per hit. Platform — does not run your chat model; it returns evidence your app injects into the model context.

Tip: the snippets are meant to be copy/paste friendly—start with the happy path, then intentionally poke at scopes, missing fields, and bad inputs so you know exactly what your app will see in production.

What to create first
- Knowledge base — Start with one KB per coherent domain (e.g. internal handbook, one product line). The search URL always includes a single kbId; multi-domain chat requires your backend to run one search per KB and merge/rank results.
- Documents — Upload PDFs, Office files, HTML, markdown, or other supported types (Ingestion formats); enable parse output formats that match your planned search format query parameter.
- KB boundaries — Split when access control, tone, or corpus freshness differ; keep together when users expect one blended answer space.
Step-by-step implementation
1. Authenticate — Your backend obtains a project access token via POST /oauth/token (client_credentials).
  - Actor: backend.
2. Create KB — POST …/kbs with name and settings appropriate for your corpus. Use the matching KB settings example under Knowledge Base Design as a starting settings template.
  - Outcome: kbId.
3. Ingest — POST …/documents (multipart file).
  - Outcome: documentId, jobId.
4. Wait for processing — Poll GET …/jobs/{jobId} until terminal success, or subscribe to webhooks later.
  - Platform: parse → chunk → index.
5. User message — User sends a question in your UI.
  - Actor: end user.
6. Retrieve — Backend calls POST …/search with the user query and settings: at minimum includeSourceMetadata: true for citations; add includeGrounding when you need richer grounding (API reference). For eval or agent evaluate → refine, optionally set diagnostics: true for SearchDiagnostics (Search diagnostics).
  - Outcome: ranked chunks + metadata.
7. Widen or synthesize (optional) — If hits are thin, GET …/chunks on the top documentId expands surrounding context; if the model needs whole-document coherence, follow with GET …/parsed.
  - Order: search → chunks (optional) → parsed (optional).
8. Generate answer — Your app sends system + user prompts and retrieved text (and citation fields) to your LLM provider.
  - Actor: your model.
9. Multi-KB (optional) — Repeat the retrieve step per kbId, then merge hits (dedupe by documentId/chunkId, re-score) before generation.
Endpoints used
- Setup
  - POST /oauth/token (machine token)
  - POST …/kbs
- Ingestion
  - POST …/documents
  - Optional Idempotency-Key header.
- Status
  - GET …/jobs/{jobId}
  - GET …/jobs when listing jobs.
- Retrieval
  - POST …/search — settings.includeSourceMetadata, optional includeGrounding, optional settings.diagnostics (see api-docs).
  - Optional follow-ups: GET …/chunks, GET …/parsed.
Why this order
- You must have indexed chunks before search returns useful hits—hence KB → upload → job success → search.
- Citation metadata is optional in search settings; add it when UX requires traceable sources (Search).
MCP (Level 3, optional)
- Typical tools for this flow
  - indexify_list_kbs
  - indexify_search (flattened parameters equivalent to the REST body); indexify_search_enriched / indexify_search_with_neighbors for MCP-side hints, citations, or neighbor chunks
- MCP is not in OpenAPI. Use MCP for interactive agents; use direct HTTP in production chat backends (MCP catalog).
Practical notes
- Each POST …/search targets exactly one kbId; parallel calls implement multi-corpus assistants.
- Typical depth progression: search → GET …/chunks for adjacent context → GET …/parsed for whole-document flow (Structured access).

Level 1 — create a knowledge base (replace the sample Bearer token with your access token from the OAuth client credentials flow)

curl -s -X POST "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Content-Type: application/json" \
  -d '{"name":"support-handbook","description":"Employee policies"}'

Open endpoint in api-docs Open in API Runner

Upload a document (multipart)

curl -s -X POST "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/documents" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -F "file=@./employee-handbook.pdf"

Open endpoint in api-docs Open in API Runner

List recent jobs (e.g. failed only); next page: append &cursor=<nextCursor> from JSON

curl -s -X GET "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/jobs?limit=20&status=failed" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk"

Open endpoint in api-docs Open in API Runner

Check one job — path IDs must be UUIDs

curl -s -X GET "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/jobs/880e8400-e29b-41d4-a716-446655440004" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk"

Open endpoint in api-docs Open in API Runner

Level 2 — search with citation metadata for the chat turn

curl -s -X POST "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/search?format=markdown" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Content-Type: application/json" \
  -d '{"query":"How many PTO days per year?","settings":{"includeSourceMetadata":true,"searchMode":"hybrid"}}'

Open endpoint in api-docs Open in API Runner

KB-aware tool-calling agent

What this is — An agent that retrieves evidence from your KB before calling external tools (ticketing, billing, CRM). Indexify reduces hallucinated parameters and grounds decisions. Your backend still owns external API keys and side effects; the agent proposes tool calls using retrieved facts.

What to create first
- KB content — Procedures, SKU lists, escalation rules, API docs—anything the agent must not invent.
- Boundaries — One KB per trust domain or use multi-KB retrieval with explicit routing logic in your orchestrator.
Step-by-step implementation
1. Create KB — POST …/kbs with settings for structured evidence (e.g. doctags, json parse output, rerank) your tools rely on. Template: KB-aware tool-calling KB example in Knowledge Base Design.
  - Outcome: kbId. Ingest sources with POST …/documents before users run the flow.
2. User goal — e.g. “Create a ticket for the billing error in Acme Corp.”
  - User.
3. Retrieve — POST …/search with includeSourceMetadata (and filters as needed). When tool args depend on tables or figures in search hits, set search.multimodal.retrievalModes on the knowledge base (not per-request).
  - Indexify returns passages + ids for citation-aware tool payloads.
4. Sufficiency check — Policy code decides if scores, hit count, or required fields are present; if not, ask the user or refine query—do not call external tools yet.
  - Your logic.
5. Pack structured tool args — From hits, call GET …/tables/{tableId} or GET …/elements/{elementId} / GET …/elements/{elementId}/related for the exact rows, nodes, or related elements your tool needs.
  - Keep execution on your backend; pass through citation metadata from the search step.
6. Tool plan — LLM emits structured tool calls only after retrieval context is attached to the prompt.
  - Agent.
7. Execute tools — Your backend validates and runs external APIs (never expose those secrets to the client).
  - Backend.
8. Respond — Assistant message cites which document supported the action when required.
  - Uses metadata from the retrieve step.
Endpoints used
- Setup / ingest (same as chatbot)
- Retrieval
  - POST …/search — filters; KB search.multimodal.retrievalModes when tables/figures should surface in search.
- Structured drill-down (tool arguments)
  - GET …/tables/{tableId} · GET …/elements/{elementId} · GET …/elements/{elementId}/related · GET …/elements
  - GET …/parsed / GET …/chunks when plain text is enough
Progression
- HTTP first — Implement retrieval + tool gateway entirely in your service (easiest to audit).
- MCP agent — Expose read tools (indexify_search, element/section tools) to the agent host; keep mutating external APIs on your server. MCP does not replace your tool execution layer.
Practical notes
- Log chunkId / documentId with each external action for compliance.
- Narrow MCP credentials to search:run + docs:parsed:read if the agent must never ingest or delete.

Evidence pull before tool execution (hybrid + citations + rerank)

curl -s -X POST "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/search?format=markdown" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Content-Type: application/json" \
  -d '{"query":"billing dispute escalation steps for enterprise accounts","settings":{"searchMode":"hybrid","includeSourceMetadata":true,"retrieveTopK":30,"rerank":{"isEnabled":"yes","topK":5},"includeGrounding":false,"diagnostics":false,"groupBy":"document","elementTypes":[]}}'

Open endpoint in api-docs Open in API Runner

Document navigation copilot

What this is — UX for moving through long documents (manuals, contracts) using GET …/sections, GET …/sections/{sectionId}/elements, GET …/elements (filters), GET …/figures and GET …/figures/{figureId}, not only ad-hoc Q&A. Add POST …/search for cross-document jumps when structured parse output is available. Why Indexify — hierarchy and parse artifacts are first-class; section and element routes extend navigation beyond a single file.

What to create first
- KB — Load long-form PDFs/DOCX with settings that preserve layout and headings (KB design).
- Expectations — Navigation quality depends on parse output; validate with GET …/sections and GET …/elements on a sample file.
Step-by-step implementation
1. Create KB — POST …/kbs with json (and doctags/markdown as needed) in parse.outputFormats so sections, tables, and figures are available. Reference Document navigation KB example in Knowledge Base Design.
  - Outcome: kbId. Upload manuals, then optional POST …/reprocess until ingest completes.
2. User picks a document
  - From library UI listing GET …/documents results.
  - User.
3. Load outline
  - GET …/sections for top-level headings.
  - Backend.
4. Drill in
  - GET …/sections/{sectionId} with optional depth for nested structure.
  - GET …/sections/{sectionId}/elements to load elements under that section in one call.
  - GET …/elements with filters (e.g. type, sectionId) for broader scans.
  - Backend.
5. Free-text intent — User says “jump to deployment prerequisites”.
  - User.
6. Resolve target
  - POST …/search with mode: keyword or semantic and query from user text—cross-document when filters.documentIds is omitted or lists multiple docs (requires completed ingest).
  - For single-doc UX, keep filters.documentIds: [documentId] and elementTypes such as ["heading","section"].
  - Pick best match node.
  - Backend.
7. Show content
  - GET …/parsed or GET …/chunks for that region.
  - Or GET …/elements/{elementId} for a specific node.
  - Backend.
8. Related navigation
  - GET …/elements/{elementId}/related.
  - Or ${AD.elemRelated} for “what relates to authentication?”.
Example user utterances → API strategy
- “Jump to deployment prerequisites” — keyword search or section title match, then scroll UI to returned elementId / section id.
- “Show the plan limits table” — GET …/tables or POST …/search with elementTypes: ["table"] + filters.documentIds.
- “What section comes after this?” — section list + reading-order relationships (reading-order neighbors).
- “Related authentication sections” — POST …/search with query: "authentication" and semantic or keyword mode; present matching headings.
Endpoints used
- Ingestion / metadata
  - POST …/documents
  - GET …/documents/{documentId}
- Navigation
  - GET …/sections · GET …/sections/{sectionId} · GET …/sections/{sectionId}/elements · GET …/elements · GET …/elements/{elementId} · GET …/elements/{elementId}/related
  - GET …/tables · GET …/figures · GET …/figures/{figureId}
- Graph
- Read text
  - GET …/parsed
  - GET …/chunks
MCP (optional)
- Tools
  - indexify_list_documents
  - indexify_get_document
  - indexify_list_document_sections
  - indexify_get_document_section
  - indexify_list_document_elements
  - indexify_search
- These tools mirror the REST flow above.

List sections (outline) for a document

curl -s -X GET "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/documents/770e8400-e29b-41d4-a716-446655440003/sections?limit=50" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk"

Open endpoint in api-docs Open in API Runner

Fetch one section subtree (depth per OpenAPI)

curl -s -X GET "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/documents/770e8400-e29b-41d4-a716-446655440003/sections/bb0e8400-e29b-41d4-a716-446655440007?depth=3" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk"

Open endpoint in api-docs Open in API Runner

List tables in a document (inventory only; use id as tableId for GET …/tables/{tableId}). Query: limit + optional cursor (UUID nextCursor).

curl -s -X GET "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/documents/770e8400-e29b-41d4-a716-446655440003/tables?limit=20" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk"

Open endpoint in api-docs Open in API Runner

Workflow automation via the HTTP API

What this is — Your backend services react to webhooks (or your own schedulers) and call Indexify to search, fetch parsed content, or run structure operations as part of ETL, routing, or notification flows. No chat UI required. MCP can mirror read operations for IDE agents, but automation in production should use REST and verified webhooks.

What to create first
- KB + webhook — One KB per automated pipeline stage if isolation helps; configure PUT …/webhook with url, events, active, retryPolicy (API reference). Store the returned secret once.
- HTTPS endpoint — Your URL must accept POST, return 2xx, and verify X-Indexify-Signature over the raw body (Webhook security).
Step-by-step implementation
1. Create KB — POST …/kbs with parse/index/search settings suited to pipeline automation (JSON/doctags, multimodal retrieval defaults, rerank). See Workflow automation KB example in Knowledge Base Design.
  - Outcome: kbId — then attach the webhook and ingestion flows below.
2. Register webhook — PUT …/webhook subscribing e.g. to job.completed.
  - Backend.
3. Ingest documents — Existing POST …/documents flows enqueue jobs.
  - Uploader.
4. Event delivery — Indexify POSTs JSON to your URL.
  - Platform → your webhook URL.
5. Verify — Compare HMAC-SHA256 (lowercase hex) of raw body to header using stored secret; reject on mismatch.
  - Webhook handler.
6. Follow-up calls — On job.completed (or after POST …/reprocess):
  - GET …/parsed and/or GET …/chunks for text feeds to downstream systems.
  - POST …/search for event-driven Q&A or classification-style snippets.
  - POST …/reprocess GET …/sections when automation needs structural labels.
  - POST …/search when automation needs ranked passages.
  - Backend → Indexify API.
7. Downstream — Update your CMS, send notifications, or enqueue another job in your system.
  - Backend.
Endpoints used
- Webhook config
  - PUT …/webhook · GET …/webhook · PATCH …/webhook · POST …/webhook/test · DELETE …/webhook
- Ingestion / status
- Retrieval / structure
  - GET …/parsed · GET …/chunks · POST …/search
  - GET …/sections · POST …/reprocess · POST …/search
  - Other document reads: API reference.
MCP note
- IDE agents can use MCP tools for interactive inspection of the same KB; webhook handlers should still use REST with server-stored tokens.
Practical notes
- Treat webhook delivery as at-least-once; dedupe using stable ids from the payload (Webhooks).
- Keep secret rotation coordinated with PATCH …/webhook when the product supports secret updates.

curl -s -X PUT "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/webhook" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://api.example.com/hooks/indexify","events":["job.pending","job.started","job.parsing_started","job.parsing_format_completed","job.parsing_structure_ready","job.parsing_partial_failure","job.parsing_completed","job.chunking_started","job.chunking_format_selected","job.chunking_partial_failure","job.chunking_completed","job.indexing_started","job.indexing_completed","job.completed","job.failed","job.canceled"],"active":true,"retryPolicy":{"maxAttempts":5,"backoffSeconds":60}}'

Open endpoint in api-docs Open in API Runner

After your handler processes job.completed — run search for tagging

curl -s -X POST "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/search?format=markdown" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Content-Type: application/json" \
  -d '{"query":"document summary keywords for routing","settings":{"retrieveTopK":5}}'

Open endpoint in api-docs Open in API Runner

Compliance and policy Q&A

What this is — Grounded answers over policies, controls, and audit materials with traceable citations. Why Indexify — search hits can include citation metadata (chunk ids + citationRef + source attributes), and optional grounding (pages/bbox/section path) when requested. Each search still targets one kbId; segregate corpora when regulations require separation.

What to create first
- KB boundaries — e.g. separate KBs per policy domain, region, or data classification so a single search does not blend incompatible sources.
- Documents — Final PDFs or controlled exports; record a revision identifier in the filename or custom metadata if your ingestion path supports it.
Step-by-step implementation
1. Create KB — POST …/kbs with policy-oriented parse/chunk/search settings (OCR, hierarchical chunks, grounding). Template: Compliance KB example in Knowledge Base Design.
  - Outcome: kbId.
2. Ingest corpora
  - POST …/documents, then wait for job success (GET …/jobs/{jobId} or webhooks).
3. User question — Often sensitive; run access control before calling Indexify.
  - Your auth.
4. Scoped retrieval
  - POST …/search on the appropriate kbId.
  - settings.includeSourceMetadata: true; add includeGrounding when the UI or audit trail needs page/bbox-style grounding (API reference).
  - Often settings.searchMode: hybrid for exact policy terms plus paraphrase matching.
5. Optional grouping — Use documented groupBy / filters so answers stay within one document when mixing would be misleading (Search).
6. Render answer — Show filename, heading, and chunk identifiers; add grounding (pages/bbox/section path) when the UI needs it; store these in audit logs.
7. Drill-down / audit — From hits, retain chunk and element ids in logs; load evidence as needed:
  - GET …/chunks for passage-level audit.
  - GET …/elements/{elementId} for a specific structured node.
  - GET …/parsed for clause-level review when one document is in scope.
  - GET …/sections to align UI with headings.
Endpoints used
- Setup / ingest
- Retrieval
  - POST …/search with includeSourceMetadata, optional includeGrounding, hybrid/rerank as needed.
- Evidence (review workflows)
  - GET …/chunks · GET …/elements/{elementId} · GET …/parsed
  - GET …/sections · GET …/sections/{sectionId}
Multi-KB
- Run sequential or parallel searches per KB, then present results per corpus so users see which regime applies—never silently merge conflicting policies.
MCP (optional)
- Internal review agents may use indexify_search + parsed/chunk tools; production compliance UIs should still prefer server-side REST with full logging.
Practical notes
- Low-confidence hits → respond with insufficient evidence rather than guessing.
- Re-evaluate after KB setting changes (embedding, chunking) using a fixed golden question set (Best practices).

Compliance-oriented search (citations + hybrid; groupBy when spec allows)

curl -s -X POST "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/search?format=markdown" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Content-Type: application/json" \
  -d '{"query":"What are the data retention requirements for user logs?","settings":{"includeSourceMetadata":true,"searchMode":"hybrid","groupBy":"document","retrieveTopK":40,"rerank":{"isEnabled":"yes","topK":8},"includeGrounding":true,"diagnostics":false}}'

Open endpoint in api-docs Open in API Runner

Fetch parsed text for audit review of a hit

curl -s -X GET "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/documents/770e8400-e29b-41d4-a716-446655440003/parsed" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Accept: text/markdown"

Open endpoint in api-docs Open in API Runner

IDE knowledge agent (MCP)

What this is — A coding assistant in Cursor, VS Code, or another MCP host that searches your internal docs while you work. Why Indexify — the hosted MCP server exposes search, chunks, sections, and elements as tools so the IDE agent retrieves grounded evidence without you wiring REST in the editor. Platform — ingest and KB admin still use REST; MCP is the interactive retrieval surface in the IDE.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

What to create first
- Knowledge base — One KB per team or product area (runbooks, API guides, architecture docs). The MCP token is project-scoped; all KBs in that project are visible to indexify_list_kbs.
- Documents — Upload the corpora developers ask about most (internal wikis exports, OpenAPI specs, onboarding guides). Wait for jobs to complete before relying on search in the IDE.
- Project access token — Mint via client credentials with scopes at least kb:read, search:run, and docs:parsed:read (add element/section reads as needed). Store in the MCP client secret store—not in repo files.
Step-by-step implementation
1. Create KB & ingest — POST …/kbs, POST …/documents, poll GET …/jobs/{jobId} or use webhooks. Template: IDE agent KB example in Knowledge Base Design.
  - Outcome: searchable corpus before MCP setup.
2. Mint token — POST /oauth/token with client_credentials; assign read/search scopes only if the IDE agent must not ingest or delete.
  - Ops / platform team.
3. Configure MCP host — Point the client at Indexify's MCP base URL, path /api/mcp, Streamable HTTP transport, Authorization: Bearer <token>. Details: Connecting MCP clients.
  - Developer workstation.
4. Developer asks in IDE — e.g. “How do we rotate API keys?” or “Where is the refund webhook documented?”
  - User → MCP host LLM.
5. Agent retrieves — Typical loop: indexify_list_kbs (or fixed kbId) → indexify_search with includeSourceMetadata: true; optional indexify_search_enriched for refinement hints.
  - Enable hybrid and rerank in tool args when exact tokens matter.
  - Full loop: MCP agent retrieval loop.
  - MCP host → Indexify.
6. Drill down when needed — indexify_get_document_chunks, indexify_get_document_section, indexify_list_document_elements, or indexify_get_parsed_document for longer passages the search snippet omitted.
  - Agent tool loop.
7. Answer with citations — The model summarizes using retrieved text and cites documentId / chunk metadata from includeSourceMetadata.
  - Agent → user in IDE.
Endpoints & tools used
- Setup (REST)
- IDE retrieval (MCP)
  - indexify_list_kbs · indexify_search · indexify_search_enriched
  - indexify_get_document_chunks · indexify_get_document_section · indexify_list_document_elements
  - indexify_get_parsed_document
MCP vs REST in this scenario
- MCP — interactive IDE sessions: tool discovery, iterative search, and drill-down without hand-written HTTP in the editor.
- REST — CI ingest, KB administration, webhooks, and production services that need full route coverage and centralized logging.
Practical notes
- Use a dedicated credential per team or environment; rotate via project credentials without sharing personal JWTs.
- Cap tool iterations and wall-clock time in the MCP host so runaway agent loops do not exhaust rate limits.
- When answers must match production chat UX, mirror the same searchMode, rerank, and includeSourceMetadata settings your app uses on REST.