Preview / closed beta. Interested? Email info@indexify.dev.

Developer Guide

Start Here

A complete, hands-on guide for developers building agents: start from scratch, get a working integration shipped, then level it up with advanced retrieval, structured document access, and the operational details you’ll want in production.

Start Here

Introduction

Indexify is a hosted, closed SaaS platform for retrieval-augmented generation (RAG) and agents: it runs the full document → parse → chunk → index → search pipeline for you so your team does not operate parsers, embedding jobs, or vector stores. You integrate only through the documented HTTP API (OpenAPI) and the hosted MCP endpoint—there is no open-source runtime to fork and no supported contract beyond those surfaces (do not rely on undocumented behavior or assumed internals). Use it when you are building agents or backends that need grounded answers, citations, stable chunk and document ids, and optional structure-aware navigation (elements, sections, tables, figures, relationships). In practice you create projects and knowledge bases, upload over HTTPS, track jobs, optionally use webhooks, then call search and related reads—or MCP tools that mirror the same API.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

Agent Ready

Indexify is agent-ready on the hosted product: agents can discover capabilities, call retrieval tools, and follow the same OpenAPI contract as your backend—without you operating parsers, chunkers, or vector indexing yourself. You ship faster by combining OpenAPI (schemas and errors), MCP tools (streamlined retrieval for agent hosts), and a downloadable agent guide for LLM context.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • What “agent-ready” means here
    • A strict contract: the OpenAPI spec drives the API docs and runner, and gives agents/tooling an authoritative schema source.
    • First-class retrieval tools: MCP exposes read-oriented tools that wrap the same REST operations (search, chunks, elements, sections, tables, figures), optimized for agent loops.
    • Copy-ready context: a short markdown guide summarizes MCP vs HTTP, golden paths, and operational rules for agents.
  • How this helps your development workflow
    • Prototype faster: use API Runner to validate auth, ingestion, and search flows in minutes before wiring your backend.
    • Move retrieval into tools: let your agent do think → retrieve → evaluate → refine with MCP tools, instead of hardcoding one-shot RAG queries.
    • Reduce integration drift: generate clients/validators from OpenAPI and keep request/response shapes consistent across services.
    • Make agents more deterministic: stable ids + citation-ready results encourage “show your work” answers and traceable tool calls.
  • Where to start (agent-focused)
  • Agent ready artifacts

Core Concepts

Four ideas you use in every integration: who is calling (human vs machine), where data lives (project vs KB), how files become searchable (parse → chunk → index), and how you observe progress (jobs and webhooks).

If you’re reading this as a reference, feel free to jump around—each section is written to stand on its own, and the left-hand search is the fastest way to find a specific endpoint or concept.

Developer vs Project identity

Humans use a developer JWT for account and project setup. Services use a project access token from POST /oauth/token for all KB, document, job, search, webhook, and structured document routes.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Rules
    • Never send project client_secret or project access tokens to browsers or untrusted devices.
    • Issue machine tokens from a backend you control; cache until shortly before expires_in, then refresh.
    • Wrong token type on a path usually yields 401 or 403—see Authentication and Security.

Project and Knowledge Base boundaries

Every KB-scoped data-plane URL starts with /projects/{projectId}/kbs/{kbId}/. The bearer token must belong to that project. Search always targets a single kbId—use multiple KBs or multiple requests when you need isolation.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Why it matters
    • Chunks and their embeddings/index are scoped to one KB—retrieval ranks within that KB’s index only.
    • Parsed outputs, chunk metadata, and multimodal structured elements are stored per KB—not shared across KBs.
    • Mixing unrelated corpora in one KB blurs ranking and weakens citations.
    • Credentials are created per project; use separate projects for prod/stage/dev when billing and access must not overlap.

Pipeline mental model

This is the RAG ingestion path the platform runs for you: after POST …/documents, work proceeds parse → chunk → index as configured on the KB (settings.pipeline, parse, chunk, index). Each upload is tied to a job you poll or learn about via webhooks—you do not host that pipeline yourself.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Observable stages
    • Parse — produces stored representations (e.g. markdown/json); json triggers structured elements/tables/figures.
    • Chunk — splits content for embedding; settings affect snippet size and metadata attached to chunks.
    • Index — writes vectors (per embeddingConfig) so POST …/search can run.
  • Details: Document Ingestion and Jobs and Webhook Lifecycle.

Retrieval outcomes

POST …/search returns ranked hits; enable includeSourceMetadata in the request when you need stable chunk/document fields for citations or UI. Multimodal and structured document endpoints add tables, figures, and section navigation—see Structured Knowledge.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

Quickstart: First Working Integration

Eleven starred cURL steps—the same endpoints as Select Quickstart endpoint in API Runner. Replace placeholders with your ids and secrets. Each block includes Open endpoint in api-docs and Open in API Runner when the URL matches the public spec.

Follow the eleven starred steps in order—the same endpoints as Select Quickstart endpoint in API Runner. Replace sample ids and tokens as you progress.

1) Sign Up

curl -s -X POST "https://api.indexify.dev/auth/signup" \
  -H "Content-Type: application/json" \
  -d '{
    "email": "developer@example.com",
    "password": "SecurePassw0rd!"
  }'

2) Login

curl -s -X POST "https://api.indexify.dev/auth/login" \
  -H "Content-Type: application/json" \
  -d '{
    "email": "developer@example.com",
    "password": "SecurePassw0rd!"
  }'

3) Create Project Optional Idempotency-Key: same key replays only for the same normalized body (name, description, settings defaults).

curl -s -X POST "https://api.indexify.dev/projects" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJkZXYifQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: f1e2d3c4-b5a6-7890-abcd-ef1234567890" \
  -d '{
    "name": "my-agent-project",
    "settings": {
      "monitoringEnabled": true
    }
  }'

4) Create Project Credentials Optional Idempotency-Key: same key may replay cached 201/409 for ~24h when body fingerprint matches (see OpenAPI).

curl -s -X POST "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/credentials" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJkZXYifQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: f1e2d3c4-b5a6-7890-abcd-ef1234567890" \
  -d '{
    "name": "backend",
    "scopes": [
      "kb:read",
      "kb:write",
      "kb:delete",
      "docs:read",
      "docs:ingest",
      "docs:parsed:read",
      "docs:delete",
      "jobs:read",
      "jobs:cancel",
      "jobs:retry",
      "search:run",
      "webhooks:read",
      "webhooks:write",
      "webhooks:delete",
      "webhooks:test"
    ]
  }'

5) Get Access Token

curl -s -X POST "https://api.indexify.dev/oauth/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "grant_type=client_credentials" \
  --data-urlencode "client_id=idx_sample_client_id_01" \
  --data-urlencode "client_secret=idx_sample_client_secret_01"

6) Create Knowledge Base Optional Idempotency-Key: same key may replay cached 201/404/409 for ~24h when KB body fingerprint matches (see OpenAPI).

curl -s -X POST "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: f1e2d3c4-b5a6-7890-abcd-ef1234567890" \
  -d '{
    "name": "support-kb",
    "description": "A knowledge base for document processing",
    "settings": {
      "pipeline": [
        "parse",
        "chunk",
        "index"
      ],
      "parse": {
        "outputFormats": [
          "markdown",
          "json"
        ],
        "pdf": {
          "tableStructureMode": "ACCURATE",
          "isOcrEnabled": "no"
        }
      },
      "chunk": {
        "attachMetadata": true,
        "strategy": {
          "method": "hybrid",
          "size": 800,
          "mergePeers": true
        }
      },
      "index": {
        "embeddingConfig": {
          "model": "openai:text-embedding-3-small"
        }
      },
      "search": {
        "rerankConfig": {
          "provider": "cohere",
          "config": {
            "model": "rerank-v3.5"
          }
        }
      }
    }
  }'

7) Create/Upsert Webhook

curl -s -X PUT "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/webhook" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-app.example.com/indexify/webhook",
    "active": true,
    "events": [
      "job.pending",
      "job.started",
      "job.parsing_started",
      "job.parsing_format_completed",
      "job.parsing_structure_ready",
      "job.parsing_partial_failure",
      "job.parsing_completed",
      "job.chunking_started",
      "job.chunking_format_selected",
      "job.chunking_partial_failure",
      "job.chunking_completed",
      "job.indexing_started",
      "job.indexing_completed",
      "job.completed",
      "job.failed",
      "job.canceled"
    ],
    "retryPolicy": {
      "maxAttempts": 1,
      "backoffSeconds": 30
    }
  }'

8) Upload Documents Defaults: 10 MB per file, 10 files per request; deployment limits may differ (413 if exceeded). Wait for webhook job.completed / job.failed or poll GET .../jobs/{jobId} before parsed/chunks/search steps.

curl -s -X POST "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/documents" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -F "file=@./handbook.pdf"

9) Get Parsed Document

curl -s -X GET "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/documents/770e8400-e29b-41d4-a716-446655440003/parsed" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Accept: text/markdown"

10) Get Document Chunks

curl -s -X GET "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/documents/770e8400-e29b-41d4-a716-446655440003/chunks?limit=20&format=markdown" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Accept: application/json"

11) Run Search

curl -s -X POST "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001/kbs/660e8400-e29b-41d4-a716-446655440002/search?format=markdown" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtYWNoaW5lIn0.dBjftJeZ4CVP-mB92K27uhbUJU1p1b_wW1gFWFOEjXk" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "PTO policy",
    "limit": 10,
    "settings": {
      "rerank": { "isEnabled": "yes", "topK": 5 },
      "retrieveTopK": 50,
      "searchMode": "hybrid",
      "includeSourceMetadata": true,
      "includeGrounding": false,
      "diagnostics": false,
      "groupBy": "document",
      "elementTypes": [],
      "tableQuery": { "mode": "cell_contains", "cellContains": "PTO" }
    }
  }'

Prerequisites

  • Before step 1
    • A sample file (PDF, DOCX, HTML, CSV, …) for Upload Documents (step 8).
    • Store projectId, kbId, documentId, clientId, and clientSecret from each response as you progress.

Starred endpoint order

  • Run in this order
    • 1. Sign Up2. Login3. Create Project4. Create Project Credentials5. Get Access Token.
    • 6. Create Knowledge Base7. Create/Upsert Webhook8. Upload Documents.
    • After ingest completes (webhook or job status), run 9. Get Parsed Document10. Get Document Chunks11. Run Search.
    • Use the same starred chips in API Runner to pre-fill each request.
  • Scopes and errors: Authentication and Security.

What to do next

Data Model

Clear entities, clear ownership, predictable retrieval. Indexify’s model is designed so credentials, quotas, and security boundaries stay stable while you configure ingestion and retrieval per knowledge base.

If you’re reading this as a reference, feel free to jump around—each section is written to stand on its own, and the left-hand search is the fastest way to find a specific endpoint or concept.

Reading left → right: a Developer owns Projects; each project holds one or more Knowledge bases. A KB relates directly to both Documents (ingestion) and an optional Webhook (job lifecycle)—not to one through the other.

How it fits together

Think in three layers: identity (developer + project), knowledge (KB + documents), and operations (jobs + webhooks). You call most APIs with a project access token (client credentials) and you scope retrieval to one KB at a time.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Identity and access
    • Developer — human account; used to create projects and credentials (developer JWT).
    • Project — the top-level container for billing/quotas/credentials and one or more KBs (project access token).
    • Credential — (clientId, clientSecret) used to mint short-lived project access tokens via POST /oauth/token.
  • Knowledge and retrieval
    • Knowledge base (KB) — where configuration lives (parse/chunk/index/search/structure settings) and where embeddings/indexes are stored.
    • Document — an uploaded source file that is parsed/chunked/indexed into a KB; documents produce jobs and yield searchable chunks and structured elements.
    • Search is always scoped to one KB — queries target /projects/{projectId}/kbs/{kbId}/search; use multiple KBs when you need isolation.
    • Once ingest has finished, the objects you fetch read-only are summarized in Processed KB read model.
  • Operations and observability
    • Job — asynchronous processing unit created by uploads and other operations; poll or subscribe to lifecycle events.
    • Webhook — optional KB-level delivery configuration to receive job lifecycle events reliably.

Entities & cardinality

Most relationships are one-to-many so you can keep boundaries explicit: a developer owns projects, projects contain KBs, KBs contain documents, and documents create jobs.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Rule-of-thumb cardinalities
    • Developer → Projects: one developer can own many projects.
    • Project → Knowledge bases: one project can contain many KBs.
    • Knowledge base → Documents: one KB can contain many documents.
    • Document → Jobs: one document upload creates at least one job (and may create additional processing jobs depending on configuration/operations).
    • Knowledge base → Webhook: at most one webhook configuration per KB (upsert-style).
  • What each entity is for
    • Developer: initial setup; human-driven operations like creating projects and credentials.
    • Project: security + billing boundary; where machine identity lives via OAuth client credentials.
    • Knowledge base: retrieval boundary; you tune pipeline and ranking per KB so relevance/citations remain stable.
    • Document: the source-of-truth artifact you ingest (a file + metadata) that becomes searchable and citable.
    • Job: the “receipt” and progress tracker for asynchronous work (parse/chunk/index/structure).
    • Webhook: the integration point to make ingestion event-driven and reliable at scale.

Modeling guidance (generic)

There’s no single correct mapping; aim for stable boundaries and predictable retrieval. These patterns hold for most products.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Choose boundaries deliberately
    • Use a project per environment (prod/stage/dev) when access control, quotas, or billing must not overlap.
    • Use a KB per content surface when relevance and citations must stay clean (help center vs runbooks vs policies vs tickets).
    • Avoid mixing unrelated corpora into one KB just because it’s convenient—ranking and citations will drift.
  • Treat jobs as first-class
    • Build your ingestion pipeline as an async state machine: upload → store jobId → poll or listen for webhook → mark ready in your system.
    • Idempotency + retries matter: use idempotency keys where supported and design webhooks to handle duplicate deliveries.
  • Keep credentials scoped to services
    • Use separate OAuth clients (credentials) per service role when you want clear blast radius and scope boundaries (ingest vs retrieval vs webhooks).
    • Never place project secrets or project access tokens in browsers; mint machine tokens server-side.

Real-world modeling examples

Three “steal this” mappings from real products into Projects + KBs. The vibe: keep retrieval boring and predictable by searching one KB at a time (one KB per surface).

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • 1) Education tutor agent
    • Project boundary (what absolutely shouldn’t mix)
    • Usually: one project per district/school. Or do one project per environment (staging vs prod) if you want a clean wall for access control/quotas.
    • KBs (think “separate drawers”)
    • - curriculum-kb: standards, syllabus, pacing guides, rubrics, learning objectives (the “what should we cover?” stuff).
    • - course-materials-kb: slides, worksheets, textbook chapters, labs, example problems, teacher notes (the actual teaching content).
    • - safety-policy-kb: age rules, refusal/escalation guidance, what to do about cheating/self-harm, citation requirements (how the tutor should behave).
    • How you actually use it (a normal flow)
    • 1. Dump course PDFs/slides into course-materials-kb.
    • 2. Put standards/rubrics into curriculum-kb (they rank differently, so keep them separate).
    • 3. For “Explain X / show an example”, search course-materials-kb so citations point at the lesson content.
    • 4. For “What are we covering / what’s required”, search curriculum-kb.
    • Common mistakes
    • - Shoving everything into one KB “because it’s easier” (ranking drifts; citations feel kinda random).
    • - Searching multiple KBs behind the scenes and blending results without making it obvious where answers came from.
  • 2) SaaS customer support agent
    • Project boundary (tenant isolation)
    • If you’re multi-tenant, the clean default is one project per customer tenant so data + credentials never mix.
    • KBs (match how support actually works)
    • - help-center-kb (public): FAQs, setup guides, troubleshooting articles, onboarding docs.
    • - runbooks-kb (internal): SOPs, escalation paths, admin-only steps, scripts, checklists, “if X then Y”.
    • - release-notes-kb (changes): changelogs, migrations, deprecations, known issues.
    • How you actually use it (two safe modes)
    • 1. Customer chat: only search help-center-kb (no chance of leaking internal ops).
    • 2. Support dashboard: hit runbooks-kb first, then check release-notes-kb for “did we change something recently?”
    • 3. If your docs live in a CMS, wire a webhook so publishes trigger re-indexing and you’re not answering with stale docs.
    • Common mistakes
    • - Mixing internal runbooks into the same KB as public docs (this is how accidental leakage happens).
    • - Treating release notes as “just more docs” instead of a separate KB (people query them differently).
  • 3) Compliance / policy assistant (HR + Security)
    • Project boundary (don’t mix jurisdictions)
    • Do one project per region/business unit (e.g. EU vs US) so the assistant doesn’t “helpfully” cite the wrong rules.
    • KBs (split by what people mean when they ask)
    • - hr-policy-kb: handbook, PTO/benefits, onboarding/offboarding, code of conduct.
    • - security-policy-kb: access control, incident response, data classification, vendor risk, security training.
    • - legal-regs-kb: GDPR/SOC2/ISO material, retention rules, contract templates, audit mappings.
    • How you actually use it (keep it honest)
    • 1. For “what’s the rule?” questions, pick the one KB that matches intent (HR vs security vs legal).
    • 2. Keep citations strict: HR answers should cite hr-policy-kb, not some other doc that happens to mention the same phrase.
    • 3. Use webhooks so policy updates re-index quickly—stale policy answers are worse than no answer.
    • Common mistakes
    • - Combining multiple regions into one project and trying to “prompt your way out” of it (eventually you’ll get wrong-scope citations).
    • - Throwing external regs and internal policies into one KB (different language, different retrieval goals).

Processed KB read model

Assume every document in the KB has completed parse → chunk → index (with json in parse.outputFormats when your product needs elements, tables, and figures), and parsing has produced structured elements. This section sketches what to call and why; every route below links to its OpenAPI entry—schemas, scopes, and pagination live in the API reference (Data plane).

If you’re reading this as a reference, feel free to jump around—each section is written to stand on its own, and the left-hand search is the fastest way to find a specific endpoint or concept.

Three layers

Most integrations think in two layers: corpus (per-document material) and query (ranked search and job-scoped artifacts).

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Layer 1 — Corpus
    • Per-document material: everything hangs off documentId—parsed bodies, chunks, typed elements, sections, tables, figures, relationships, processing artifacts.
  • Layer 2 — Query
    • Search ranks evidence across the KB.
    • Job artifacts list outputs for a specific job when you need that slice.
  • Conceptual background: Structured Knowledge and Retrieval fundamentals.

Layer 1 — Corpus per document

These routes are all scoped to /projects/{projectId}/kbs/{kbId} and a documentId (except the document list). Responses are paginated where noted in the spec.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

Reading left → right: one documentId in the KB (list documents is KB-scoped, no documentId in path).

Layer 2 — Query surfaces

These endpoints summarize or traverse the corpus without requiring you to assemble every document first.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

Reading left → right: use Search for ranked hits (content in the response format) plus optional fields for stable citations; Sections & elements for outline navigation and typed reads within a document; Job artifacts for outputs scoped to one job.

How the pieces connect

Use documentId as the anchor for parsed text, chunks, and structure APIs. Search returns rows that point back to those IDs (and chunk indices) so you can deep-link into GET …/chunks or structured routes.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

  • Practical rule of thumb
    • Start from search for user-facing Q&A; use groupBy: section or elementTypes when you need section- or modality-scoped hits.
    • Drop to per-document routes when you need full parsed content, adjacent chunks, or fine-grained element / section trees.
    • Schema details (Chunk, Element, SearchResult, Element, …) live in API reference components.