indexify.dev - Document Processing for AI Agents

Production Readiness

Reliability and operations

Production integrations should combine idempotent writes, respect for rate-limit headers, structured logging, and alerting on job and webhook health. This section ties together Authentication, Jobs and webhooks, and Retrieval—without requiring any platform-internal access.

If you’re reading this as a reference, feel free to jump around—each section is written to stand on its own, and the left-hand search is the fastest way to find a specific endpoint or concept.

Retries, rate limits, and idempotency

Retry only when the API contract says it is safe; combine Idempotency-Key on uploads (Idempotency) with bounded exponential backoff. Interpret limits using Throttling headers.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

Retries
- Use jittered exponential backoff on 429 and transient 5xx; honor Retry-After when present.
- Send Idempotency-Key on create/upload operations you may retry automatically.
- Centralize retry policy in one HTTP layer and classify fatal vs retryable errors from Problem JSON.

Throttling headers and how to interpret them

Responses may include X-RateLimit-* and Retry-After so your client can proactively pace before hitting 429. Names and presence are defined in the API reference for each route.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

Header meanings
- X-RateLimit-Limit — Max requests in the active window for your key.
- X-RateLimit-Remaining — Left in window; near 0 → slow down.
- X-RateLimit-Reset — Unix seconds when the window refills.
- Retry-After (429) — Seconds to wait before retrying that route/key.
Client behavior
- Pace proactively when remaining is low and reset is far.
- On 429: Retry-After first, then jittered backoff within a budget.
- Headers are per-response; do not reuse stale counts under concurrency.

Monitoring, alerting, and email notifications

You own dashboards on top of public signals: job outcomes, search latency, webhook delivery logs, and auth error rates. The product can also email failure and recovery notices when project monitoring is enabled—see When failure emails fire.

Tip: the snippets are meant to be copy/paste friendly—start with the happy path, then intentionally poke at scopes, missing fields, and bad inputs so you know exactly what your app will see in production.

What to monitor
- Jobs — success vs failure, stage durations (Jobs).
- Search — latency percentiles, rerank errors, empty-result rate (Retrieval).
- Webhooks — delivery failures, retry exhaustion, consumer backlog (Webhooks).
- Auth — spikes in 401/403 from bad rotation or scope drift (Authentication).
Project settings
- Enable project.settings.monitoringEnabled and optional notificationEmail via the documented PATCH project route (API reference); the snippet below opens api-docs / Runner.
- You configure webhook URLs and retry policy through the public KB webhook API—no internal services required.

Configure where monitoring notifications are sent (developer console or PATCH project API). Failure and recovery emails are delivered by the product when monitoring is enabled—you set the destination; no mail infrastructure to run yourself. Full settings object (both fields):

curl -s -X PATCH "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJkZXYifQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "monitoringEnabled": true,
      "notificationEmail": "alerts@yourcompany.com"
    }
  }'

Open endpoint in api-docs Open in API Runner

Partial settings: only change monitoringEnabled; notificationEmail stays as stored (deep merge).

curl -s -X PATCH "https://api.indexify.dev/projects/550e8400-e29b-41d4-a716-446655440001" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJkZXYifQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": { "monitoringEnabled": false }
  }'

Open endpoint in api-docs Open in API Runner

Public API health (no auth; 200 ok/degraded or 503 down — JSON HealthResponse; X-RateLimit-* after rate-limit pass)

curl -s "https://api.indexify.dev/health"

Open endpoint in api-docs Open in API Runner

When failure and recovery emails are sent

Email notifications are high-level signals for humans: one alert when a class of problem starts for a KB or webhook, and a recovery message when it clears—designed to avoid spamming on every retry. Pair with your own metrics; event names align with Webhook events.

Use the bullets as a checklist: read once, then keep this open while you implement so you don’t miss the small but important details (like token type, scopes, and strict query rules).

Recipients
- Monitoring on → settings.notificationEmail if set, else project owner email.
Job failures
- Failure mail on terminal job.failed (deduped per KB while failing).
- Recovery mail after job.completed if a prior failure was reported for that KB.
- Partial parse/chunk: separate mail for which formats failed (job.parsing_partial_failure / job.chunking_partial_failure).
Webhooks
- Failure when retries exhaust; recovery when a delivery succeeds after a failing state.
- Design: one failure notice per issue scope, one recovery when healthy — not per retry.