# Fault-Tolerant Document Ingestion Agent > Event-Driven AI Document Ingestion with Durable Orchestration **Status:** Measured to an artifact. **Role:** Backend Engineer Intern (NodesNow LLC). **Timeline:** Dec 2025 - Feb 2026. **Category:** Agentic AI & MCP. Production-grade event-driven AI ingestion engine using Inngest durable steps, NestJS, and dual pgvector/Neo4j storage to eliminate orphaned state during multi-stage document processing. ## Measurements | Value | What it measures | | --- | --- | | 8 | durable Inngest steps across 4 worker functions, each retryable on its own | | 2 stores | pgvector embeddings and Neo4j graph entities, written per step | ## The artifact that settles this Every stage is its own durable step, so a failure at one resumes there rather than restarting the run. Quoted from `Ravicha2/document-ingestion-agent` at `ingestion.ts#L24-L70` (a698cbf), `backend/src/inngest/functions/ingestion.ts#L24-L70`, verbatim: > const extractionResult = await step.run("extract-chunks", async () => { > await publish({ > channel: channelName, > topic: "progress", > data: { status: "PROCESSING", action: "SPLITTING",message: "Parsing PDF...", percent: 10 } > }); > > return await agentService.run(filePath); > }); > > const rawChunks = extractionResult.chunks || []; > if (rawChunks.length === 0) { > return { success: false, message: "No text found in PDF" }; > } > > // 2. Save Initial Chunks to DB (Text only, Embedding is NULL) > const chunkIds = await step.run("save-initial-chunks", async () => { > const ids = await toolsService.saveInitChunks(rawChunks, runId, filePath); > return ids; > }); > > // 3. Create Batches > const BATCH_SIZE = 10; > const idBatches = batchArray(chunkIds, BATCH_SIZE); > > const events = idBatches.flatMap((batchOfIds, index) => { > const payload = { > runId, > chunkIds: batchOfIds, > batchIndex: index, > totalBatches: idBatches.length, > channelName > }; > > return [ > { > name: "app/rag.process_vector_batch", > data: payload > }, > { > name: "app/rag.process_graph_batch", > data: payload > } > ]; > }); > // 5. Notify Client of "Queued" Status > await step.sendEvent("dispatch-parallel-jobs", events); Permalink: ## The intuition Achieve fault-tolerant, resumable document processing where failure at any stage recovers automatically. - The spark: Enterprise knowledge bases require multi-stage document processing pipelines (text extraction, OCR, semantic chunking, vector embedding, knowledge graph entity extraction) that run asynchronously across high volumes of files. - The naive failure mode: When background job queues experience transient network blips or LLM rate limits halfway through a multi-stage pipeline, the entire job aborts or requires re-running from scratch, leaving orphaned records and duplicate embeddings. ## The problem Orphaned database state and high re-computation costs when long-running AI pipelines fail midway. **Constraints** - Zero manual database cleanup required after transient external API outages - Full pipeline idempotency across all processing steps **Edge cases handled** - API rate limiting during parallel embedding generation - Partial writes leaving vector embeddings in pgvector without corresponding nodes in Neo4j - Inconsistent state from concurrent uploads of identical document revisions ## Why it was built this way Durable step execution with Inngest, step-level checkpointing, and dual-layer vector/graph storage. Decomposing the pipeline into Inngest durable steps guarantees that each step is idempotent and individually retryable with exponential backoff without re-running prior completed steps. **Trade-offs** - Inngest Durable Orchestration vs Traditional Message Queues — Inngest preserves intermediate step state in persistent storage. If step 4 (Graph ingestion) fails, only step 4 is retried, saving expensive LLM API tokens already spent in steps 1-3. (against: BullMQ / Celery standard queues) - Dual-Layer Persistence (pgvector + Neo4j) — Vector embeddings enable fast semantic similarity search, while Neo4j property graphs enable relational reasoning across extracted entities. (against: Vector-only storage) **Guardrails** - Idempotency keys on every ingestion job based on document content hash - Exponential backoff with jitter on all LLM and vector store network calls - Transactional rollback handlers for failed graph node insertions ## Outcomes Fault-tolerant durable orchestration and hybrid GraphRAG retrieval for document ingestion. **How it was verified** - Eight durable steps across four Inngest worker functions, each checkpointed and retryable with exponential backoff, so a failure resumes at the failed step rather than the start. - Containerized full stack (NestJS, Inngest server, Neo4j, pgvector) with Docker Compose for automated CI test suites. **Impact** - Removes manual engineer intervention on ingestion pipeline failures. - Enabled hybrid RAG search combining semantic vector similarity with multi-hop knowledge graph queries. **Takeaway:** Long-running AI agent workflows must be architected as durable state machines where steps checkpoint state rather than monolithic background scripts. ## Stack Inngest, NestJS, Neo4j, pgvector, PostgreSQL, React, Docker, TypeScript ## References - github: --- ## Fetching the rest of this site - This page as HTML: - Concise index: - Full dossier: - When to use this material: - API description (OpenAPI 3.1): - API reference: - Sitemap: