Fault-Tolerant Document Ingestion Agent

Reading
8 durable steps / 4 worker functions · each stage retries alone
Datum
Ravicha2/document-ingestion-agent. ingestion.ts#L24-L70 @ a698cbf
Verdict
Measured · holds

Event-Driven AI Document Ingestion with Durable Orchestration

Production-grade event-driven AI ingestion engine using Inngest durable steps, NestJS, and dual pgvector/Neo4j storage to eliminate orphaned state during multi-stage document processing.

Role
Backend Engineer Intern (NodesNow LLC)
Timeline
Dec 2025 - Feb 2026
Subject
Agentic AI & MCP

Ravicha2/document-ingestion-agent

8
durable Inngest steps across 4 worker functions, each retryable on its own
2 stores
pgvector embeddings and Neo4j graph entities, written per step

  • Inngest
  • NestJS
  • Neo4j
  • pgvector
  • PostgreSQL
  • React
  • Docker
  • TypeScript
Fault-Tolerant Document Ingestion Agent, captured at its live state.
github.com/Ravicha2/document-ingestion-agent — captured at the current default branch.

Durable run, in order

Ravicha2/document-ingestion-agenta698cbfL24–L70

Every stage is its own durable step, so a failure at one resumes there rather than restarting the run.

  1. 01extract-chunksdocument-ingestion-agent#L24
  2. 02save-initial-chunksdocument-ingestion-agent#L40
  3. 03dispatch-parallel-jobs ⇢document-ingestion-agent#L70

ingestion.ts#L24-L70 @ a698cbf

Core intuition & friction

Achieve fault-tolerant, resumable document processing where failure at any stage recovers automatically.

The spark and the initial hypothesis

Enterprise knowledge bases require multi-stage document processing pipelines (text extraction, OCR, semantic chunking, vector embedding, knowledge graph entity extraction) that run asynchronously across high volumes of files.

The naive failure mode

When background job queues experience transient network blips or LLM rate limits halfway through a multi-stage pipeline, the entire job aborts or requires re-running from scratch, leaving orphaned records and duplicate embeddings.

The root problem encountered

Orphaned database state and high re-computation costs when long-running AI pipelines fail midway.

  • API rate limiting during parallel embedding generation
  • Partial writes leaving vector embeddings in pgvector without corresponding nodes in Neo4j
  • Inconsistent state from concurrent uploads of identical document revisions
  • Zero manual database cleanup required after transient external API outages
  • Full pipeline idempotency across all processing steps

Why built this way

Key architectural insight

Decomposing the pipeline into Inngest durable steps guarantees that each step is idempotent and individually retryable with exponential backoff without re-running prior completed steps.

Inngest Durable Orchestration vs Traditional Message Queues

vs BullMQ / Celery standard queues

Inngest preserves intermediate step state in persistent storage. If step 4 (Graph ingestion) fails, only step 4 is retried, saving expensive LLM API tokens already spent in steps 1-3.

Dual-Layer Persistence (pgvector + Neo4j)

vs Vector-only storage

Vector embeddings enable fast semantic similarity search, while Neo4j property graphs enable relational reasoning across extracted entities.

Guardrails and defensive design

  • Idempotency keys on every ingestion job based on document content hash
  • Exponential backoff with jitter on all LLM and vector store network calls
  • Transactional rollback handlers for failed graph node insertions

Outcomes & verification

Verification benchmarks

  • Eight durable steps across four Inngest worker functions, each checkpointed and retryable with exponential backoff, so a failure resumes at the failed step rather than the start.
  • Containerized full stack (NestJS, Inngest server, Neo4j, pgvector) with Docker Compose for automated CI test suites.

Real-world impact

  • Removes manual engineer intervention on ingestion pipeline failures.
  • Enabled hybrid RAG search combining semantic vector similarity with multi-hop knowledge graph queries.

Core engineering takeaway

Long-running AI agent workflows must be architected as durable state machines where steps checkpoint state rather than monolithic background scripts.

Get in touch

Master of IT at UNSW Sydney (WAM 83 / Distinction, graduating Dec 2026) · Open to full-time Applied AI & Backend Systems roles.

palm.ravicha@outlook.com

EmailCV (PDF)GitHubLinkedInSydney, Australia