Shepherd: GraphRAG Compliance Engine

Reading
two-arm benchmark · 5 repos / 65 commits · precision 23.3% → 43.8% · tokens −62% · wall −27%
Datum
Researcher & Developer (UNSW Sydney), Jun 2026
Verdict
In progress · not yet settled

GraphRAG-Enhanced Architectural Decision Graph & Compliance Engine

End-to-end ADR violation detector for AI-generated code, ingesting source ASTs and architectural markdown documents into a Neo4j property graph to catch multi-file constraint conflicts.

Role
Researcher & Developer (UNSW Sydney)
Timeline
Jun 2026
Subject
Graph & Knowledge

Ravicha2/Shepherd

23.3% → 43.8%
pooled precision across the five-repo two-arm benchmark (baseline reviewer vs the CPT engine)
564.1M → 216.8M
reviewer tokens over the same 65 commits, 62% lower with the engine
5 repos / 65 commits
two-arm benchmark behind those pooled figures, 60 scored annotation sheets

  • Python
  • Neo4j
  • Cypher
  • Docker
  • uv
  • GitHub Actions
  • GraphRAG
The Shepherd project mark: a German shepherd’s head.
The Shepherd mark, supplied by the author — it names the project rather than settling a claim. The benchmark figures are carried from the two-arm annotation study, which no public permalink settles.

Core intuition & friction

Bridge the gap between natural-language architectural specifications and AST-level code reality.

The spark and the initial hypothesis

As AI code generators produce code at unprecedented velocity, software engineering teams lose visibility over whether newly generated implementations adhere to historical Architectural Decision Records (ADRs) buried in repo markdown files.

The naive failure mode

Naive vector search (RAG) retrieves semantically similar markdown paragraphs, but completely fails to detect multi-file dependency hierarchies, transitive imports, and strict layer boundaries (e.g. Domain layer importing Infrastructure).

The root problem encountered

Vector search cannot compute graph reachability or transitive import violations across code layers.

  • ADR specifications written in ambiguous natural language requiring structured constraint extraction
  • False positive alerts on legitimate mock dependencies in test suites
  • lean token consumption on architectural dependencies review
  • high precision and recall on AI code review.

Why built this way

Key architectural insight

Representing both code AST structures and ADR constraints as a unified Neo4j property graph enables deterministic Cypher path queries to verify architectural boundaries mathematically.

Neo4j Property Graph vs Flat Vector Embeddings

vs Flat vector embeddings / RAG

Vector similarity answers "is this conceptually related?", whereas Cypher queries answer "does module A depend on module B via path P?". Graph traversal eliminates vector hallucinations.

Tiered Violation Severity Engine

vs Binary pass/fail checks

Differentiates fatal architectural boundary violations from advisory suggestions, resolved in order by explicit supersession, specificity, recency, then human review.

Guardrails and defensive design

  • Automated AST parsing with robust syntax error handling and fallback modes
  • Isolated Cypher query execution with query timeouts
  • Violation lifecycle persistence, so a dismissed false positive stays dismissed across runs

Outcomes & verification

Verification benchmarks

  • cpt detect --json emits machine-readable violations for CI scripts and PR checks
  • Pooled over 5 repos / 65 commits, the engine lifted reviewer precision from 23.3% to 43.8% while reporting 80 findings instead of 133
  • Single annotator: every verdict is the author’s own call, and no inter-annotator agreement was measured

Real-world impact

  • Automated architecture governance in AI-augmented codebases, ensuring AI-written code obeys human architectural decisions.
  • Eliminated architectural erosion in rapidly evolving multi-agent repositories.

Core engineering takeaway

Non-deterministic AI code generation requires deterministic graph-based structural verification to maintain long-term software maintainability.

Get in touch

Master of IT at UNSW Sydney (WAM 83 / Distinction, graduating Dec 2026) · Open to full-time Applied AI & Backend Systems roles.

palm.ravicha@outlook.com

EmailCV (PDF)GitHubLinkedInSydney, Australia