Operating Context & The Engineering Problem
When we first started building our engineering knowledge base, the goal was straightforward: help developer agents navigate 1,000+ page ISO 8583 specification PDFs, internal project documentation, and legacy source code to assist with base-template configuration and test orchestration.
Our testing platform derives simulated transactions across 16 global payment networks from base templates representing thousands of field definitions. However, our initial experiments with naive vector-only semantic search quickly ran into practical limitations:
- Text Similarity Was Insufficient: In payment specifications, text similarity rarely tells you how to configure a field. A data element might have different formats or conditional requirements depending on the message type, processing code, or network implementation. Searching for a term like "reversal processing" surfaced dozens of paragraphs across multiple chapters, but failed to isolate the exact structural constraint or bitmask requirement needed for a specific transaction flow.
- The Specification Didn't Match Our Data Model: Our internal platform data model was deliberately generalized to support multiple payment networks, and certain production transaction flows were intentionally simplified for testing. External specifications could be technically correct according to network documentation, but inappropriate for our testing platform. Vector search simply retrieved raw text verbatim without bridging the gap between external rules and our internal data structures.
- Loss of Structural Context: Splitting technical PDFs into arbitrary 500-token chunks destroyed parent-child relationships between message headers, composite data elements, and conditional business rules.
Architecture & System Boundaries
To address these limitations, I redesigned the retrieval architecture around a Neo4j knowledge graph coupled directly to developer agents via Model Context Protocol (MCP):
- Section-Aware Ingestion Pipeline: - Implemented custom PDF parsing that segmented specifications into functional chapters (Message Template Definitions, Data Element Dictionaries, and Processing Flow Rules), stripping boilerplate headers and legal disclaimers. - Parsed Java/Spring Boot domain models and enums from our codebase to map internal platform entities alongside external specifications.
- Neo4j Knowledge Graph Model: - Structured entities and relationships to represent domain reality: `SpecDocument`, `PaymentFlow`, `MessageClass`, `DataElement`, and `ValidationRule`. - Connected nodes via explicit relationships: `CONTAINS`, `REQUIRES`, and `MAPS_TO_INTERNAL_MODEL`. - Enabled multi-hop graph queries that retrieve a field definition together with its conditional constraints and internal platform mappings.
- Hybrid Retrieval Engine: - Combined BM25 full-text indexing (for exact numeric ISO field codes and tags) with dense vector embeddings (for semantic intent) and Cypher queries for deterministic relationship expansion. - Indexed over 21,400 technical knowledge chunks across specification rulebooks, internal codebases, and documentation.
- Model Context Protocol (MCP) Toolset: - Exposed the knowledge graph to developer IDE agents (Cursor, Claude Code, Antigravity) through a set of focused MCP tools and reusable agent skills. - Tools provided targeted operations: fetching canonical field definitions, tracing conditional dependencies, reconciling external specifications to internal platform models, and auditing draft base templates.
- Deterministic Safeguards & Feedback: - Paired agent generation with deterministic validation: proposed configurations were validated against strict schema rules and foreign-key constraints before engineer review. - Added an agent-reported relevance feedback loop to evaluate retrieval usefulness and detect context drift over time.
Consequential Trade-offs & Alternative Paths
- Vector Search vs. Knowledge Graph: Vector retrieval was trivial to set up, but repeatedly hallucinated on hierarchical financial specifications. Building a Neo4j knowledge graph required upfront schema modeling and query curation, but provided explicit relationship tracking and eliminated structural omissions.
- Tool-Assisted Retrieval (MCP) vs. Large Context Windows: Dumping entire specification chapters into LLM context windows caused attention degradation and high latency. Providing granular MCP tools allowed agents to lazily fetch only the specific data elements required for their immediate task.
- Assistance vs. Autonomous Generation: Generative models excel at cross-referencing and drafting, but should not be the final arbiters of platform correctness. Restricting agent output to structured JSON validated by deterministic compiler checks ensured production safety.
Verified Production Outcomes
Staff-Level Engineering Retrospective
- In complex enterprise systems, natural language text alone does not capture system truth. The connections between specifications, code, and business rules are relational.
- The most valuable engineering work in applied AI is not prompt tweaking, but building the surrounding systems, domain models, and validation gates that make the information usable and verifiable.