When we first started building our engineering knowledge base, the approach seemed straightforward. We had ISO 8583 specification documents running into thousands of pages, along with internal project documentation and the codebase that implemented those specifications.
The initial plan was to parse the documents, chunk the content, generate embeddings, and expose the retrieved information to our agents.
It worked for straightforward questions. But once we started using it for specification generation and base-template auditing, the limitations became apparent.
The problem wasn't necessarily that the agent retrieved the wrong document. It was that the retrieved information often lacked the relationships needed to make a correct engineering decision.
Our internal data model was deliberately generalized to support multiple payment networks. We also simplified certain parts of the transaction flow because we were building a testing platform, not reproducing an entire live payment-processing system.
That meant a specification could be technically correct according to the network documentation but still be inappropriate for our implementation.
We needed the agent to understand not just what the specification said, but how it related to our code, data model, and test execution.
That was the point where I decided vector retrieval alone wasn't going to solve the problem.
1. Why vector retrieval wasn't enough
Semantic search relies on a foundational premise: questions that mean similar things will map closely in high-dimensional embedding space to documents that contain relevant answers. In general documentation, that premise often holds.
In technical engineering domains, however, text similarity is rarely enough. A data element might have different formats or conditional requirements depending on the message type, processing code, or network implementation. Retrieving the field definition alone doesn't necessarily tell an engineer—or an agent—how to configure it for a specific transaction flow.
When an agent relies strictly on semantic similarity, a few practical limitations quickly emerge:
- Contextual Hierarchy: A single payment field often acts as a composite container. Its internal elements may be conditionally mandatory only under specific transaction states or message classes. Searching for a term like "reversal processing" surfaces dozens of paragraphs across multiple chapters, but it rarely isolates the exact conditional rule buried in a specific appendix.
- Structural Constraints: Vector similarity cannot evaluate positional bitmasks or structural rules. If a message format requires unpacking a secondary bitmap before subsequent elements can even be addressed, semantic distance offers no help.
- Loss of Relationship: Chunking text into isolated blocks destroys the connections between parent definitions, child elements, and procedural validation rules.
Vector similarity measures linguistic likeness. Software engineering requires structural correctness. When the two are confused, agents produce confident suggestions that sound plausible but fail in execution.
2. The specification didn't match our data model
The problem became more pronounced when developer agents were tasked with drafting configurations or auditing test templates rather than simply answering informational queries.
In any enterprise platform, the internal software architecture does not mirror external regulatory standards verbatim. In our payment testing platform, the internal data model was intentionally generalized to create a unified abstraction across Visa, Mastercard, and regional debit networks. Some complex fields were flattened; others were derived dynamically during transaction orchestration.
When an agent was asked to audit a base template against network rules, a standard retrieval system fed the agent raw excerpts from external documentation. The agent then attempted to enforce external network rules directly against the testing platform's data structures, flagging false positives and recommending configurations that violated our internal architecture.
To produce useful engineering output, the agent needed to navigate three distinct layers:
- The External Specification: What the official network standard requires.
- The Operational Context: How message classes, processing codes, and lifecycles interact.
- The Internal Implementation: How those concepts are represented in our codebase, schemas, and test execution engine.
Our vector-based approach wasn't preserving the relationships we needed between those three layers.
3. Why we moved to Neo4j
To preserve those connections, we transitioned from vector-only retrieval to a knowledge graph using Neo4j.
We decomposed incoming technical specifications, Confluence documentation, and Java/Spring Boot domain classes into structured entities and relationships.
Below is a simplified conceptual model of how those relationships were structured:
// Conceptual schema for specification relationships
(:SpecDocument)-[:DEFINES]->(:MessageClass)
(:MessageClass)-[:REQUIRES_FIELD]->(:DataElement)
(:DataElement)-[:HAS_SUB_ELEMENT]->(:SubElement)
(:DataElement)-[:GOVERNED_BY]->(:ValidationRule)
(:DataElement)-[:MAPS_TO_INTERNAL_MODEL]->(:PlatformEntity)
With this structure in place, retrieval was no longer limited to proximity search. Instead, we combined hybrid full-text indexing (BM25) for exact numeric field tags with Cypher graph traversals:
- When an agent evaluates a specific data element, the retrieval engine traverses the graph to resolve outgoing validation rules and internal platform entity bindings in a single deterministic query.
- Over 21,400 technical knowledge chunks were indexed and anchored to specific entities in the graph, giving the agent access to the structural neighborhood of that field rather than an isolated snippet of text.
4. Exposing domain knowledge through MCP
Storing structured knowledge was only part of the solution; developers needed their agents to interact with that knowledge while working in their IDEs.
Rather than dumping entire chapters into prompt contexts, we exposed the knowledge graph through a set of Model Context Protocol (MCP) tools and reusable agent skills.
The tools provide granular, task-oriented operations. For example, conceptual tools included:
get_field_definition(network, field_num): Returns canonical field formats, length types, and character encodings.trace_field_dependencies(field_id): Returns graph-derived conditional constraints and related validation rules.reconcile_to_canonical_model(spec_field): Translates external network rules into internal platform database entities.audit_base_template(template_json): Evaluates a proposed configuration against known validation rules and flags discrepancies.
By delegating retrieval to structured MCP tool calls, the agent operates with a lean context window. It queries only the specific data elements required for its immediate task, verifying requirements step by step.
5. Verification and safeguards
A key takeaway from this work is that generative models should never be the final arbiters of correctness in high-stakes platforms. LLMs are helpful for synthesis, cross-referencing, and drafting, but their output must be gated by deterministic validation.
In our workflow:
- The agent synthesizes a proposed specification update or template audit using graph context retrieved via MCP.
- The output is emitted as a structured JSON payload.
- The payload is validated against strict schema rules, foreign-key relationships, and platform integrity checks.
- Only when deterministic checks pass is the proposed configuration staged for human engineer review.
Deterministic validation code remains the gatekeeper; the agent serves as an accelerator for navigating complex domain specifications.
Conclusion: Applied AI as systems engineering
What started as a document retrieval problem turned into a much broader engineering problem.
Vector search helped us find relevant information, but relevance alone wasn't enough. We needed to understand how specifications related to our internal implementation, how different requirements interacted, and where the constraints of our testing platform differed from those of the actual payment networks.
Moving to a knowledge graph gave us a way to represent those relationships explicitly. Exposing that knowledge through MCP allowed our agents to use it in specification generation and auditing workflows.
There are still limitations. Retrieved information can be incomplete, relationships can be incorrectly modeled, and an LLM can draw the wrong conclusion even when the underlying context is relevant.
That's why I see retrieval as one part of the solution. The more difficult engineering work is building the surrounding systems, tools, and validation that make the information usable, consistent, and verifiable.
For me, that is where applied AI becomes an engineering discipline rather than simply an exercise in prompting a language model.