Operating Context & The Engineering Problem
Enterprise payment testing relies on base templates and reference data representing approximately 9,000 ISO 8583 field definitions across 16 payment networks. Child transaction templates are derived from these base templates to validate transaction switching, authorization, and clearing rules.
Historically, changes to this reference data were stacked sequentially as Liquibase database migrations. Even with only two networks configured, this workflow created serious operational bottlenecks:
- 10+ Minute Replay Bottleneck: Bootstrapping a local development environment or test database required replaying a deep, cumulative migration chain that frequently exceeded 10 minutes.
- Locking & Crash Vulnerability: Bootstrapping frequently crashed midway due to execution timeouts and lock contention, requiring manual intervention to release migration locks in PostgreSQL.
- Zero State Visibility: Developers could not inspect the final reference-data state without executing the entire migration history.
- Coordination Overhead: Concurrent feature branches constantly collided on Liquibase changeset sequences, creating painful merge conflicts that blocked deployments.
The existing approach was simply not a sustainable developer workflow for expanding network coverage.
Architecture & System Boundaries
To resolve the bottleneck, I proposed and implemented a bidirectional Configuration-as-Code platform that decoupled business reference data from database schema evolution:
- Decoupling DDL from Reference Data: - Retained Liquibase strictly for database schema evolution (DDL). - Extracted volatile payment reference data into version-controlled, human-readable canonical JSON specifications.
- Bidirectional Synchronization with Clear Authority: - Supported two complementary workflows: direct relational database editing (for rapid UI previewing and relational inspection) and JSON editing (for Git pull-request reviews, bulk edits, and automated workflows). - Resolved synchronization ambiguity with a strict architectural rule: the relational database is the authoritative source of truth in production; Git-versioned JSON is the reproducible transport and deployment mechanism across environments.
- JSON-Level Foreign-Key Relationship Engine: - Implemented an internal dependency graph service that models foreign-key relationships at the JSON level. - Computes topological dependency graphs to calculate safe, ordered deletions when synchronizing destructive updates, preventing foreign-key constraint violations in PostgreSQL.
- Progressive Governance & Fail-Fast Safeguards: - Integrated preflight CI validation, schema dry-runs, drift detection, and startup consistency checks to safeguard production environments as multiple engineering squads adopted the platform.
Consequential Trade-offs & Alternative Paths
- Liquibase Migration Chaining vs. Desired-State Synchronization: Retaining cumulative migrations incurs execution overhead that scales with history. Desired-state synchronization operates on current diffs, maintaining constant-time initialization regardless of how many historical revisions occurred.
- Database Authority vs. Git Authority: Treating Git JSON as the master authority would risk overwriting active database mutations during concurrent testing sessions. Making the database record authoritative while using JSON as the transport manifest preserves ACID integrity and avoids split-brain state.
- Single Shared Migration vs. Modular Network Governance: Centralized migrations forced all 16 networks into lockstep deployments. Decoupling reference data allowed individual payment network squads to version, test, and release their field definitions independently.
Verified Production Outcomes
Staff-Level Engineering Retrospective
- Developer friction is an architectural signal. When migration tools designed for schema DDL are coerced into managing large volumes of volatile business reference data, failure is inevitable.
- Successful platform engineering requires meeting developers where they work: preserving familiar database UI workflows while adding the safety of Git versioning and automated CI gates.