Operating Context & The Engineering Problem
Retail operations in regulated sectors (such as controlled inventory, serialized merchandise, or licensed goods) operate under strict regulatory scrutiny. Every product acquisition, transfer, and sale must be tracked and synchronized with external compliance tracking APIs.
Building an enterprise platform for this domain presented distinctive architectural challenges:
- Zero Data Leakage Tolerance: In a multi-tenant model, strict data isolation between competing retail organizations is mandatory. Accidental cross-tenant query contamination would trigger legal and regulatory penalties.
- Checkout Latency vs. Compliance API Outages: External regulatory APIs are notorious for high latency and intermittent outages. Forcing register point-of-sale transactions to synchronously wait for external compliance acknowledgment would cause long customer checkout lines during network degradation.
- Operational Velocity for Small Teams: Managing complex multi-store role-based access control (RBAC), serial number verification, and inventory auditing required high development velocity without compromising software correctness.
Architecture & System Boundaries
I designed a decoupled platform utilizing asynchronous Python services, strict PostgreSQL tenant isolation, and custom Model Context Protocol (MCP) tooling:
- FastAPI Asynchronous Backend: - Asynchronous Python backend built on FastAPI and SQLAlchemy, providing low-latency REST endpoints for store register operations and real-time inventory updates.
- Tenant-Scoped PostgreSQL Schema Isolation: - Rather than relying on fragile row-level tenant IDs in shared tables, each merchant tenant is allocated a dedicated PostgreSQL schema (`tenant_<id>`). - Connection pools dynamically set the database search path per request, providing a physical boundary against accidental cross-tenant data leakage.
- Transactional Outbox for Compliance Integration: - Designed a transactional outbox pattern to decouple register checkout transactions from external regulatory compliance APIs. - Register sales commit immediately to the local tenant database; asynchronous background workers reliably publish tracking records to external compliance endpoints with exponential backoff and replay safety.
- MCP Operational Toolset for AI Agents: - Created a custom Model Context Protocol (MCP) server exposing store operational capabilities directly to developer AI agents. - Tools allow authorized agents to safely perform operational tasks such as auditing inventory discrepancies, verifying tax calculations, and checking EOD reconciliations without manual SQL queries.
- Agentic Development Velocity: - Leveraged LLM coding agents throughout development to rapidly explore prototypes and scaffold CRUD services, while maintaining engineering ownership over schema designs, security boundaries, and transaction guarantees.
Consequential Trade-offs & Alternative Paths
- Schema-Level Isolation vs. Shared Table RLS: Shared-table multi-tenancy with `tenant_id` columns is common in lightweight SaaS, but a single missing WHERE clause in complex queries leaks data across clients. Schema-level isolation guarantees physical boundary separation with minimal operational overhead for mid-tier enterprise deployments.
- Transactional Outbox vs. Synchronous API Calls: Calling external compliance APIs synchronously within the payment flow introduces unacceptable latency and register freezes during external outages. The transactional outbox pattern guarantees eventual consistency while keeping point-of-sale transactions responsive.
- Agent Automation Boundaries vs. Direct Database Scripts: Providing operational tools via strict MCP contracts rather than allowing agents to execute raw SQL guarantees that business validation rules and audit logs are triggered on every automated action.
Verified Production Outcomes
Staff-Level Engineering Retrospective
- In regulated systems, architectural boundaries must be enforced at the infrastructure and database levels, not solely in application code.
- AI coding agents excel at accelerating implementation velocity, but their output must be governed by strict architectural boundaries, explicit typing, and deterministic integration tests.