Distributed Systems / DevSecOps6 min technical read

Event-Driven SBOM Generation from CI Artifact Telemetry

Engineered core event consumption, correlation, and persistence components for a greenfield telemetry-driven Software Bill of Materials (SBOM) platform at Goldman Sachs, transforming asynchronous build telemetry into deterministic, build-scoped dependency manifests.

Role & Architectural Scope:Core engineering contributor within Goldman Sachs distributed platform engagement. Personally owned Kafka event consumer pipelines, AWS Lambda event persistence into PostgreSQL, build-scoped event ordering/deduplication, and S3 artifact storage within the team's wider architectural design.
Kafka Event StreamingGitLab CI TelemetryFirst-WRITE CutoffLambda / PostgreSQLAWS S3 / CycloneDX
Verified System Boundaries & Data FlowSanitized Architectural Blueprint
Event-Driven Software Bill of Materials (SBOM) Generation CORE DISTRIBUTED PLATFORM CONTRIBUTOR • GOLDMAN SACHS CLIENT ENGAGEMENT 01 / CI ARTIFACT TELEMETRY GITLAB CI RUNNER • Parallel Build Workers • Emits Build / Event IDs NEXUS REPOSITORY • READ: Dependencies • WRITE: Built Artifact • Out-of-band telemetry ZERO BUILD PENALTY Non-blocking to developers 02 / CORRELATION ENGINE (OWNED) APACHE KAFKA TOPIC • Build-ID Partitioned Streams • Scalable asynchronous buffer FIRST-WRITE CUTOFF FILTER • First artifact WRITE = Boundary • Drops post-build READ telemetry • Eliminates test-tool contamination LAMBDA & POSTGRESQL STATE • Event ID Replay Deduplication • Deterministic ORDER BY sequence 03 / SBOM PERSISTENCE CLOSING EVENT TRIGGER • Pipeline Complete Signal • Assembles Dependency Graph • CycloneDX / SPDX format AMAZON S3 & SCANNING • Build-Scoped S3 Keys • Cryptographic Traceability • Vulnerability Auditing • Immutable Manifest Store
01

Operating Context & The Engineering Problem

Enterprise software supply-chain governance traditionally relies either on static repository scanners (which miss dynamic build resolutions) or blocking CI plugins (which add latency to build pipelines and fail when remote vulnerability services experience downtime).

Goldman Sachs initiated an in-house greenfield platform to generate Software Bills of Materials (SBOMs) out-of-band by observing read and write telemetry emitted between GitLab CI/CD build runners and Nexus enterprise artifact repositories.

However, capturing and reconstructing supply-chain provenance from raw telemetry presented distinct distributed systems challenges:

  1. Out-of-Order Telemetry Streams: High-frequency read and write events from parallel build runners arrived out of order due to network retries and worker scheduling.
  2. Replay Deduplication: Network retries emitted duplicate telemetry events that risked corrupting dependency counts if ingested naively.
  3. Post-Build Dependency Contamination: During a build, runners download dependencies, compile the binary, and publish the artifact to Nexus. After publishing, however, runners frequently execute post-build reporting tools or test uploaders that trigger subsequent Nexus READ events. These post-build reads do not belong to the compiled artifact.
02

Architecture & System Boundaries

Within the greenfield platform architecture, I personally implemented the Kafka consumption, persistence, and build-correlated reconstruction pipeline:

  1. Telemetry Event Ingestion: - Captured raw telemetry emitted by Nexus repository listeners into an enterprise Kafka messaging topic. - Preserved composite tracking identifiers across the entire lifecycle: `gitlab_build_id`, `pipeline_id`, and `event_id`.
  1. The First-WRITE Artifact Cutoff Logic: - Applied an artifact-lifecycle cutoff rule: the first relevant Nexus artifact WRITE event marks the strict boundary of compilation dependencies. - Any Nexus READ events observed *after* this cutoff were excluded from that artifact's inventory, ensuring post-compilation test runners and reporting plugins did not contaminate the compiled binary's SBOM.
  1. Deterministic Replay Deduplication in PostgreSQL: - Orchestrated AWS Lambda consumers to persist correlated telemetry events into PostgreSQL. - Enforced deduplication using preserved event identifiers and reconstructed deterministic event sequences via indexed `ORDER BY` queries on persisted telemetry records.
  1. Closing Event Assembly & S3 Persistence: - A dedicated pipeline closing event triggered the aggregation engine to materialize the build's accumulated dependency graph. - Formatted standard CycloneDX/SPDX machine-readable manifests and stored immutable, build-scoped SBOM files in Amazon S3 for downstream enterprise vulnerability scanning.
03

Consequential Trade-offs & Alternative Paths

  • Asynchronous Event-Driven Aggregation vs. Synchronous CI Build Scanning: Synchronous CI plugins force developers to wait for scanner execution and break builds on network timeouts. Event-driven telemetry reconstruction operates out-of-band with zero impact on build latency or developer cycle times.
  • Kafka Partitioning vs. Point-to-Point Queues: Utilizing Kafka with build-ID partitioning guaranteed ordered delivery per CI execution while allowing parallel processing across thousands of independent repositories.
  • Strict First-WRITE Boundary vs. Full-Lifecycle Ingestion: Without the first-WRITE cutoff rule, dependency manifests were routinely contaminated with incidental build tooling. The cutoff rule restored deterministic build fidelity.
04

Verified Production Outcomes

Implemented non-blocking, build-scoped SBOM generation across enterprise repositories with zero CI build latency overhead.
Achieved deterministic event reconstruction and replay deduplication across distributed GitLab runners.
Delivered automated, cryptographically traceable Software Bills of Materials stored directly in Amazon S3 for compliance auditing.
05

Staff-Level Engineering Retrospective

  • Real-world distributed telemetry cannot be trusted to arrive in order or exactly once. System boundaries must be designed around idempotent consumption, persistent event identifiers, and logical cutoff rules.
  • Supply-chain security is most effective when it is invisible to developers—capturing truth from infrastructure telemetry rather than forcing engineers to maintain fragile local build plugins.