This is a working note from an active engagement, not a retrospective. I’m currently leading the design and build of a Payments Domain Data Store (DDS) on Azure Databricks inside the Wealth Management and International Core Banking grid of a Tier-1 European bank. What follows is the architecture-level thinking — the implementation details and platform internals stay inside the bank, but the shape of the decisions is shareable.
Table of contents
Open Table of contents
Why a Domain Data Store, not a generic warehouse
A common reflex is to land payments data in the enterprise warehouse and let downstream consumers model what they need. That works at small scale. At the volume and regulatory exposure of a Tier-1 European bank’s payments flow it does not — three things break.
First, ownership becomes diffuse. Every consumer ends up with their own “view” of what a payment is, with subtle differences in scope and timing. Reconciling those views eats more analyst time than the original modelling would have.
Second, regulatory delivery becomes brittle. ECB and CESOP reporting cycles require deterministic, audit-traceable definitions of in-scope transactions, counterparty resolution, and timing. If those definitions live in the BI layer rather than the data layer, they drift.
Third, operational reporting and analytics pull in different directions. Operational teams need fresh, granular records with clear lineage. Analysts want stable, slowly-changing aggregates. Trying to serve both off the same model produces a compromise that serves neither well.
A Domain Data Store solves these by putting business-meaningful, governed entities at the centre — payment, instruction, settlement, counterparty, account — and pushing the warehouse-style consumption outward.
Why Databricks specifically
The choice of Azure Databricks (over staying on Synapse, which the rest of the estate uses) came down to four things.
Delta Lake’s transactional guarantees — for a regulated domain, the ability to assert ACID semantics on the lake layer, with time-travel for audit, is load-bearing. It changes what kind of failure modes you have to engineer around.
Unity Catalog as the governance plane — being able to push lineage, access control, and column-level masking down into the catalog (rather than scattering them across ADF, Synapse, and Power BI) is a real maintenance saver.
Notebook + job parity — engineering on a notebook, then shipping the same code as a scheduled job, removes a class of “works locally, breaks in production” bugs.
The skills market — easier to hire and easier to retain. Synapse SQL Pool talent is harder to find than it was three years ago.
Logical and physical modelling in erwin
The model lives in erwin Data Modeler, not in code, and that’s deliberate. Two reasons.
First, the modelling exercise is a business conversation, not an engineering one. Walking through entity definitions in erwin with the payments product team produces decisions; reviewing a CREATE TABLE statement produces nods. The visual modelling tool keeps the room engaged and surfaces disagreements you’d otherwise discover at UAT.
Second, erwin is the single source of truth that emits both forward and reverse. Forward: it generates the DDL for the Databricks tables. Reverse: when a column gets added in production, the model gets updated to match — so the documentation is never six months behind the data.
The logical model is third-normal-form for the transactional entities, with a deliberate denormalisation layer for reporting consumption. Physical model adds Databricks-specific concerns: partitioning, clustering, file sizing.
What “data products” actually means here
The word “data product” is overloaded. In this engagement it means: a curated, versioned, self-describing dataset on top of the DDS, with a documented consumer contract. Not a Power BI report. Not a notebook export. A first-class asset with an owner, a refresh cadence, an SLA, and a discoverable entry in the catalog.
Concretely, the payments DDS ships a handful of these:
payments_regulatory_view— the slice that feeds ECB and CESOP submissions. Frozen schema, versioned definitions, deterministic timing.payments_operational_state— near-real-time view of in-flight payments for ops dashboards.payments_counterparty_360— the customer-level rollup that backs Wealth and Core Banking client dashboards.
Self-service Power BI sits on top of these products. Analysts build their own visualisations; they don’t get to redefine what a payment is.
Non-functional realities
A few non-negotiables that shaped the build:
Audit traceability. Every record needs to be reconstructible at any point in the regulatory reporting window. Delta time-travel and immutable raw-zone retention are how that’s enforced.
Availability. Regulatory cycles are hard deadlines. The job graph has explicit fail-fast points so we know within minutes — not hours — whether a cycle is at risk.
Cost discipline. Databricks is excellent and Databricks is expensive. Cluster autoscaling, photon for SQL workloads where it actually helps (not everywhere), and aggressive use of liquid clustering on the high-cardinality tables.
Lineage to source. Every column in a downstream data product traces back to a source-system column through the catalog. When something looks wrong in a Power BI report, that lineage is the thirty-second debugging path.
What I’d do differently if starting over
Three things.
Start with the data product contracts, not the entities. It’s tempting to model the entities first and “figure out the consumption later.” The contracts force you to answer hard questions earlier — what’s the timing definition, what’s the grain, what’s the SLA — and those answers shape the entity model.
Invest in the test data sooner. Synthetic payments data that covers the messy real-world cases (failed settlements, FX legs, reversals, restitution) is a permanent investment. Building it after the model means rewriting tests once the messy cases turn up.
Pick the governance plane before you pick the platform. We chose Unity Catalog after Databricks. In hindsight, the catalog/governance decision should drive the platform decision, not the other way round.
Where it sits in the bigger picture
The DDS isn’t an end state. It’s the foundation that lets the next two things happen: (a) move ECB and CESOP regulatory reporting off legacy and onto governed, auditable products; and (b) make the Wealth and Core Banking client view consumable by self-service teams without re-fetching the same data five times. Both are visible business outcomes, not just architecture diagrams.
Those second-order outcomes are what justifies the work to the people writing the cheques. Worth keeping that lens.