Schema & Data Architecture
The semantic layer is the backbone of MeshLens — a normalized, vendor-agnostic metadata schema that models the entire data mesh lifecycle: domains, applications, connections, data products, lineage, governance, and operational health. Any organization adopting data mesh can use this schema as the single source of truth for mesh observability — regardless of the underlying warehouse or tooling.
Design Principles
Domain-Driven Ownership
Every entity belongs to a domain. Applications, data products, and governance policies carry a domain_id FK — enabling per-domain ownership, cost attribution, and health scoring.
Lineage-First Modeling
Lineage edges connect data products — not tables or columns. This gives a high-level, product-centric view of data flow (FEEDS, DERIVES, AGGREGATES) that maps directly to business understanding.
Operational Separation
Metadata tables (domain, product, lineage) are distinct from time-series operational tables (sync_log, pipeline_health). Operational data scales independently.
Entity Relationship Diagram
14 tables organized across four zones — core platform entities model the organizational structure, the data mesh layer captures products and lineage, pipeline operations track time-series health, and governance enforces policies and SLA compliance.
Schema Design
Three logical groups organize the schema — core platform entities, the data mesh product layer, and operational health with governance controls.
Core Platform
4 tablesThe organizational backbone — domains own applications, applications connect to destinations through managed connections.
Business domains — the fundamental ownership and organizational unit
SaaS, ERP, Database, and API systems with environment tags
Target data warehouses with region for latency considerations
Application → Destination pipe with sync cadence, cost, and status
Data Mesh Layer
4 tablesThe product layer — every data product has a type, quality score, SLA, and directional lineage edges.
The mesh quantum — typed, quality-scored, and SLA-bound
Directed edges (FEEDS, DERIVES, AGGREGATES) between products
Junction linking products to upstream connections and tables
Who/what consumes each product — dashboards, ML models, APIs
Pipeline Operations
4 tablesTime-series health data and sync telemetry — separated from core entities so operational data can scale independently.
Event-level sync audit trail with timing and volume
Pre-aggregated daily metrics — avoids expensive log scans
Latest health snapshot — healthy, degraded, or down
Column adds, drops, and renames — drift alerting
Governance & Quality
2 tablesGovernance policies and SLA breach tracking — enforcing security, quality, and compliance standards across the mesh.
RBAC, PII, retention, quality rules — domain-scoped or global
Freshness and quality SLA breaches with resolution tracking
View Catalog
Analytical views that power MeshLens dashboards. Select a card for schema, lineage, and query logic.
v_mesh_overview
Single-row mesh summary providing a global health snapshot
| # | Column Name | Type | Description |
|---|---|---|---|
| 1 | domain_count | INT | Total number of business domains |
| 2 | app_count | INT | Total registered applications across all domains |
| 3 | product_count | INT | Total data products (all types) |
| 4 | source_products | INT | Count of source-aligned data products |
| 5 | business_products | INT | Count of business data products |
| 6 | consumer_products | INT | Count of consumer-aligned data products |
| 7 | connection_count | INT | Total managed connections |
| 8 | active_connections | INT | Connections with status = ACTIVE |
| 9 | broken_connections | INT | Connections with status = BROKEN |
| 10 | lineage_edges | INT | Total directed edges in lineage graph |
| 11 | total_consumers | INT | Total downstream consumer registrations |
| 12 | overall_health_pct | REAL | % of pipelines in HEALTHY state (latest snapshot) |
| 13 | total_monthly_cost_usd | REAL | Sum of monthly_cost_usd for non-paused connections |
| 14 | open_sla_breaches | INT | Unresolved SLA breaches (resolved_at IS NULL) |
Adopting the MeshLens Semantic Model
The MeshLens semantic model provides a practical, enterprise-ready foundation for standardizing mesh metadata across domains, data products, lineage, operational health, and governance. Its schema and views are designed to be portable across platforms such as Snowflake, BigQuery, Postgres, and other SQL-based environments, so architecture and engineering teams can adapt them to their own naming standards, ingestion pipelines, and governance requirements. Rather than forcing each tool or team to redefine core concepts, the model establishes a consistent structure that can support reporting, observability, lineage analysis, and platform operations from a shared semantic layer.
In practice, enterprises typically map their existing metadata, catalog, orchestration, and warehouse signals into these reference entities, then extend the model with additional attributes or bridge tables where needed. Core views such as v_mesh_overview and v_lineage_graph help transform raw metadata into reusable analytical outputs for BI, engineering dashboards, and custom applications, while keeping product-level lineage and operational signals aligned with business ownership and governance. The result is a common semantic contract for mesh observability that can be queried by any team and hosted on any enterprise data stack.