Abstract
Cloud data warehouses are commonly organized into successive tiers, typically a landing tier that receives data in
a form close to its original source representation, a staging tier that cleanses and conforms that data into a
consistent, typed structure, and a base tier that models the data into business ready, consumption facing structures.
Each transition between these tiers is an opportunity for data fidelity to be lost, whether through a schema change
that silently drops a field, a deduplication rule that removes a record it should have kept, or a transformation that
introduces a subtle calculation error. This article treats the verification of data fidelity across these tiers as a distinct
engineering discipline, and proposes a layered Tiered Fidelity Assurance Framework, referred to throughout as
TFAF, built around five validation layers aligned to the natural structure of a landing, staging, and base tiered
warehouse. The article reviews the data warehousing, data quality, and continuous delivery literature that informs
the framework, enumerates failure modes specific to each tier transition, and describes concrete validation checks,
tooling categories, governance practices, and illustrative scenarios needed to operate the framework in practice.
In keeping with the goal of making this article a practical, referenceable resource, a substantial number of simple,
structured tables are included throughout, summarizing tier characteristics, validation checks, failure modes,
tooling, roles, and maturity indicators in a form intended for direct reuse by practitioners. All numeric figures
outside the structured tables are presented as indicative, order of magnitude observations drawn from publicly
discussed industry practice prior to August 2021, rather than as a validated empirical result, and the article closes
with a discussion of limitations and directions for continued practice
Keywords
data warehousing, data fidelity, data quality, landing zone, staging layer, data validation, data observability, cloud data platform, ELT, data lineage
References
1) Apache Software Foundation. (2019). Apache Airflow Documentation.
2) Batini, C., and Scannapieco, M. (2006). Data Quality: Concepts, Methodologies and Techniques. Berlin:
Springer.
3) DAMA International. (2017). DAMA-DMBOK: Data Management Body of Knowledge, 2nd edition.
Basking Ridge: Technics Publications.
4) Dasu, T., and Johnson, T. (2003). Exploratory Data Mining and Data Cleaning. Hoboken: Wiley.
5) dbt Labs. (2020). dbt Documentation: Testing.
6) English, L. P. (1999). Improving Data Warehouse and Business Information Quality. New York: Wiley.
7) Fowler, M. (2006). Continuous Integration. martinfowler.com.
8) Fowler, M. (2012). TestPyramid. martinfowler.com.
9) Great Expectations. (2020). Great Expectations Documentation.
10) Helland, P. (2015). Immutability Changes Everything. Proceedings of the Seventh Biennial Conference
on Innovative Data Systems Research.
11) Humble, J., and Farley, D. (2010). Continuous Delivery: Reliable Software Releases through Build, Test,
and Deployment Automation. Boston: Addison Wesley.
12) Inmon, W. H. (1992). Building the Data Warehouse. New York: Wiley.
13) International Organization for Standardization. (2008). ISO/IEC 25012:2008, Software Engineering,
Software Product Quality Requirements and Evaluation, Data Quality Model. Geneva: ISO.
14) Kimball, R., and Ross, M. (2013). The Data Warehouse Toolkit: The Definitive Guide to Dimensional
Modeling, 3rd edition. Hoboken: Wiley.
15) Kleppmann, M. (2017). Designing Data Intensive Applications: The Big Ideas Behind Reliable, Scalable,
and Maintainable Systems. Sebastopol: O'Reilly Media.
16) Linstedt, D., and Olschimke, M. (2015). Building a Scalable Data Warehouse with Data Vault 2.0.
Burlington: Morgan Kaufmann.
17) Loshin, D. (2010). Master Data Management. Burlington: Morgan Kaufmann.
18) Nygard, M. T. (2018). Release It! Design and Deploy Production Ready Software, 2nd edition. Raleigh:
Pragmatic Bookshelf.
19) Rahm, E., and Do, H. H. (2000). Data Cleaning: Problems and Current Approaches. IEEE Data
Engineering Bulletin, 23(4), 3 to 13.
20) Redman, T. (1996). Data Quality for the Information Age. Norwood: Artech House.
21) Snowflake Inc. (2020). Snowflake Documentation: Time Travel and Zero Copy Cloning.