Abstract
Enterprise data warehouses serve as centralized repositories for integrating, transforming, and analyzing data from heterogeneous operational systems. Their reliability directly influences business intelligence (BI), organizational reporting, strategic planning, and data - driven decision - making. However, enterprise data pipelines frequently encounter data qual ity problems, including missing values, duplicate records, inconsistent formats, invalid entries, unexpected distribution changes, and violations of business rules. Traditional data validation approaches generally rely on predefined constraints and manual inspection, which may be insufficient for identifying complex or previously unseen anomalies in large - scale analytical environments. Consequently, automated and scalable data quality management has become an important requirement for modern enterprise data architecture. This research proposes an integrated framework for automated data quality assessment and anomaly detection in enterprise data warehousing. The framework combines data profiling, rule - based validation, statistical analysis, and machine - learning - based anomal y detection to identify and classify potential data quality issues throughout the data integration lifecycle. The proposed architecture consists of data ingestion, preprocessing, quality assessment, anomaly detection, severity classification, automated ale rting, and validation reporting components. Data quality is assessed using dimensions such as completeness, validity, consistency, uniqueness, and conformity to predefined business constraints. Statistical or machine - learning methods may subsequently ident ify unusual patterns that cannot be detected through conventional validation rules alone. The framework is designed for integration with enterprise data warehouses, including SQL Server and Snowflake, and BI platforms such as Power BI. Its effectiveness should be evaluated through comparative experiments using established baseline
Keywords
Enterprise Data Warehousing, Automated Data Quality Assessment; Anomaly Detection; Data Profiling; ETL Automation; Data Governance; Machine Learning.
References
Enterprise data warehouses serve as centralized repositories for integrating, transforming, and analyzing data from heterogeneous operational systems. Their reliability directly influences business intelligence (BI), organizational reporting, strategic planning, and data-driven decision-making. However, enterprise data pipelines frequently encounter data quality problems, including missing values, duplicate records, inconsistent formats, invalid entries, unexpected distribution changes, and violations of business rules. Traditional data validation approaches generally rely on predefined constraints and manual inspection, which may be insufficient for identifying complex or previously unseen anomalies in large-scale analytical environments. Consequently, automated and scalable data quality management has become an important requirement for modern enterprise data architecture. This research proposes an integrated framework for automated data quality assessment and anomaly detection in enterprise data warehousing. The framework combines data profiling, rule-based validation, statistical analysis, and machinelearning-based anomaly detection to identify and classify potential data quality issues throughout the data integration lifecycle. The proposed architecture consists of data ingestion, preprocessing, quality assessment, anomaly detection, severity classification, automated alerting, and validation reporting components. Data quality is assessed using dimensions such as completeness, validity, consistency, uniqueness, and conformity to predefined business constraints. Statistical or machine-learning methods may subsequently identify unusual patterns that cannot be detected through conventional validation rules alone. The framework is designed for integration with enterprise data warehouses, including SQL Server and Snowflake, and BI platforms such as Power BI. Its effectiveness should be evaluated through comparative experiments using established baseline methods, representative datasets, and appropriately verified anomaly labels.