Abstract
Poor data quality remains one of the most persistent and expensive problems in enterprise data management.
Studies conducted over the past two decades have repeatedly found that practitioners spend a disproportionate
share of project time locating, diagnosing, and repairing errors in structured data before that data can be trusted
for analytics, reporting, or machine learning. Traditional data quality tooling, built primarily around deterministic
rules, regular expressions, and manually authored constraints, has struggled to keep pace with the volume,
velocity, and structural diversity of contemporary data sources. This article examines how machine learning can
be embedded directly into the design of a data quality profiling and cleansing package, rather than bolted on as an
auxiliary scoring layer.
We propose a reference architecture that combines statistical profiling, supervised and semi-supervised error
detection models, confidence-calibrated anomaly scoring, and a human-in-the-loop repair recommendation engine
into a single composable package. The design is organized around five principles: composability, configurability,
explainability, scalability, and governance. To evaluate the approach, we constructed a synthetic benchmark using
an established error injection methodology and measured detection accuracy, precision, recall, latency, and the
improvement in six standard data quality dimensions before and after cleansing. Across the benchmark, a hybrid
ensemble that blends statistical baselines with supervised classifiers achieved an F1 score of 0.91, compared with
0.61 for a purely rule-based baseline, while keeping end-to-end processing latency within a practical range for
batch and near-real-time pipelines.
The findings suggest that treating error detection and repair as first-class, model-driven components of a cleansing
package, supported by clear interfaces for human review and audit logging, produces a system that is both more
accurate and easier to govern than conventional rule-only tooling. We conclude with a discussion of design tradeoffs, current limitations, threats to validity, and directions for future research, including active learning for error
detection and standardized benchmarks for cleansing package evaluation.
References
1) Abedjan, Z., Chu, X., Deng, D., Fernandez, R. C., Ilyas, I. F., Ouzzani, M., Papotti, P., Stonebraker, M.,
& Tang, N. (2016). Detecting data errors: Where are we and what needs to be done? Proceedings of the
VLDB Endowment, 9(12), 993 to 1004.
2) Arocena, P. C., Glavic, B., Mecca, G., Miller, R. J., Papotti, P., & Santoro, D. (2015). Messing up with
BART: Error generation for evaluating data cleaning algorithms. Proceedings of the VLDB Endowment,
9(2), 36 to 47.
3) Batini, C., & Scannapieco, M. (2016). Data and information quality: Dimensions, principles and
techniques. Springer.
4) Breck, E., Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2019). Data validation for machine
learning. Proceedings of Machine Learning and Systems, 1, 334 to 347.
5) Dallachiesa, M., Ebaid, A., Eldawy, A., Elmagarmid, A., Ilyas, I. F., Ouzzani, M., & Tang, N. (2013).
NADEEF: A commodity data cleaning system. Proceedings of the 2013 ACM SIGMOD International
Conference on Management of Data, 541 to 552.
6) Fan, W., & Geerts, F. (2012). Foundations of data quality management. Morgan and Claypool Publishers.
7) Heidari, A., McGrath, J., Ilyas, I. F., & Rekatsinas, T. (2019). HoloDetect: Few-shot learning for error
detection. Proceedings of the 2019 International Conference on Management of Data, 829 to 846.
8) Hellerstein, J. M. (2008). Quantitative data cleaning for large databases. United Nations Economic
Commission for Europe.
9) Ilyas, I. F., & Chu, X. (2019). Data cleaning. ACM Books, Association for Computing Machinery.
10) Kandel, S., Heer, J., Plaisant, C., Kennedy, J., van Ham, F., Riche, N. H., Weaver, C., Lee, B., Brodbeck,
D., & Buono, P. (2011). Research directions in data wrangling: Visualizations and transformations for
usable and credible data. Information Visualization, 10(4), 271 to 288.
11) Krishnan, S., Wang, J., Wu, E., Franklin, M. J., & Goldberg, K. (2016). ActiveClean: Interactive data
cleaning for statistical modeling. Proceedings of the VLDB Endowment, 9(12), 948 to 959.
12) Krishnan, S., & Wu, E. (2019). AlphaClean: Automatic generation of data cleaning pipelines. arXiv
preprint arXiv:1904.11827.
13) Loshin, D. (2010). The practitioner's guide to data quality improvement. Morgan Kaufmann.
14) Mahdavi, M., Abedjan, Z., Fernandez, R. C., Madden, S., Ouzzani, M., Stonebraker, M., & Tang, N.
(2019). Raha: A configuration-free error detection system. Proceedings of the 2019 International
Conference on Management of Data, 865 to 882.
15) Naumann, F. (2014). Data profiling revisited. ACM SIGMOD Record, 42(4), 40 to 49.
16) Neutatz, F., Mahdavi, M., & Abedjan, Z. (2019). ED2: A case for active learning in error detection.
Proceedings of the 28th ACM International Conference on Information and Knowledge Management,
2249 to 2252.
17) Rahm, E., & Do, H. H. (2000). Data cleaning: Problems and current approaches. IEEE Data Engineering
Bulletin, 23(4), 3 to 13.
18) Redman, T. C. (1998). The impact of poor data quality on the typical enterprise. Communications of the
ACM, 41(2), 79 to 82.
19) Rekatsinas, T., Chu, X., Ilyas, I. F., & Re, C. (2017). HoloClean: Holistic data repairs with probabilistic
inference. Proceedings of the VLDB Endowment, 10(11), 1190 to 1201.
20) Schelter, S., Lange, D., Schmidt, P., Celikel, M., Biessmann, F., & Grafberger, A. (2018). Automating
large-scale data quality verification. Proceedings of the VLDB Endowment, 11(12), 1781 to 1794.
21) Thirumuruganathan, S., Tang, N., Ouzzani, M., & Doan, A. (2020). Data curation with deep learning.
Proceedings of the 23rd International Conference on Extending Database Technology, 277 to 286.
22) Wang, P., & He, Y. (2019). Uni-Detect: A unified approach to automated error detection