Enterprise Lakehouse Pipelines for Analytics
Keywords:
Governed Lakehouse Intelligence, Data Lakehouse, Data Governance, Enterprise Data Pipelines, Private Cloud, Machine Learning Operations (MLOps), Data Engineering, Enterprise Analytics, AI-Driven Data Platforms, Data Products.Abstract
The Governed Lakehouse Intelligence Framework provides end-to-end governance of data pipelines for enterprise data serving, machine learning, and analytics on encrypted customer and partner data in a private cloud environment. Its central use case consolidates many heterogeneous data sources into a single governed lakehouse, addressing the operational challenges of running cloud services while delivering scalable big-data-lake capabilities. This foundation supports both operational and analytical workloads across multiple business areas and lays the groundwork for an AI/ML-driven enterprise. Governance controls ensure that sensitive data is protected throughout every stage of the process.
Implemented at the Singapore Economic Development Board, the approach is broadly applicable to other organizations seeking to reduce the cost of building, maintaining, and operating data pipelines while making analytics-ready data easy to discover. A representative set of pipelines is run as data products, each monitored against a service-level agreement and managed through a portfolio-based investment model. Focusing delivery on business-facing products improves service quality, while an abstraction layer for data engineers lessens the effort of setting up and orchestrating pipelines.
References
1. Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. In Proceedings of the 11th Annual Conference on Innovative Data Systems Research (CIDR).
2. Begoli, E., Goethert, I., & Knight, K. (2021). A lakehouse architecture for the management and analysis of heterogeneous data for biomedical research and mega-biobanks. In Proceedings of the 2021 IEEE International Conference on Big Data (Big Data 2021) (pp. 4643–4651).
3. Sawadogo, P. N., & Darmont, J. (2021). On data lake architectures and metadata management. Journal of Intelligent Information Systems, 56(1), 97–120.
4. Eichler, R., Giebler, C., Gröger, C., Schwarz, H., & Mitschang, B. (2021). Modeling metadata in data lakes—A generic model. Data & Knowledge Engineering, 136, 101931.
5. Danda, R. R., Maguluri, K. K., Yasmeen, Z., Mandala, G., & Dileep, V. (2023). Intelligent Healthcare Systems: Harnessing Ai and Ml To Revolutionize Patient Care And Clinical Decision-Making. International Journal of Applied Engineering and Technology https://papers. ssrn. com/sol3/papers. cfm.
6. Schoenenwald, A., Kern, S., Viehhauser, J., & Schildgen, J. (2021). Collecting and visualizing data lineage of Spark jobs. Datenbank-Spektrum, 21, 179–189.
7. Glavic, B. (2021). Data provenance: Origins, applications, algorithms, and models. Foundations and Trends in Databases, 9(3–4), 209–441.
8. Loukiala, A., Joutsenlahti, J.-P., Raatikainen, M., Mikkonen, T., & Lehtonen, T. (2021). Migrating from a centralized data warehouse to a decentralized data platform architecture. In L. Ardito, A. Jedlitschka, M. Morisio, & M. Torchiano (Eds.), Product-Focused Software Process Improvement. PROFES 2021 (pp. 36–48). Springer.
9. Pamisetty, V. (2022). Making IoT Data Processing Scalable: Architectures That Actually Work. International Journal of Scientific Research and Modern Technology, 281. https://doi.org/10.38124/ijsrmt.v1i12.1187
10. Priebe, T., Neumaier, S., & Markus, S. (2021). Finding your way through the jungle of big data architectures. In Proceedings of the 2021 IEEE International Conference on Big Data (Big Data) (pp. 5994–5996).
11. Candela, L., Grossi, V., Manghi, P., & Trasarti, R. (2021). A workflow language for research e-infrastructures. International Journal of Data Science and Analytics, 11, 361–376.
12. Nadal, S., Jovanovic, P., Bilalli, B., & Romero, O. (2022). Operationalizing and automating data governance. Journal of Big Data, 9, 117.
13. Srinivas Kalisetty. (2023). Big Data–Driven Cloud Collaboration Models for Enhancing Supplier–Retailer Synchronization in Modern Manufacturing Supply Chains. Journal of Computational Analysis and Applications (JoCAAA), 31(4), 2188–2205. Retrieved from https://eudoxuspress.com/index.php/pub/article/view/4232
14. Wieder, P., & Nolte, H. (2022). Toward data lakes as central building blocks for data management and analysis. Frontiers in Big Data, 5, 945720.
15. Machado, I. A., Costa, C., & Santos, M. Y. (2022). Data mesh: Concepts and principles of a paradigm shift in data architectures. Procedia Computer Science, 196, 263–271.
16. Machado, I. A., Costa, C., & Santos, M. Y. (2022). Advancing data architectures with data mesh implementations. In J. De Weerdt & B. Polyvyanyy (Eds.), Intelligent Information Systems. CAiSE Forum 2022 (pp. 10–18). Springer.
17. Azeroual, O., Schöpfel, J., Ivanovic, D., & Nikiforova, A. (2022). Combining data lake and data wrangling for ensuring data quality in CRIS. Procedia Computer Science, 211, 3–16.
18. Meda, R., & Pamisetty, A. (2023). Intelligent Infrastructure for Real-Time Inventory and Logistics in Retail Supply Chains. Educational Administration: Theory and Practice, 29 (4), 5215–5233.
19. Gujjala, P. K. R. (2022). Data science pipelines in lakehouse architectures: A scalable approach to big data analytics. World Journal of Advanced Research and Reviews, 16(3), 1412–1425.
20. Bauer, D., Giblin, C., Garcés-Erice, L., Pardon, N., Rooney, S., Toniato, E., & Urbanetz, P. (2022). Revisiting data lakes: The meta-data lake. In Proceedings of the 2022 ACM/IFIP International Middleware Conference (pp. 1–7).
21. Berisha, B., Mëziu, E., & Shabani, I. (2022). Big data analytics in cloud computing: An overview. Journal of Cloud Computing, 11, 24.
22. Belcastro, L., Cantini, R., Marozzo, F., Orsino, A., Talia, D., & Trunfio, P. (2022). Programming big data analysis: Principles and solutions. Journal of Big Data, 9, 4.
23. Zichichi, M., Ferretti, S., D’Angelo, G., & Rodríguez-Doncel, V. (2022). Data governance through a multi-DLT architecture in view of the GDPR. Cluster Computing, 25(6), 4515–4542.
24. Pandugula, C., & Nampalli, R. C. R. Optimizing Retail Performance: Cloud-Enabled Big Data Strategies for Enhanced Consumer Insights.
25. Schneider, J., Gröger, C., Lutsch, A., Schwarz, H., & Mitschang, B. (2023). Assessing the lakehouse: Analysis, requirements and definition. In Proceedings of the 25th International Conference on Enterprise Information Systems (ICEIS) (pp. 44–56). SciTePress.
26. Hai, R., Koutras, C., Quix, C., & Jarke, M. (2023). Data lakes: A survey of functions and systems. IEEE Transactions on Knowledge and Data Engineering, 35(12), 12571–12590.
27. Ait Errami, S., Hajji, H., Ait El Kadi, K., & Badir, H. (2023). Spatial big data architecture: From data warehouses and data lakes to the LakeHouse. Journal of Parallel and Distributed Computing, 176, 70–79.
28. Zagan, E., & Danubianu, M. (2023). Data lake architecture for storing and transforming web server access log files. IEEE Access, 11, 40916–40929.
29. Ma, C., & Hu, X. (2023). A data analysis privacy regulation compliance scheme for lakehouse. In Proceedings of the 2023 2nd International Conference on Algorithms, Data Mining, and Information Technology (pp. 1–5). ACM.
30. Yandamuri, U. S. (2023). An Intelligent Analytics Framework Combining Big Data and Machine Learning for Business Forecasting. Zenodo.
31. Zouari, F., Ghedira-Guegan, C., Boukadi, K., & Kabachi, N. (2023). A semantic and service-based approach for adaptive multi-structured data curation in data lakehouses. The World Wide Web, 26, 4001–4023.
Additional Files
Published
Data Availability Statement
None
Issue
Section
License
Copyright (c) 2024 Alexander Petrov (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
