Big Data normalization for massively parallel processing databases
Number of Authors: 22017 (English)In: Computer Standards & Interfaces, ISSN 0920-5489, E-ISSN 1872-7018, Vol. 54, no 2, p. 86-93Article in journal (Refereed) Published
Abstract [en]
High performance querying and ad-hoc querying are commonly viewed as mutually exclusive goals in massively parallel processing databases. Furthermore, there is a contradiction between ease of extending the data model and ease of analysis. The modern 'Data Lake' approach, promises extreme ease of adding new data to a data model, however it is prone to eventually becoming a Data Swamp - unstructured, ungoverned, and out of control Data Lake where due to a lack of process, standards and governance, data is hard to find, hard to use and is consumed out of context. This paper introduces a novel technique, highly normalized Big Data using Anchor modeling, that provides a very efficient way to store information and utilize resources, thereby providing ad-hoc querying with high performance for the first time in massively parallel processing databases. This technique is almost as convenient for expanding data model as a Data Lake, while it is internally protected from transforming to Data Swamp. A case study of how this approach is used for a Data Warehouse at Avito over a three-year period, with estimates for and results of real data experiments carried out in HP Vertica, an MPP RDBMS, is also presented. This paper is an extension of theses from The 34th International Conference on Conceptual Modeling (ER 2015) (Golov and Ronnback 2015) [1], it is complemented with numerical results about key operating areas of highly normalized big data warehouse, collected over several (1-3) years of commercial operation. Also, the limitations, imposed by using a single MPP database cluster, are described, and cluster fragmentation approach is proposed.
Place, publisher, year, edition, pages
2017. Vol. 54, no 2, p. 86-93
Keywords [en]
Big Data, MPP, Database, Normalization, Analytics, Ad-hoc, Querying, Modeling, Performance, Data Lake
National Category
Electrical Engineering, Electronic Engineering, Information Engineering Computer and Information Sciences
Identifiers
URN: urn:nbn:se:su:diva-144754DOI: 10.1016/j.csi.2017.01.009ISI: 000401888800004OAI: oai:DiVA.org:su-144754DiVA, id: diva2:1127699
2017-07-182017-07-182018-01-13Bibliographically approved