A novel approach to data vault flexibility and temporal capabilities in data warehouse systems

Abstract

Data Warehouses have become essential for organizations aiming to leverage data effectively. The three primary approaches to Data Warehousing are Inmon’s normal- ized model, Kimball’s dimensional model, and Data Vault. Data Vault is a modeling technique designed for flexibility in response to changes in data sources and analytical re- quirements, offering robust auditing and time-travel features by storing all data versions. This thesis reviews the latest developments in Data Vault, classifying recent papers into five categories: theoretical evaluation, practical evaluation, automation, metadata mod- eling, and alternative models. The literature generally agrees on Data Vault’s advantages in model evolution and temporal handling, while also highlighting challenges with model complexity and query performance. Data Vault patterns facilitate the automation of data ingestion, presentation, and model management. Proposed alternative models enhance flexibility and temporal handling but increase complexity due to a higher number of tables and joins, and its performance, model management and automation was not covered in depth. To address these issues, this study explores a new data model aiming to balance improved flexibility and temporal features with reduced complexity. It evaluates models inspired by Document Oriented and Wide Column stores against the Data Vault model. Initial results indicate that the Wide Column inspired model improves data ingestion and query performance, whereas the Document Oriented inspired model does not. With further evaluation, the Wide Column inspired model demonstrated consistent improve- ments, making it suitable for Data Warehouses requiring high auditability, flexibility, and temporal capabilities.

Data Warehouses tornaram-se essenciais para organiza¸c˜oes que procuram aproveitar os dados de forma eficaz. As principais abordagens s˜ao o modelo normalizado de Inmon, o modelo dimensional de Kimball e o Data Vault. Este enfatiza a flexibilidade em re- sposta a altera¸c˜oes nas fontes de dados e nos requisitos anal´ıticos, oferecendo recursos robustos de auditoria e temporais. Esta tese revisa os desenvolvimentos mais recentes no Data Vault, classificando artigos recentes em cinco categorias: avalia¸c˜ao te´orica, avalia¸c˜ao pr´atica, automa¸c˜ao, modelagem de metadados e modelos alternativos. A literatura con- corda geralmente nas vantagens do Data Vault de flexibilidade e aspetos temporais, e destaca desafios com a complexidade do modelo e desempenho das consultas. Os padr˜oes do Data Vault facilitam a automa¸c˜ao da ingest˜ao e apresenta¸c˜ao de dados e gest˜ao do modelo. Modelos alternativos propostos melhoram a flexibilidade e aspetos temporais, mas aumentam a complexidade devido a um maior n´umero de tabelas e joins, e o seu de- sempenho, gest˜ao de modelos e automa¸c˜ao n˜ao foram abordados com profundidade. Este estudo explora um novo modelo de dados que visa equilibrar a melhoria da flexibilidade e caracter´ısticas temporais com a redu¸c˜ao da complexidade. Avalia modelos inspirados em armazenamentos Document Oriented e Wide Column em compara¸c˜ao com o modelo Data Vault. Resultados indicam que o modelo inspirado em Wide Column melhora a in- gest˜ao de dados e desempenho de consultas, enquanto o modelo inspirado em Document Oriented n˜ao. O modelo inspirado em Wide Column demonstrou melhorias consistentes, tornando-o adequado para Data Warehouses que exigem alta auditabilidade, flexibilidade e capacidades temporais.