Arquitetura serverless: Economia e Automatização no Armazenamento e Processamento de Dados

Abstract

The exponential growth in data generation has driven the need for efficient Datalake architectures for data storage, processing, and analysis. This final project aims to create a medallion-style lakehouse architecture based on events using Amazon Web Services (AWS) as the public cloud provider. A Data Lakehouse architecture with medallion focuses primarily on three divisions, which can be called gold, silver, and bronze, where each stage has characteristics ranging from fidelity to the original data to high analytical quality of aggregated data. The main objective of this work was to create a low-cost, event-based medallion architecture. With this, better management of the data to be used for training artificial intelligence models, generating business insights, and assisting decision-making, thus guiding directors on where to focus available resources, besides saving on data processing and storage. This applied research compares services and analyzes the costs associated with a serverless architecture for processing small data sets. It concludes that the use of serverless services brings significant benefits in terms of automation and cost.