Please enable JavaScript.
Coggle requires JavaScript to display documents.
Miral - AAZ data onboarding - Architecture Approach Requirements - Coggle…
Miral - AAZ data onboarding - Architecture Approach Requirements
Objectives
Changing the approach for onboarding.
Feasibility of aligning & Complying AAZ Architecture with upcoming Data Modernization architecture
Save the snowflake cost
Current Approach
Data is currently fetched from the Data sources as files into Raw Zone
Sharepoint
Oracle
SQL Server
ETL - ADF
Converting the files in raw files into Parquet format and storing into Standard zone
ETL - Using Databricks
The data is then processed by applying transformation to cleanse, join & consolidate based on the business rules to load the data into dimensional model deployed in snowflake
from here only incremental data is extracted using combination of columns to identify as a key for each table
ETL - using databricks
Tableau Completely work on the Table view
Reports
Self Service datasets
The granularity of the Table view is at ticket level
Aggregate layer is built on top of the table views for sales and combined view of sales & revenue togather and to create these aggregated layers dimensions are used.
Challenges
No CDC in the source data, every time full datasets are loaded from source to datalake (Full)
Desired Approach
Should be similar to Miral Fusion data onboarding
Medellion architecture should be followed
Dimensional Model should be available in Databricks
Table Views required for reporting should be populated into Snowflake