Please enable JavaScript.
Coggle requires JavaScript to display documents.
FEDB generic loader - Coggle Diagram
FEDB generic loader
Features
ease of learning & supporting
Config files should handle variations
Actionable emails
Dashboard
Single pipe mostly
ease of dev and deploy
minimize coding using configuration
Ease of scheduling
AWS Batch
Autosys
cloudwatch
triggered by mail?
Java stack?
Cloud native/friendly
Graphical UI?
avoid repetitive code
Options
AWS Glue
-ve Scala/python (pure java ok)
+ve Graphical UI
getting files from vendors
GIN loader
-ve cant handle computations
Third party
Airflow
-ve python
Pentaho
Transform to an intermediate format
and then simple streamlined load from there?
-ve Two steps
+ve complexity isolated
raw load into tables and
then process it to destination table
Challenges
Multiple file formats
pivoting needed for some files
complex logic/computations
before loading
Year->Month
ID construction like
BP.R.BP015.06M.05Yx05Y
some jobs involve downloading
current problems
multitude of technologies
repetitive coding
Too many emails