Please enable JavaScript.

Coggle requires JavaScript to display documents.

KE Data Engineer (DW/BI (MDM vs DQ (Data Quality Systems: Ref-tables…

- - - - RDBMS structure
      - Clustered indexes
  - - - MongoDB
      - HBASE
      - Cassandra
      - Redis
      - Dynamo DB
  - - - OLTP: Maintain fundamental business tasks
      - OLAP: Planning, Problem solving, and Decision support
  - - - Informatica
      - Talend
      - SSIS / Azure Data Factory
      - Pentaho
      - Oracle Data Integrator
    - - Batch processing
        
        HDFS
        
        DB
        
        batch files
        
        Hadoop
        
        Airflow
        
        S3
      - Streaming
        (Real-Time)
        
        Kafka
        
        Spark
        
        Cassandra
        
        Storm
        
        Apache Flink stream-processing framework
    - - AWS Glue
      - GCP BigQuery vs Dataflow
      - Azure Data Factory
  - - - Oracle DQ
      - Informatica DQ
      - Talend DI
    - - systems
        
        ~~custom scripts~~
        
        SAS MDM
        
        SAP MDGov
        
        Ataccama ??
        
        Talend MDM
      - Data Integration
      - Data Governance is a collection of processes, roles, policies, standards, metrics for effective and efficient use of information in organization
  - - - AWS Lambda/Step Functions (Cloud Task Management)
        GCP ~~DataPrep~~ DataFlow BigQuery
        Azure Data Factory
  - - - пример снежинки
    - - Inmon - витриньі ELT top-down appr
        
        use relations for atomic data
        use :star: for aggregate data (mart)
        
        DWH - это физически целостный реально существующий объект
        
        more about star up-bottom
        more about type2
      - Kimball - звезда ETL - bottom-up
        seems to be more popular
        
        use :star: for both atomic and aggregate data (mart)
        
        DWH - "virtual" object
        like коллекция datamart, которые могут быть пространственно разобщенными.
        
        more about :snowflake: bottom-up
        more about type4 (hist_table)
      - Common
        
        Goal: to implement DWH
        
        requirements:
        
        the most effective exploration of the source data
        
        data accurate and actual (works with both BUS and CIF)
        
        Need to:
        
        store both Atomic + Aggregated data at a same time
      - who is fucken Linstedt ??!
        
        Data Vault
        Load all of the data, all of the time
        
        Hub - business Entity (used for DIM)
        business key (VIN ІНН)
        primary key - MD5 or SHA-1 (type bunary)
        
        Link table - to join HUBs (used for FACT)
        
        Satellite - enriched data from HUB or Link
        
        Point-In-Time - table
        to store SCD with different speed of change
        
        Bridge Table - enriched table with b-key ?? what exactly
        
        Info Marts based on BUSINESS VAULT ( PIT, BRIDGE)
    - - star
      - snowflake
    - - BUS шина
      - CIF info fabrique
- - - - Pandas
        
        DataFrame
        
        2D labeled, size-mutable tabular structure
        
        df.parse.to_csv()
        
        df = pd.read_json()
        df = pd.ExcelFile()
      - pymongo
        
        collection (json)
        
        document
        
        insert()
        
        update(query, update, options)¶
        
        find(query, projection)
        
        MongoClient('localhost', 27017)
        
        client.test_database
  - - - DataaS
        
        Similar to SaaS. it’s an API, which returns data e.g. curEx, weather
        
        Strava, Yahoo! Weather, Wikipedia, Spotify, UPS, Google Cloud Natural
      - IaaS
        
        You manage: OS, runtime, APP & Data
        
        DigitalOcean, AWS, Microsoft Azure, Google Compute Engine (GCE)
      - SaaS
        
        They manage everything
        
        Google Apps, Dropbox, WebEx
      - PaaS
        
        You manage: APP & Data
        
        Windows Azure, AWS Elastic Beanstalk, Google App Engine
      - On-Premise
        
        You manage everything