Please enable JavaScript.

Coggle requires JavaScript to display documents.

EMR - Coggle Diagram

- - - - need to specify the executor size
      - redundant inter process communication
  - - - iam
      - strong consistency
      - encryption
      - multipart uploads
      - pushdown using s3 select
  - - - Cloudwatch metrics
      - Prometeus and grafana
      - Splunk
      - Unravel data
      - data dog
      - sumo logic
  - - - Fully managed IDE
      - SSO
      - Integrated with GIT repo
      - Steps to connect
        
        Login with corporate credentials
      - Workspaces
        
        Saving your notebooks
      - File Browser, attach, kernel management, Git Operations, workspace git link, sql explorer, collaborate, sample notebooks
      - Deep integration between EMR and SageMaker Studio
- - - - Spark SQL
        
        Hive Like queries
        
        Advance file format like parquet, orc etc
        
        BI Tools
        
        Metastore
        
        Amazon RDS or MySQL based megastore
        
        AWS Glue Data catalog as metastore
        
        API - Dataframes
        
        are successors to RDD - available in Scala, java, pays-ark, R
        
        Uses spark session instead of sparkconteext
        
        Structured and semi structured - manual or auto discovery
        
        more optimisations - catalyst optimiser, reduced serialisation tasks, better GC
        
        Errors detected at runtime
        
        apark dataframe API's and pure sparkSQL support
        
        DataSets API
        
        Successors to RDD
        
        Seal and Java support available
        
        Provides type safety and compile time safety
        
        Performance is typically better than RDDs but data frame apis offer the best performance
      - Streaming
        
        Process streaming data
        
        Micro batches
        
        Managed streaming for Kafka
        
        Structured streaming
      - Spark R
      - Spark ML
        
        used by data scientist
        
        Training data
        
        data prep
        
        Split
        
        70
        
        Train
        
        30
        
        Test
        
        Model
        
        Prediction
        
        Sagemaker Spark ML
        
        Connect to sagemaker ecosystem
      - Graph X
      - what is spark core
        
        Oldest spark data structure - available in Scala, Java, Pyspark
        
        Uses Spark Context
        
        Works for both structured and instructed data
        
        Low Level API access
        
        Limited Efficiency
        
        Serialisation and GC operations are costly
        
        No built in optimisation engine
      - Spark on YARN Workflow
        
        Client (execute block)
        
        Driver (Spark Context)
        
        DAG Compier (Fetch DAG & Linegage)
        
        Cluster Manager (YARN) --> Excuse DAG
        
        Resource manager (Launch Yarn Application)
        
        App Master
        
        2 more items...
    - - Batch processing ETL using apache spark
        
        Data Sources
        
        S3
        
        EMR - data transformation with spark sql or data source apis
        
        Redshift (Warehouse)
        
        S3
        
        1 more item...
      - Real time data
        
        AWS Lambda Kafka producers running on lambda write data to ask topic
        
        Amazon MSK
        
        Amazon EMR spark streaming
        
        S3
        
        Redshift
        
        1 more item...
      - Interactive transforms using SQL or core API's
        
        Amazon EMR (Jupyterhub
        
        S3
        
        Quicksight
  - - - RDDS
      - dataframes/datasets
      - wide transforms
        
        RDD - Groupie
    - - RDD
      - Dataframes