Please enable JavaScript.

Coggle requires JavaScript to display documents.

Big Data (EMR (Using Open Source Frameworks (Apache Hadoop (Modules…

- - - - Distributed Processing of large datasets across clusters of computers using simple programming models
      - each machine in a cluster has local compute and storage
      - Modules
        
        Hadoop Common
        
        Core
        
        Common libraries for other modules
        
        Provides abstractions for OS and Filesystem
        
        HDFS
        
        Distributed Filesystem
        
        Fault tolerant
        
        Data Replication
        
        YARN
        
        Yeat another resource negotiatore
        
        Job scheduling and cluster resource managements
        
        MapReduce
        
        Programming Framework for distributed processing large datasets
        
        Supports eiterh Filesystem (usntructered) or database (structured)
        
        Tasks
        
        Map
        
        Split data
        
        Reduce
        
        Data is reduced to create output file
        
        Input is data from Map task
        
        Input
        
        Shuffle
        
        REsult
  - - - Core Node
        
        equals HAdoop Slave Node
        
        runs Tasks
        
        Store data as part of HDFS or EMRFS
        
        Runs Data Node daemon
        
        Manages Data blobs and read write requests
        
        Runs Node Manager Daemon
        
        Manages Resources on the Core Node
        
        Application Master - executes application workloads
      - Task Node
        
        Similar to core nodes
        
        No HDFS
        
        Can be added or removed from running clulster
        
        Provide addition compute capacity
      - Master Node
        
        Single node in a cluster
        
        Manages cluster resources
        
        Coordinates distribution and parallel execution
        
        Monitor health
    - - Categorize EC2 instances according to node types
      - Up to 50 Instance groups
      - Up to 48 task instance groups
      - Handy to have instance group per EC2 instance type
    - - HDFS
        
        provides single global namespace
        
        maintained by master node
        
        Fiels a split into blocks
        
        Default replication factor of 3
        
        64MB to 256MB blocks
        
        Block size can be set by file
      - EMRFS
        
        Allows to use S3 directly
        
        allows to use reliability, durability and scalability of s3
        
        Resize and terminate clusters without data loss
        
        Consistent Views
        
        allows to check for read after write conistency and do retries if necessary
        
        Done using metadata in dynamodb
      - HDFS and EMRFS can be used together
        
        data cna be copied using S3DistCp
        
        perfectg for high I/O performance with instance store families
        
        perfect if you process same dataset frequently
    - - Long running cluster
        
        Used if tasks are run frequently and data is too large to load each time
        
        Auto termination is disabled by default
        
        Stayes up and running
      - Transient clusters
        
        Only up while task is runnign
        
        Data in S3, Hive meta store in mysqlrds
        
        saves cost
  - - - use M3/M4 instances
      - Batch oriented
    - - P2/C3 and C4 instance types
    - - use R instance types
    - - I2/I3
      - D2
- - - - Counterpart to Apache Kafka
      - Used for processing by custom applications
      - Scaling needs to be done manually
      - Shard
        
        1/MB /sec Input/write
        
        2/MB Output/read
        
        5 transactions/sec for reads
        
        1000 transactions/sec for writes
        
        You can shard split/merge
      - Record
        
        consists of partition key,sequence number, data blob
      - Kinesis Stream Ingestion Methods
        
        Kinesis API
        
        AWS SDK for Java
        
        PutRecord
        
        PutRecords
        
        Amazon Kinesis Producer Library
        
        For Producers requiring high throughput
        
        write to one ore more kinesis streams with auto retry
        
        Can write to multiple shards per requets
        
        Aggregates multiple user records into stream records
        
        Batching/Collection
        
        sends multiple records in one http call /PutRecords)
        
        Can detect partial failures
        
        Kinesis Agent
        
        Can monitor directories
        
        Can preprocess
        
        Multiline to single line
        
        Convert Log to JSON
      - Consume
        
        Kinesis Client Library
        
        Consume and process tasks
        
        Consists of Workers
        
        Shards are mapped to workers to process records
        
        supports deaggregation of records which where aggregated by KPL
        
        uses DynamoDB
        
        might need additional throughput
        
        table to keep track of processed records
      - Further Ingestion
        
        Kinesis Connector Library
        
        Java based
        
        allows ingestion of Data into S3/DynamoDB/ElasticSearch/Redshift
        
        Spark Streaming
        
        Using Lambda
        
        Stream can be polled
    - - Used to put data into other AWS Services e.g. S3, Redshift,ElastiSearch, Splunk
      - Scales automatically
- - - - Partitions - underlying storage nodes
        
        Can stora 10 GB data
        
        Support 3000RCU / 1000 WCU
        
        capacity and performance relationship to number of partitions
        
        are increased automatically but not decreased
        
        Partition calculation
        
        Performance
        
        (Desired RCU / 3000) + (Desired WCU /1000)
        
        Size
        
        Data Size / 10 GB
        
        Total
        
        MAX (Prefomance,SIze)
      - Local Secondary Index
        
        Can only be created at table creation
        
        Allows efficient filtering on additional attributes
        
        Shares global partition key
        
        data written is copied async to LSI (Eventual consistency)
        
        Share RSU/WCU
        
        Sparse Index (only has an item if sort key attribute
        is in table item)
        
        Non key values are not stored in LSI unless projected
        
        Carefulle plan LSI and projections
        
        Item update on main table requires 2 operations on LSI item (Delete/Create)
      - Global Secondary Index
        
        Attribute projection options
        
        Keys only
        
        Include (custom projection values)
        
        All
        
        Separate set of RCU/WCU
        
        Only eventually consistent ready
      - Streams
        
        records table changes
        
        stores 24 hours of changes
        
        streams endpoint
        
        allows processing
        
        streams.dynamodb.us-west-2.amazonaws.com
        
        changes occur only once and near realtime
        
        only partition keys are stored in stream
        
        different views
        
        Keys only
        
        OLD_IMAGE
        
        NEW_IMAGE
        
        NEW_AND_OLD_IMAGE
      - Performance
        
        Key Design
        
        uniform write pattern across all values
        
        uniform write pattern across time
        
        consider created/hybrid values
        
        should have many distinct values
        
        Avoid hot keys/partitions
        
        Avoid hot times/burst load
        
        change your application to spread writes over time
        
        Use sqs a managed write buffer
        
        Rely on burst capacity (Risky)
        
        Caching (with cache updates using stream/lambda)