Please enable JavaScript.

Coggle requires JavaScript to display documents.

Big Data, Nosql, Data grid/ cache products, Relational DB, NewSQL,…

- - - - Data access
      - Data integration
  - - - stores metaData
      - Only one server holds it
      - only operates on metaData
      - Handles replications after dataNode failure
      - SPOF
      - Access by Hadoop & API
      - OLAP oriented
      - Job tracker
    - - real data is here
      - can execute on any fileSystem
      - notifies namnode of any problem
      - replication 2x locally, 1x elsewhere
      - Task tracker
    - - hadoop fs -mkdir -p /input
      - hadoop fs -put file.txt
      - hadoop fs -get file.txt
    - - Cluster Size Support++
      - Max Task count++
      - Namenode Availability++ (active/passive)
      - MapRed task repartition++
      - ++programming models
      - YARN introduced
      - NFS Access (R/W)
      - Encryption
      - Architecture
        
        Containers
        
        Ressource manager
        
        Application master
      - Openness
        
        SOLID
        
        Syncsort
        
        ETL
    - - Pseudo-distributed mode
      - Fully distributed
      - Standalone mode
  - - - Table
      - Row
      - Column Family
      - Column qualifier
        
        When inserting
      - Cell
      - Version
    - - foreach column
    - - HBase HMaster
      - Zookeeper
      - Region Servers
        
        client gateway
    - - Creating Table
        
        HTableDescriptor
        
        ht = new HTableDescriptor(TableName.valueOf(tableName))
        
        add family
        
        ht.addFamily(new HColumnDescriptor(family1));
        
        Admin
        
        Admin admin = connection.getAdmin();
        
        if (admin.tableExists(table.getTableName())) { admin.disableTable(table.getTableName()); admin.deleteTable(table.getTableName());
        }
        admin.createTable(table);
      - Put
        
        byte[] row1 = Bytes.toBytes("user1");
        Put p = new Put(row1);
        
        p.addColumn(family1.getBytes(), "name".getBytes(), Bytes.toBytes("ahmed"));
        
        p.addColumn(family1.getBytes(), "address".getBytes(),Bytes.toBytes("tunis"));
        
        table1.put(p);
      - get Table
        
        table1 = connection.getTable(TableName.valueOf(tableName));
      - Get
        
        Get g = new Get(row1);
        
        Result r = table1.get(g);
        
        System.out.println(Bytes.toString(r.getValue(family1.getBytes(), "name".getBytes())));
- - - - With master
      - Without master
    - - Horizontal partitioning
        
        Consistent hashing
        
        P = Hash(o) % n
        
        V-nodes (cassandra)
        
        Gossip (for coherence)
        
        Dynamo
        
        Riak
        
        Uniform repartition
        
        Replication
        
        By access time
      - Vertical partitioning (by column)
    - - Auto-sharding by the system
      - Distribution horizontale
      - Load balancing
  - - - scalable-friendly
      - répartition de charge
      - abstraction quasi-totale de l’infrastructure matérielle
    - - treats (K,V), generates (K,Vs)
    - - Fusions Vs
    - - Restarting all over in case of data loss
      - Traitements Interactifs
        Map Reduce
      - Impersistence
- - - - Driver program (Context)
        
        RDD Graph
        
        One task per partition
        
        Characteristics
        
        Coarse-grained operation
        
        In-memory Computation
        
        Lazy evaluation
        
        Immutability
        
        Fault tolerance
        
        persistence
        
        Partitioning
        
        Location-stickiness
        
        Operations
        
        Transformations
        
        Lazy evaluated operations
        
        Variants
        
        map
        
        flatMap
        
        filter
        
        groupByKey : to be avoided, use reduceByKey
        
        reduceByKey
        
        mapValues
        
        sample
        
        union
        
        distict
        
        sortByKey
        
        join
        
        cogroup
        
        cartesian
        
        Types
        
        Narrow
        
        Wide
        
        Actions
        
        Output, saving, counting, input
        
        Les actions produisent quelque chose qui peut être
        inspecté directement au lieu de créer un autre RDD
        
        Variants
        
        takeSample
        
        saveAsTextFile
        
        take
        
        saveAsSequenceFile
        
        first
        
        countByKey
        
        foreach
        
        count
        
        collect
        
        reduce
        
        DAG schedueler
        
        Into stages
        
        Task sheduler
        
        Scheduler backend
        
        Life cycle
        
        Créez des RDD d'entrée à partir de
        données externes
        
        count, collect... that can be parallelised
        
        transform them lazily
        
        Ask for persistence
      - Cluster manager
      - Nodes (Executors)
      - Client program
  - - - Holds the Spark driver
    - - Driver is in machines
- - - - Json
      - No referential integrity
      - No explicit schema in server
    - - Specific needs
    - - Computation dist
      - Data dist
    - - Distribute computation to nodes
      - Caching
  - - - Simple
      - Horizontal scalability
      - Evolutivity
      - No maintenance required
      - Poor complex structs
      - Key only interrogation
      - Put big cplexity on app layer
    - - Serialisation for exchg
      - Schemaless
      - Auto description
      - Rich structs
      - Flexible typing
      - Serialisation
      - No need for Join
      - No need for transaction
      - Distribution adaptability
      - Simple version manipulation
      - Privileged path
      - Entities no longer autonomous
      - Redundence
      - Slow big requests
    - - High volumetry
      - Column family
        
        predefined
      - columns
        
        are not predefined
        
        regrouped by line
        
        each line has key
        
        easily compressible when similarities are there
      - semi-struct data model
      - superColumns
        
        often used for joins
      - naturally indexed
      - horizontally scalable
      - mapRed used for horizontal scaling
      - not for interconnected data
      - not for cplex data
      - needs maintenance
      - requests are predefined
    - - Connection focus
      - Navigable
      - tools
        
        storage engine
        
        arc description mechanism
      - powerful data model
      - fast for highly connected data
      - performant requesting (CYPHER)
      - sharding, even though there might be solutions.