Please enable JavaScript.
Coggle requires JavaScript to display documents.
TOPIC 2: BIG DATA STORAGE CONCEPT - Coggle Diagram
TOPIC 2: BIG DATA STORAGE CONCEPT
CLUSTER COMPUTING
The cluster components are connected through local area networks (LANs).
Used for
high availability
(always there, available at all time) as well as
load balancing
with better system performance and reliability.
Definition
: Distributed/parallel computing system comprising multiple stand-alone PCs connected working as a single, integrated, highly available resource.
Benefits: high availability, scalable performance, fault tolerance, cost-effective commodity hardware.
Types of Cluster
1)
High Availability
Designed to
minimize downtime
and provide
uninterrupted service when nodes (servers) fail
.
Example of important systems: Healthcare, defense, banking, aerospace
If one node failed, the cluster detects the failure and
removes the failed node from service
.
All nodes
have access to a shared storage
and synchronize data.
Without clustering, if one server failed, the apps will crash and will not be available until the server is up again.
Types of Server
i) Primary Server (Active)
ii) Secondary Server (Passive/ Standby)
2)
Load Balancing
(traffic cop)
Used to
improvise the overall performance
when the computational workload has to be share among the cluster nodes.
Not focus on one machine only --
more machine, more balance
. High load, machine might crash (as there are no other machine can backup the only machine)
If one node failed, the load balancer
distribute the workload of the one failed node among three other cluster nodes
.
Example: A food delivery apps during lunch rush
Types of Cluster Structure
I)
Symmetric Clusters
All nodes active, equal load sharing, shared/synced data and seamless scale (can add more active servers during peak time).
Each node functions as an individual computer
capable of running applications.
Example: E-commerce platform during sales
II)
Asymmetric Clusters
One machine acts as the head node
and servers as the gateway between the user and the remaining nodes.
Example: Online banking system
Consists of primary server (active) and secondary server (passive/standby)
DISTRIBUTION MODELS
To overcome the difficulty and to cut the cost of buying extensive servers.
Disadvantage: Complex as it introduces as the number of computers added to the cluster increases.
Techniques of Data Distribution
A)
Sharding
The process of
partitioning very large data sets into smaller and easily manageable chunks (shards)
.
The partitioned shards are
stored by distributing them across multiple machines (nodes)
.
No two shards of the same file are stored in the same node.
Example: E-hailing apps (Grab)
splits (shards)
their customer and trip data, drivers, and riders
by region
,
| Use
horizontal scalability
(as traffic grows in a new market, they can simply spin up one region (shard) without affecting existing regions.
| Isolated outages -- if 1 shard fail, other shards still okay.
B)
Replication
The process of
creating copies of the same set of data into blocks of copied data (replica) across multiple servers
.
One node crashes, the data stored in that crashed node will be lost.
Example: Streaming service (Netflix) where the primary copy (origin server) are copied to local server: replicated copies (edge servers)
| If origin server crashes, the edge servers still okay.
Types of Replication Model
a)
Master-Slave Model
Master = primary database, Slaves = replica databases
One centralized device (master) controls other devices (slaves).
The flow of control is only from master to slaves.
All data are written on the master node, and the same data is replicated over the slave nodes.
Example: Database replication for a news website where
master node (primary database) handles all writes
while
slave nodes (replica databases) handle all reads
.
b)
Peer-to-Peer Model
All the nodes have the same responsibility and level (no central server) where both act as client and server, and the workload is partitioned among the nodes.
Example: Downloading a large open-source software file
C)
Sharding and Replication
The data is sharded, then duplicated.
The techniques are combined to make the system fault-tolerant and highly available.
DISTRIBUTED FILE SYSTEM
Files are distributed across the nodes.
DFS provides access to more than one client simultaneously.
Example: Google Drive
TYPES OF DATABASE
1)
Relational Database
Organize data into tables of rows (records) and columns (attributes/fields).
Flat database = a database with only one table.
Relational database = a database with two or more tables that are related.
2)
Non-Relational Database
Does not rely on tables as its primary structure.
Examples of Database
I)
Relational Database Management System (RDBMS) Database
(ACID)
Vertically scalable.
ACID Properties
-- supports data that adhere to a specific schema:
Atomicity (only 1 data)
Consistency (all data must be same type)
Isolation
Durability (not always changing)
II)
Not Only SQL (NoSQL) Database
(CAP & BASE)
Non-structured database which includes all non-relational database (not in table form) to
cater different types of data
.
CAP Theorem
Consistency
Availability
Partition Tolerance (same information, different types and size)
BASE
-- the storage devices provide eventual concictency:
Basically
Available
Soft State
Eventually Consistenet
III)
NewSQL Databases
Provide scalable performance similar to NoSQL systems combining the ACID properties of RDBMS to
cater large data
.
Horizontally scalable, fault-tolerance, distributed in nature, and support relational data models with 3 layers (administrative, transactional, storage).
Highly scalable and operates in shared-nothing architecture.
Types of NewSQL Databases: Clustrix, NuoDB, VoltDB, MemSQL.
STORAGE SCALABILITY
Scalability = the ability of the system to meet the increasing demand for storage capacity.
Scaling to increase performance and efficiency.
Ways to scale storage platforms
A)
Scaling-up (Vertical Scalability)
Increase the original memory storage.
B)
Scaling-out (Horizontal Scalability)
Multiple the memory storage by increasing the number of memory storage.