Please enable JavaScript.
Coggle requires JavaScript to display documents.
TOPIC 1: INTRODUCTION TO THE WORLD OF BIG DATA - Coggle Diagram
Understanding Big Data
Definition: Enormous amount of generated data that are too large in size, complex in nature (structured/unstructured), and arriving at high velocity.
Characteristics
-
Velocity: The speed at which data is generated and the rate at which data is processed and analyzed.
-
Big Data Technology: Apache Hadoop is an open-sources platform and Hadoop is a framework for storing and processing the data
Traditional Database
-
Limitations
- Difficult to scale up (adding more resources to a single system/server).
| Effect: expensive, time-consuming, and prone to failure.
- Difficult to maintain high performance when dealing with large varieties and velocities of data.
| Effect: Affect the efficiency and accuracy of data processing and analysis.
Big Data Life Cycle
-
2) Data Aggregation
I) Data acquisition by collecting and retrieving data from multiple sources, then transmitting the data to the storage platform.
II) Data Preprocessing for effective, efficient, reliable, error-free, consistent and accurate data analysis.
-
ii) Integration data: combining data from different sources to give the end users a unifies data view (to avoid data redundancies).
iii) Data Reduction: reducing the volume/dimension of the data through data compression.
| Types of data reduction:
a) Lossless: original data is retrieved back without any loss of information.
b) Lossy: data retrieval is only partial
iv) Data Transformation: transforming/consolidating the data using ETL tools into an appropriate format and converting them into logical and meaningful information for data management and analysis
| Strategies of data transformation:
a) Smoothing: removes noise by incorporating binning, clustering, and regression.
b) Aggregation: applies summary/aggregation on the data to give a consolidated data.
c) Generalization: generalized to a higher level overlooking the attributes at a lower level.
d) Discretization: raw values in the data are replaced by conceptual/interval labels.
-
3) Data Analytics: combines big data technologies and analytic tools to allow the end users to make decisions based on the results.
4) Data Visualization: assisting the end users to gain insights through line graphs, bar charts, scatterplots, bubble plots, or pie charts.
-
Big Data Applications
- Banking and Securities: customer data analytics
- Healthcare sector: storing patient data
- Marketing: analyzing customer purchase history to reach the right customers
- Web Analysis: social media data/ data from search engines - to broadcast ads based on their interests
- Call Center Analytics: identify recurring problems
- Agriculture: sensors to optimize crop efficiency - big data used in analyzing the sensor data