Please enable JavaScript.
Coggle requires JavaScript to display documents.
project_Overview (Preload (1)SSH to landing server (Source Location…
project_Overview
Preload
1)SSH to landing server
Source Location
/staging/push_zone/probe/others/
2) List the Files (
To know Location using Find command
)
3) SFTP from server to edgenode(
Loc A
)
4)Put entry into
MasterList
5) Put chunk file details for
every batch
6)Execute above all steps for every 1/2 hour
Script_Location
/apps/kogentix/ingestion/probe/scripts
(Edge_Node)
Landing_Server to Edgenode
dpi_eventx_load_new.sh
probe_event_load.hql
dpi_hdfs_load_new_version_3.sh
Hive table load
dpi_hive_load.sh
hql scripts
probe_event_load.hql
Hive table location
/data/discovery/amp/probe
/probe_eventx
Load
1)Reads chunk file &put it in processing directory (
Batch
)
2) Copy from
Loc A
to
LocB
4) Put into HDFS
/data/discovery/amp/probe_data/dpi_processing/eventx
3) Unzip at
LocB
Hive Load
1)Find hql File and update it
2) Based on property file we will modify the columns as Null
3)update load data into Hive Temp table
(
probe_eventX
- TableName)
4) Execute hql file