Please enable JavaScript.

Coggle requires JavaScript to display documents.

Big Data-(pySpark) - Coggle Diagram

- - - - SparkConf
        Point to Masternode and application Name ...
        
        Load Operations ...
        
        Prepare Tables ...
        
        Data Analysis using SQL Interface
        
        Stop Spark Session
        spark.stop()
        
        Plotting with Koalas
        gdf.pig.plot(kind="hist")
        
        Spark DataFrame from/to Koalas DataFrame ...
        
        Multi Data frame Join Oprations ...
        
        Prepare DataFrame with User Define Function ...
        
        InLine Column Casting
        Using SQL Expresstion ...
    - - Load/Data Wrangliing with Data Frame ..
      - Load/Data Wrangliing with Spark SQL ..
      - Deal with Neasted Object For Data Wrangliing
        Solution-1
        Solution-2
        Solution-2
    - - Using Accumlator as Global variable
      - Spark Brodcaster with spark context
      - Known port for Sparks ...
      - Debug with Spark UI
      - Set Log Level i.e.info,error etc..
      - Deal With DataSkew Issue
        
        Reason Data is imbalance between node
        
        Solution:Spread data across the worker equally
        
        Run small job to identify issues
        
        use repartition() and pass number of core,It'll nicely
        Spread out.
    - - Load Data (Import) ...
        Load Data(Load into Memory) ...
        
        Transform Data ...
        
        Training Data ...
        
        Model Evolultion ..
        
        Predictions...
        
        Fetch Parameter Information ...
        
        Feature Engineering
        
        Create Only Two Column
        (Feature/Labels)
        
        PipeLine Reference...
        
        Feature Engineering Reference...
  - - - Use VSCode to Connect with Container and Set various Software
  - - - Convert RDD to DataFrame
        
        Apply SQL for Transformation
        
        Apply ML
        
        Up Stream to Other Services
        
        UDF within sql Statement
        
        Convert to SQL Shell on port:10000 ...