Please enable JavaScript.

Coggle requires JavaScript to display documents.

Big Data (BigQuery (Load Data (Formats (CSV, AVRO, JSON, ORC, Parquet,…

- - - - list and start jobs
    - - create datasets
    - - delete dataset
  - - - table
  - - - sub sec queries
      - column storage
      - batch/streams based
        (for input data)
  - - - same scheema
    - - transformed after loaded:
        
        transformation not intensive
        
        data amount is ~ the same
        
        views can be used to transform data on fly
    - - CSV
      - AVRO
      - JSON
      - ORC
      - Parquet
      - Datastore/Firestore files
    - - daily - most frequent
  - - - <dataset>. TABLES
      - <dataset>.INFORMATION_SCHEMA
    - - DATE/ TIMESTAMP column
      - integer column
      - ingestion time
      - CREATE TABLE ... PARTITION BY ...
    - - sorted by clustering column - similar to INDEX
      - CREATE TABLE ... CLUSTER BY ...
      - auto re clustering - as with time sorting can be lost
      - table MUST be partitioned
      - in case clustered by several columns - where must contain filters in the same order for effitiency
- - - - keep HDFS:
        when a lot of writes (appends),
        a lot of files,
        folder's manipulations
        when <10ms latency expected
  - - - SSH
- - - - sink
  - - - by time intervals
    - - overlap
    - - difference between expected
        and real arrival time
      - watermark is used
      - at watermark window flush,
        aggregations are triggered
        called 'AfterWatermark'
      - can setup custom triggers
        based on time inside message
        'AfterProcessingTime'
        or 'AfterCount'
      - Java allows to process late data
        (after watermark)