Please enable JavaScript.

Coggle requires JavaScript to display documents.

Solution space (Cloudwatch + Lambda + AWS ECS (Problems (Less…

- - - - Temporary failures can
        go unnoticed
    - - Potentially insecure &
        error prone
    - - Schedule is behind cloudwatch
      - Conditionals behind Lambda
    - - Solution can't be delivered
        as quickly, scope of work
        has slightly increased
    - - Custom code required
        for resource pooling
        
        This will have its
        own testing effort
- - - - Operationally
        expensive
    - - 2.1. Manually
        restart/redeploy
        services/crons
        
        Error prone
      - 2.2 Local logs
        might be lost
        
        Reduce debug-ability
    - - Insecure &
        error prone
    - - Resource management
        is inefficient and done manually
- - - - EC2 billing is by hour,
        we might not be
        saving much
      - Lambda is cheap,
        but not free
    - - Potentially insecure &
        error prone
    - - Schedule is behind cloudwatch
      - Conditionals behind Lambda
    - - Temporary failures can
        go unnoticed
- - - - Not a solution until
        its up for other use
        cases too
    - - 2.1. Manually
        restart/redeploy
        services/schedules
        
        Error prone
      - 2.2 Local logs
        might be lost
        
        Reduce debug-ability
    - - Potentially quite expensive
  - - - Good UX
- - - - Use airflow's plugin mechanism
        to build an interface for changing
        schedule
  - - - Good UX
  - - - Not a solution until
        its up for other use
        cases too
    - - 2.1. Manually
        restart/redeploy
        services/schedules
        
        Error prone
      - 2.2 Local logs
        might be lost
        
        Reduce debug-ability
- - - - Write a basic dag to run
        the python executable as task
        for each inventory type
      - Package the python executable into
        a docker container
      - Jenkins to push the docker
        container in Amazon ECS Repo
      - Usable solution
        
        Use airflow's plugin mechanism
        to build an interface for changing
        schedule
        
        Use airflow's pool mechanism
        for resource management
    - - Airflow EC2 will
        always be up & running
        
        Not a solution until
        its up for other use
        cases too
      - The service is not
        highly available
        
        2.1. Manually
        restart/redeploy
        services/schedules
        
        Error prone
        
        2.2 Local logs
        might be lost
        
        Reduce debug-ability
      - More committed to AWS stack
      - Need to change packaging
        & deployment of existing python
        executable to be docker and ECS
        friendly
        
        Solution can't be delivered
        as quickly, scope of work
        has slightly increased
      - ECS has no inbuilt queuing
        mechanism. We're solving it with
        airflow's pools.
        
        Airflow pools are basic.
        Airflow is a workflow manager
        and not resource manager.
    - - Operational costs are reduced
      - Not reinventing how cloud deploys
        should be happening
      - Inventory allocation &
        airflow resources can
        grow independently
      - More maintainable as DAG,
        its definition & schedule are
        explicit
      - Consistent way of finding
        logs & documentation of jobs
        
        Good UX