Please enable JavaScript.

Coggle requires JavaScript to display documents.

Data engineering _old MMap (5 ways to deploy (Serverless (pros (economical…

- - - - docker run -p 8888:8888 jupyter/scipy-notebook:2c80cf3537ca
    - - docker exec -it <mycontainer> bash
    - - docker cp wine.data afb6b3df4b9a:/home/jovyan/wine.data
- - - - handy for demos, just to show someone at meeting
      - great for hacking with webhooks (tweelio)
      - fast and easy
    - - stops when you close laptop
      - random domains (can have one with paid account)
      - doesn’t scale (up to 12 people at the same time…)
  - - - one app 24/7 for free (the easiest way to create such app, can get one app running for free)
      - zero server management
    - - scaling is easy but gets pricey. if you need to run your app at more then one server at heroku at some period of time, probably you need to look at other options
      - harder server customization
  - - - our code is only gonna be running when someone actually need to use it
    - - pay only for compute time you consume
      - all big cloud providers have their own serverless feature
    - - economical for small-to-medium loads (if you don’t need smth 24/7 available, but just need it to be quickly available at any time of the day)
      - good for "spikey” traffic (unexpected spikes of traffic to your server. AWS will take care of all scaling for you)
      - zero server config (even less then heroku)
    - - relatively new technique (fast moving ecosystem, not much to read)
      - less fun without Zappa, can be tricky to troubleshoot (b/c its a combination of another Amazon web services behind the scene)
  - - - the way most big companies, and a lot of people in the world run their code in the cloud
      - all big cloud providers have their own VM server
    - - full control
      - scales as much as your wallet
      - is still economical if you’re careful (set things up in the right way)
    - - much more work for you
      - there’s lot more to learn (if set up for your org, need to learn about configuration management, monitoring, alerting system, etc.)
      - harder to predict ultimate costs (especially if you add like load balancers)
  - - - much more lightweight than VM and easier to manipulate, but control like in VM
      - helps dev/prod parity. once you’re running your app in production, you’ll find that a lot of your biggest errors happening b/c smth that was set up in your local env is not the same way that is actually running in prod. Docker is great to help with that
      - nice for microservices
    - - new technique
      - works best when all your team is in it
      - has its own learning curve
- - - - ls, cd, pwd, man
    - - mkdir, nano, touch, rm, mv, cp
    - - pipe: combine existing programs in new ways
      - filter is a program like wc or sort that transforms a stream of input into a stream of output
      - The best way to use the shell is to use pipes to combine simple single-purpose programs (filters)
      - wc, [*?.], >, >>, <, cat, less, sort, head, echo, |, uniq, cut, grep
    - - for, do, done, $fn, ${fn}
    - - sh, bash, "$1”, "$@ ", sed '$d’, sh -x, awk {'print $2'}
    - - grep -nwoivE
      - find -type -name -mtime -user -maxdepth
      - $(command)
  - - - man, more, h, q, b, space, /, n
    - - ssh, scp
    - - wget, curl, lftp
    - - cat /etc/group, ls -F, chmod u/g/a=, find . -type f -perm -u=x
    - - /, /bin, /boot, /etc, /dev, /home, ...
      - tree -CdiugshDFvrt -L level -P pattern -I pattern -o file —forepart —dirsfirst —filelimit #
    - - ps -f, program &, jobs, fg %1 ^Z, kill
    - - cp -i ~/.bashrc.bak ~/.bashrc, source(.) .bashrc
      - alias bashrc="nano ~/.bashrc && source ~/.bashrc”
      - alias | grep 'Visual Studio'
    - - set, echo $PATH, export SECRET_IDENTITY=Dracula
- - - - less abstraction (directly modifying code and data objects)
      - less structure to the code (less modularity)
      - helping you and others understand the code and data
      - iteration speed on small pieces of code
    - - Understandability
        
        Markdown cells to describe our observations and assumptions
        
        Small pieces of code for the actual logic
        
        Lots of visualizations and counts
      - Minimal abstractions
        
        Most code shouldn't be in functions (should feel more object-oriented)
  - - - more abstraction (modifying parameter values instead)
      - more structure to the code (more modularity)
      - helping a computer run code automatically
      - iteration speed on the full pipeline
    - - Healthy abstractions
        
        Code should generalize to be compatible with similar data sources
        
        Code shouldn't be so general that it becomes cumbersome to understand
      - Pipeline stability
        
        Reliability should match how frequently its run (daily? weekly? monthly?)
  - - - Use optional, positional, and required arguments
      - Use if / then statements along with Boolean input values within the functions
      - Use new data structures (dictionaries, lists, etc.) to represent custom actions for specific datasets