Please enable JavaScript.
Coggle requires JavaScript to display documents.
Scraper (Distributor (Client (5.1 Initial setup react + redux, 5.2 API…
Scraper
Distributor
Scrapers (Applied to all websites to be scraped)
3.1 Create selectors.py containing various element selectors for scraping websites
3.2 Create tests for selectors to make sure they are targeting the right data
3.3 Create heavy page request tests to trigger bot detection
3.4 Use different methods and repeat 5.3 till it passes the test
3.5 Register scraping task to Celery
3.6 Register an endpoint for the task to access
Initial setup
Initiate virtual environment
1.1 Modules
requests
bs3
flask
flask-restplus
marshmallow
sqlite
celery
gunicorn
marshmallow-slqlachemy
sqlalchemy
Celery
1.2 broker setup (rabbit-mq)
1.2 result setup (n/a)
1.3 API Flask integration
API (Applied to each endpoint)
4.1 Registering endpoint to flask
4.2 Database model creation
4.3 Marshmallow schema integrate to database model
4.4 main endpoint creation
4.5 Sub endpoints creation
Client
5.1 Initial setup react + redux
5.2 API helper functions
5.3 Component creation
Dunno if you'd like an SPA or Multi Page
5.4 API integration tests
Kindly note if we're going to do user authentication
Also need to read the template docs before starting here
I'll build the prototype behind the scenes for debugging and testing