豆瓣影评爬虫

Tutorial

运行mysql

brew services start mysql
mysql -u root -p
source init.sql

首先，执行爬虫池

cd Proxy_Pool/Run
python3 main.py

启动任务队列

redis-server

获取url放到队列中

python3 spider.py

开始从任务队列中获取任务执行

python3 worker.py

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
__pycache__		__pycache__
flask		flask
proxy_pool		proxy_pool
tests		tests
.DS_Store		.DS_Store
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
data.json		data.json
database.py		database.py
index.html		index.html
init.sql		init.sql
spider_comment.py		spider_comment.py
spider_movie.py		spider_movie.py
utils.py		utils.py
utils.pyc		utils.pyc
worker.py		worker.py