A distributed Sina Weibo Search spider base on Scrapy and Redis.
tpeng pengtaoo@gmail.com
$ sudo apt-get install mysql-server
$ sudo apt-get install redis-server
$ sudo apt-get install python-mysqldb
$ sudo pip install -r requirements.txt
- put your keywords in items.txt
- scrapy crawl weibosearch -a username=your_weibo_account:your_weibo_pw
- add another spider with scrapy crawl weibosearch -a username=another_weibo_account:another_weibo_pw