Total Pageviews

Wednesday, 11 November 2015

pyspider-用python写的搜索引擎程序

A Powerful Spider(Web Crawler) System in Python. 
  • Write script in Python
  • Powerful WebUI with script editor, task monitor, project manager and result viewer
  • MySQLMongoDBRedisSQLitePostgreSQL with SQLAlchemy as database backend
  • RabbitMQBeanstalkRedis and Kombu as message queue
  • Task priority, retry, periodical, recrawl by age, etc...
  • Distributed architecture, Crawl Javascript pages, Python 2&3, etc...

Installation

from  https://github.com/binux/pyspider