Can I use Scrapy functions inside Django classes
I have build a website in Djang开发者_运维知识库o. I need to use the web crawling features. So I installed Scrapy. Scrapy is working, as stated in their tutorial, by using
scrapy startproject dmoz
But that works on a diff folder with its own files.
Is it possible to use Scrapy functions in Django classes and put data in Django variables?
Crawling is a long, time consuming task. It doesn't mix well with short HTTP requests. Therefore, you should use a user interface (presumably written in Django) to fire off Scrapy crawls, and a shared storage to monitor the state of the crawl.
But you wouldn't want to crawl during the lifetime of a HTTP requests. That's not how Scrapy is meant to be used.
精彩评论