Using multiple spiders at in the project in Scrapy
I wanna know if it is possible to us开发者_JAVA技巧e multiple spiders within the same project together. Actually I need 2 spiders. The first one gathers the links on which the second spider should scrape. They both work on the same website, so the domain is similar.Is it possible? If yes can you give me an example? Thanks
Maybe this is what you're looking for:
def parse(self, response):
# parse the links (aka your first spider)
for link in hxs('//XPATH'):
yield Request(link.extract(), callback=self.parse_link)
def parse_link(self, response):
# continue parsing (aka your second spider)
Hope this help you :)
精彩评论