Skip to main content
 首页 » 编程设计

python之Scrapy不同数量的url返回

2025年05月04日7tuyile006

我已经为固定域内的爬虫构建了一个爬虫,并提取与修复正则表达式匹配的 url。如果看到特定的 url,爬虫就会跟踪该链接。爬虫可以完美地提取 url,但每次我运行爬虫时,它都会返回不同数量的链接,即每次运行时链接的数量都不同。我正在使用 Scrapy 进行抓取。这是 scrapy 的问题吗?代码是:

class MySpider(CrawlSpider): 
   name = "xyz" 
   allowed_domains = ["xyz.nl"] 
   start_urls = ["http://www.xyz.nl/Vacancies"]  
   rules = (Rule(SgmlLinkExtractor(allow=[r'\/V-\d{7}\/[\w\S]+']), callback='parse_item'),Rule(SgmlLinkExtractor(allow=[r'\?page\=\d+\&sortCriteria\=1']), follow=True),) 
 
 
 
 def parse_item(self, response): 
 
  outputfile = open('urllist.txt','a') 
  print response.url 
  outputfile.write(response.url+'\n') 

请您参考如下方法:

与其手动编写链接并在 parse_item() 方法中使用 a 模式打开文件,不如使用 scrapy 的内置 item exporters .定义一个带有链接字段的项目:

from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.item import Item, Field 
 
 
class MyItem(Item): 
    url = Field() 
 
 
class MySpider(CrawlSpider): 
    name = "xyz" 
    allowed_domains = ["xyz.nl"] 
    start_urls = ["http://www.xyz.nl/Vacancies"] 
    rules = (Rule(SgmlLinkExtractor(allow=[r'\/V-\d{7}\/[\w\S]+']), callback='parse_item'), 
             Rule(SgmlLinkExtractor(allow=[r'\?page\=\d+\&sortCriteria\=1']), follow=True),) 
 
    def parse_item(self, response): 
        item = MyItem() 
        item['url'] = response.url 
        yield item