Skip to main content
 首页 » 编程设计

python之使用 Scrapy 而非 Scraped Items 从表中抓取数据

2025年04月02日17lyj

我是第一次尝试 Scrapy。在做了相当多的研究之后,我掌握了基础知识。现在我试图获取表格的数据。它不工作。 没有抓取任何数据。检查下面的源代码。

设置.py

BOT_NAME = 'car'  
SPIDER_MODULES = ['car.spiders'] 
NEWSPIDER_MODULE ='car.spiders'  
DEFAULT_ITEM_CLASS = 'car.items.Car58Item' 
ITEM_PIPELINES = {'car.pipelines.JsonLinesItemExporter': 300} 

项目.py

from scrapy.item  import Item,Field 
class Car58Item(Item): 
    # define the fields for your item here like: 
    # name = scrapy.Field() 
    url = Field()  
    tip = Field()  
    name = Field()  
    size = Field()  
    region = Field()  
    amt = Field()  

car_spider.py

# -*- coding=utf-8 -*- 
from __future__ import absolute_import 
from scrapy.linkextractors import LinkExtractor 
from scrapy.spiders import CrawlSpider,Rule,Spider 
from car.items import Car58Item 
 
 
class CarSpider (CrawlSpider): 
    name ='car' 
    allowed_domains = ['58.com'] 
    start_urls = ['http://quanguo.58.com/ershouche'] 
    rules = [Rule(LinkExtractor(allow=('/pn\d+')),'parse_item')] #//页面读取策略 
 
def parse_item(self,response): 
    trs = response.xpath("//div[@id='infolist']/table[@class='tbimg']/tr")[1:-2] 
    items = [] 
    #for tr in sel.xpath("id('infolist')/table/tr"): 
    for tr in trs: 
        item = Car58Item() 
        item['url'] = tr.xpath("td[@class='img']/a/@href").extract() 
        item['tip'] = tr.xpath("td[@class='t']/a/font/text()").extract() 
        item['name'] = tr.xpath("td[@class='t']/a[1]/text()").extract() 
        item['size'] = tr.xpath("td[@class='t']/p").extract() 
        item['region'] = tr.xpath("td[@class='tc']/a/text()").extract() 
        item['amt'] = tr.xpath("td[@class='tc']/b/text()").extract() 
        items.append(item) 
    return items 

管道.py

# -*- coding: utf-8 -*- 
import json 
import codecs 
 
class JsonLinesItemExporter(object): 
    def __init__(self): 
       self.file = codecs.open('car.json','w',encoding='utf-8') 
 
    def process_item(self, items, spider): 
        line = json.dumps(dict(items),ensure_ascii=False) + "\n" 
        self.file.write(line) 
        return items 
 
    def spider_closed(self,spider): 
        self.file.close() 

我在 shell 中运行 scrapy

 [mayuping@i~/PycharmProjects/car] $**scrapy crawl car** 
 
2016-05-18 10:35:36 [scrapy] INFO: Scrapy 1.0.6 started (bot: car) 
2016-05-18 10:35:36 [scrapy] INFO: Optional features available: ssl, http11 
2016-05-18 10:35:36 [scrapy] INFO: Overridden settings: {'DEFAULT_ITEM_CLASS': 'car.items.Car58Item', 'NEWSPIDER_MODULE': 'car.spiders', 'SPIDER_MODULES': ['car.spiders'], 'BOT_NAME': 'car'} 
2016-05-18 10:35:36 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState 
2016-05-18 10:35:36 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2016-05-18 10:35:36 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2016-05-18 10:35:36 [scrapy] INFO: Enabled item pipelines: JsonLinesItemExporter 
2016-05-18 10:35:36 [scrapy] INFO: Spider opened 
2016-05-18 10:35:36 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2016-05-18 10:35:36 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 
2016-05-18 10:35:37 [scrapy] DEBUG: Redirecting (301) to <GET http://quanguo.58.com/ershouche/> from <GET http://quanguo.58.com/ershouche> 
2016-05-18 10:35:39 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/> (referer: None) 
2016-05-18 10:35:40 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn2/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:42 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn7/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:42 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn6/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:42 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn12/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:43 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn11/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:44 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn9/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:45 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn8/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:45 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn5/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:46 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn10/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:46 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn4/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:46 [scrapy] DEBUG: Crawled (200) <GET http://quanguo.58.com/ershouche/pn3/> (referer: http://quanguo.58.com/ershouche/) 
2016-05-18 10:35:47 [scrapy] INFO: Closing spider (finished) 
2016-05-18 10:35:47 [scrapy] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 5550, 
 'downloader/request_count': 13, 
 'downloader/request_method_count/GET': 13, 
 'downloader/response_bytes': 339809, 
 'downloader/response_count': 13, 
 'downloader/response_status_count/200': 12, 
 'downloader/response_status_count/301': 1, 
 'finish_reason': 'finished', 
 'finish_time': datetime.datetime(2016, 5, 18, 2, 35, 47, 45187), 
 'log_count/DEBUG': 14, 
 'log_count/INFO': 7, 
 'request_depth_max': 1, 
 'response_received_count': 12, 
 'scheduler/dequeued': 13, 
 'scheduler/dequeued/memory': 13, 
 'scheduler/enqueued': 13, 
 'scheduler/enqueued/memory': 13, 
 'start_time': datetime.datetime(2016, 5, 18, 2, 35, 36, 733155)} 
2016-05-18 10:35:47 [scrapy] INFO: Spider closed (finished) 

但不抓取任何数据...

[mayuping@i~/PycharmProjects/car] $more car.json 

car.json 中的项目输出为零。

谢谢

请您参考如下方法:

问题解决了。 parse_item 在 car_spider.py 中没有缩进。缩进后:

# -*- coding=utf-8 -*- 
from __future__ import absolute_import 
from scrapy.linkextractors import LinkExtractor 
from scrapy.spiders import CrawlSpider,Rule,Spider 
from car.items import Car58Item 
 
 
class CarSpider (CrawlSpider): 
    name ='car' 
    allowed_domains = ['58.com'] 
    start_urls = ['http://quanguo.58.com/ershouche'] 
    rules = [Rule(LinkExtractor(allow=('/pn\d+')),'parse_item')] #//页面读取策略 
 
    def parse_item(self,response): 
        trs = response.xpath("//div[@id='infolist']/table[@class='tbimg']/tr")[1:-2] 
        items = [] 
        #for tr in sel.xpath("id('infolist')/table/tr"): 
        for tr in trs: 
            item = Car58Item() 
            item['url'] = tr.xpath("td[@class='img']/a/@href").extract() 
            item['tip'] = tr.xpath("td[@class='t']/a/font/text()").extract() 
            item['name'] = tr.xpath("td[@class='t']/a[1]/text()").extract() 
       #     item['description'] = trs.xpath("/td[@class='t']/a/text()").extract() 
            item['size'] = tr.xpath("td[@class='t']/p").extract() 
            item['region'] = tr.xpath("td[@class='tc']/a/text()").extract() 
            item['amt'] = tr.xpath("td[@class='tc']/b/text()").extract() 
            items.append(item) 
        return items 

scrapy 内置的 JSON 导出是可以的。