Skip to main content
 首页 » 编程设计

python之Scrapy管道html解析

2025年05月04日7Terrylee

我有一个包含 3 个项目的蜘蛛:url、标题和类别。

它们在原始 html 中加载正常,但现在我想在管道中使用 html2test 将标题和类别转换为纯文本..

这是我错误的流水线代码,谁能帮忙调试一下。

谢谢

import html2text 
import csv 
from tutorial import settings 
 
def write_to_csv(item): 
    writer = csv.writer(open(settings.csv_file_path, 'a'), lineterminator='\n') 
    writer.writerow([item[key] for key in item.keys()]) 
 
 
class TutorialPipeline(object): 
    def process_item(self, item, spider): 
        h = html2text.HTML2Text() 
        h.ignore_images = True 
        h.handle(item['title']).strip() 
        h.handle(item['category']).strip() 
        write_to_csv(item) 
        return item 

蜘蛛代码

import scrapy 
from scrapy.http import Request 
from scrapy.contrib.spiders import CrawlSpider,Rule 
from scrapy.contrib.linkextractors import LinkExtractor 
from tutorial.items import TutorialItem 
 
class tuto(CrawlSpider): 
    name = "tuto" 
    allowed_domains = ['emedicine.medscape.com'] 
    start_urls=["http://emedicine.medscape.com"] 
    rules=( 
        Rule( LinkExtractor(restrict_xpaths ='//div[@id="browsespecialties"]'),callback='follow_pages', follow=True), 
    ) 
    def follow_pages(self, response): 
        for sel in response.xpath('//div[@class="maincolbox"]//a/@href').extract(): 
            yield Request("http://emedicine.medscape.com/" + sel, callback = self.parse_item) 
 
    def parse_item(self, response): 
        item = TutorialItem() 
        item['url'] = response.url 
        item['background'] = response.xpath('//div[@class="refsection_content"]').extract() 
        item['title'] = response.xpath('//h1').extract() 
        yield item 

请您参考如下方法:

您不需要在管道中解析 HTML。

提取蜘蛛中元素的text()

替换:

item['background'] = response.xpath('//div[@class="refsection_content"]').extract() 
item['title'] = response.xpath('//h1').extract() 

与:

item['background'] = response.xpath('//div[@class="refsection_content"]/text()').extract() 
item['title'] = response.xpath('//h1/text()').extract()