Skip to main content
 首页 » 编程设计

python之scrapy 无法从网站中提取一些数据

2025年05月04日14dudu

我正在使用 scrapy 抓取页面,我能够获取所有可见文本的简单内容。然而,有些文本对于爬虫来说是不可见的,最终显示为空格。

例如,查看页面源允许我查看这些字段:

https://www.dropbox.com/s/f056mffmuah6uu4/Screenshot%202015-07-23%2018.23.32.png?dl=0

我已经多次尝试通过 xpath 和 css 访问此字段,但每次尝试后都无法获取这些字段。

当我尝试这样的事情时:

response.xpath('//text()').extract() 

我根本无法在文本转储中看到这些字段。

有人知道为什么这些字段对 scrapy 不可见吗?网址是:https://www.buzzbuzzhome.com/uc/units/houses/sapphire

请您参考如下方法:

在您的爬虫中,您需要向 https://www.buzzbuzzhome.com/bbhAjax/Development/UnitPriceHistory 端点发出一个额外的 XHR POST 请求以获取价格历史提供必要的标题和 POST 参数:

import json 
import scrapy 
 
 
class BuzzSpider(scrapy.Spider): 
    name = 'buzzbuzzhome' 
    allowed_domains = ['buzzbuzzhome.com'] 
    start_urls = ['https://www.buzzbuzzhome.com/uc/units/houses/sapphire'] 
 
    def parse(self, response): 
        unit_id = response.xpath("//div[@id = 'unitDetails']/@data-unit-id").extract()[0] 
        development_url = "uc" 
        new_relic_id = response.xpath("//script[contains(., 'xpid')]").re(r'xpid:"(.*?)"') 
 
        params = {"developmentUrl": development_url, "unitID": unit_id} 
        yield scrapy.Request("https://www.buzzbuzzhome.com/bbhAjax/Development/UnitPriceHistory", 
                             method="POST", 
                             body=json.dumps(params), 
                             callback=self.parse_history, 
                             headers={ 
                                 "Accept": "*/*", 
                                 "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36", 
                                 "X-Requested-With": "XMLHttpRequest", 
                                 "X-NewRelic-ID": new_relic_id, 
                                 "Origin": "https://www.buzzbuzzhome.com", 
                                 "Host": "www.buzzbuzzhome.com", 
                                 'Content-Type': 'application/json; charset=UTF-8' 
                             }) 
 
    def parse_history(self, response): 
        for row in response.css("div.row"): 
            title = row.xpath(".//div[@class='content-title']/text()").extract()[0].strip() 
            text = row.xpath(".//div[@class='content-text']/text()").extract()[0].strip() 
 
            print title, text 

打印:

05/25/2015 Unit listed as Sold 
12/18/2014 Unit listed as For Sale 
11/24/2014 Unit price increased  by 1.54% to $461,990 
11/04/2014 Unit price increased  by 6.81% to $454,990 
10/02/2014 Unit price increased  by 4.67% to $425,990 
01/22/2014 Unit price increased  by 2.52% to $406,990 
12/06/2013 Unit listed as For Sale at $396,990