Skip to main content
 首页 » 编程设计

python之在 Scrapy 中获取 http.response 对象的最简单方法

2025年05月04日30lvdongjie

我是 Scrapy 的新手，我正在尝试将“获取网页内容”放入响应对象(如果我理解正确的话)。

我正在关注 http://doc.scrapy.org/en/latest/topics/selectors.html , 然而，它在 scrapy shell 中工作。我想让它直接在 python 代码中工作。

我写代码来抓取http://doc.scrapy.org/en/latest/_static/selectors-sample1.html 

import scrapy 
from scrapy.http import HtmlResponse 
URL = 'http://doc.scrapy.org/en/latest/_static/selectors-sample1.html' 
response = HtmlResponse(url=URL)     
print response.selector.xpath('//title/text()')

输出是

>> []

为什么我无法获得正确的标题值？ HtmlResponse() 似乎没有从网络上下载数据……为什么？我该如何解决!

非常感谢你!

帽子

请您参考如下方法:

你的声明

response = HtmlResponse(url=URL)

只构建a "local scope" HtmlResponse object ， body 空虚。它不下载任何东西，尤其是 http://doc.scrapy.org/en/latest/_static/selectors-sample1.html 处的资源。 .

在 Scrapy 中，你通常不会自己构建 HtmlResponse 对象，你让 Scrapy 框架为你构建它们，当它完成处理你给它的 Request 实例时，例如请求(url='http://doc.scrapy.org/en/latest/_static/selectors-sample1.html')

如果你正在尝试 Scrapy，我建议你使用 scrapy shell : 在交互式 shell 中，您可以使用 fetch('http://someurl') 触发下载(并获得要使用的“真实”Response 对象):

$ scrapy shell 
2016-06-14 10:59:31 [scrapy] INFO: Scrapy 1.1.0 started (bot: scrapybot) 
(...) 
[s] Available Scrapy objects: 
[s]   crawler    <scrapy.crawler.Crawler object at 0x7f1a6591d588> 
[s]   item       {} 
[s]   settings   <scrapy.settings.Settings object at 0x7f1a6ce290f0> 
[s] Useful shortcuts: 
[s]   shelp()           Shell help (print this help) 
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects 
[s]   view(response)    View response in a browser 
>>> fetch('http://doc.scrapy.org/en/latest/_static/selectors-sample1.html') 
2016-06-14 10:59:51 [scrapy] INFO: Spider opened 
2016-06-14 10:59:51 [scrapy] DEBUG: Crawled (200) <GET http://doc.scrapy.org/en/latest/_static/selectors-sample1.html> (referer: None) 
>>> response.xpath('//title/text()').extract() 
['Example website']

在 shell 之外，要真正下载数据，您需要:

  • 子类scrapy.Spider,
  • 定义开始下载的 URL，
  • 并编写回调方法来处理下载的数据，包裹在传递给它们的 Response 对象中

一个非常简单的示例(在一个名为 test.py 的文件中):

import scrapy 
 
 
class TestSpider(scrapy.Spider): 
 
    name = 'testspider' 
 
    # start_urls is special and internally it builds Request objects for each of the URLs listed 
    start_urls = ['http://doc.scrapy.org/en/latest/_static/selectors-sample1.html'] 
 
    def parse(self, response): 
        yield { 
            'title': response.xpath('//h1/text()').extract_first() 
        }

然后你需要运行蜘蛛。 Scrapy 有一个运行单文件爬虫的命令:

$ scrapy runspider test.py

然后您在控制台中得到了这个:

2016-06-14 10:48:05 [scrapy] INFO: Scrapy 1.1.0 started (bot: scrapybot) 
2016-06-14 10:48:05 [scrapy] INFO: Overridden settings: {} 
2016-06-14 10:48:06 [scrapy] INFO: Enabled extensions: 
['scrapy.extensions.logstats.LogStats', 'scrapy.extensions.corestats.CoreStats'] 
2016-06-14 10:48:06 [scrapy] INFO: Enabled downloader middlewares: 
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 
 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware', 
 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 
2016-06-14 10:48:06 [scrapy] INFO: Enabled spider middlewares: 
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 
 'scrapy.spidermiddlewares.referer.RefererMiddleware', 
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 
 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 
2016-06-14 10:48:06 [scrapy] INFO: Enabled item pipelines: 
[] 
2016-06-14 10:48:06 [scrapy] INFO: Spider opened 
2016-06-14 10:48:06 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2016-06-14 10:48:06 [scrapy] DEBUG: Crawled (200) <GET http://doc.scrapy.org/en/latest/_static/selectors-sample1.html> (referer: None) 
2016-06-14 10:48:06 [scrapy] DEBUG: Scraped from <200 http://doc.scrapy.org/en/latest/_static/selectors-sample1.html> 
{'title': 'Example website'} 
2016-06-14 10:48:06 [scrapy] INFO: Closing spider (finished) 
2016-06-14 10:48:06 [scrapy] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 252, 
 'downloader/request_count': 1, 
 'downloader/request_method_count/GET': 1, 
 'downloader/response_bytes': 501, 
 'downloader/response_count': 1, 
 'downloader/response_status_count/200': 1, 
 'finish_reason': 'finished', 
 'finish_time': datetime.datetime(2016, 6, 14, 8, 48, 6, 564591), 
 'item_scraped_count': 1, 
 'log_count/DEBUG': 2, 
 'log_count/INFO': 7, 
 'response_received_count': 1, 
 'scheduler/dequeued': 1, 
 'scheduler/dequeued/memory': 1, 
 'scheduler/enqueued': 1, 
 'scheduler/enqueued/memory': 1, 
 'start_time': datetime.datetime(2016, 6, 14, 8, 48, 6, 85693)} 
2016-06-14 10:48:06 [scrapy] INFO: Spider closed (finished)

如果你真的想玩选择器，而不实际下载任何网络数据，假设你已经在本地拥有数据(例如从浏览器中的 view-source: 复制)，你可以这样做那但是你需要提供 body :

>>> response = HtmlResponse(url=URL, body=''' 
... <!DOCTYPE html> 
... <html> 
...   <head> 
...   </head> 
...   <body> 
...       <h1>Herman Melville - Moby-Dick</h1> 
...  
...       <div> 
...         <p> 
...           Availing himself of the mild, summer-cool weather that now reigned in these latitudes, ... them a care-killing competency. 
...         </p> 
...       </div> 
...   </body> 
... </html>''', encoding='utf8') 
>>> response.xpath('//h1') 
[<Selector xpath='//h1' data='<h1>Herman Melville - Moby-Dick</h1>'>] 
>>> response.xpath('//h1').extract() 
['<h1>Herman Melville - Moby-Dick</h1>'] 
>>>


  文章来源：
  https://www.windname.com/post/adhiea2b1.html
阅读延展