Skip to main content
 首页 » 编程设计

python之将 cookie 附加到请求

2025年04月02日22sharpest

我正在尝试从 Castorama 网站获取工具的价格。但到目前为止,我在构建正确请求时遇到问题。

http://www.castorama.pl/produkty/narzedzia-i-artykuly/elektronarzedzia-przenosne-i-akcesoria/szlifierki-i-polerki/szlifierki-oscylacyjne/szlifierka-oscylacyjna-pp-110w.html 

不幸的是,这并不容易。价格取决于商店的本地化。在获得价格之前,您需要定义您的商店本地化。在网站上,我单击 'ZOBACZ CENĘ'(右侧的黄色框)。稍后我在中间字段 f.e. 中填写我的邮政编码。 '05-123' 然后点击右边的 'SZUKAJ PO KODZIE' 按钮。最后,我在弹出框中单击 'USTAW' 的黄色弹出按钮。

多亏了它,我得到了想要的产品价格。我想用 scrappy 复制这种行为。为此,我在 Chrome 开发人员工具中检查了网络选项卡和 XHR 选项卡,以确定负责获取价格的请求。我认为正确的方法是 'getProductPriceStockByStore/'。

请求

URL:http://www.castorama.pl/bold_all/data/getProductPriceStockByStore/ 
Request Method:POST 
Status Code:200 OK 
Remote Address:109.205.50.98:80 

请求 header

Accept:text/javascript, text/html, application/xml, text/xml, */* 
Accept-Encoding:gzip, deflate 
Accept-Language:en-GB,en;q=0.8,pl;q=0.6 
Connection:keep-alive 
Content-Length:39 
Content-type:application/x-www-form-urlencoded; charset=UTF-8 
Cookie:selected_shop_flag=3; CACHED_FRONT_FORM_KEY=2MxQx5N1GeBOoDFl; localizationPopup=1; selected_shop=1; selected_shop_store_view=8002; bold_wishlist=3lg7qtm3teba7s1sbfg77hi352; frontend=3lg7qtm3teba7s1sbfg77hi352; VIEWED_PRODUCT_IDS=30052; cSID_VM=1460629378710; _ga=GA1.2.91284606.1460626559; _ceg.s=o5mcub; _ceg.u=o5mcub; _dc_gtm_UA-27193958-1=1 
Host:www.castorama.pl 
Origin:http://www.castorama.pl 
Referer:http://www.castorama.pl/produkty/narzedzia-i-artykuly/elektronarzedzia-przenosne-i-akcesoria/szlifierki-i-polerki/szlifierki-oscylacyjne/szlifierka-oscylacyjna-pp-110w.html 
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/49.0.2623.108 Chrome/49.0.2623.108 Safari/537.36 

表单数据:

isAjax:true 
product_id:30052 
store:8002 

响应:

{"products":{"30052":{"price":"93.98","qty":"7.00","stock_status":1,"html":"in"}},"store":"8002","templates":{"in":"<span><span class=\"in-stock\">Dost\u0119pny<\/span><\/span>","out":"<span><span class=\"out-of-stock\">Niedost\u0119pny<\/span><\/span>","phone":"<span><span class=\"low-stock\">Na zam\u00f3wienie<\/span><\/span>","backorder":"<span><span class=\"backorder-stock\">Na zam\u00f3wienie<\/span><\/span>"},"status":true} 

所以我转向 Scrapy 来实现该问题的解决方案。我决定创建带有 cookie 的 post 请求,请求的 header 与上面类似:

import scrapy 
from Castorama.items import CastoramaItem 
 
class DmozSpider(scrapy.Spider): 
    name = "Castorama" 
    allowed_domains = ["castorama.pl"] 
    start_urls = ["http://www.castorama.pl/bold_all/data/getProductPriceStockByStore/"] 
 
    def start_Request(self): 
 
        req=scrapy.Request(start_urls[0] 
            , method='POST' 
            , cookies ={'selected_shop_flag':3, 
                'CACHED_FRONT_FORM_KEY':'2MxQx5N1GeBOoDFl', 
                'selected_shop':1, 
                'selected_shop_flag':3, 
                'selected_shop_store_view':8002, 
                'VIEWED_PRODUCT_IDS':30052, 
                'frontend':'3lg7qtm3teba7s1sbfg77hi352', 
                'cSID_VM':1460626558358} 
            ,callback='Rozkoduj' 
            ) 
        yield req 
 
    def Rozkoduj(self, response): 
        print response.body 

但是我对这段代码很不成功。我的控制台日志:

2016-04-14 12:54:09 [scrapy] INFO: Scrapy 1.0.5 started (bot: Castorama) 
2016-04-14 12:54:09 [scrapy] INFO: Optional features available: ssl, http11, boto 
2016-04-14 12:54:09 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'Castorama.spiders', 'SPIDER_MODULES': ['Castorama.spiders'], 'BOT_NAME': 'Castorama'} 
2016-04-14 12:54:09 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState 
2016-04-14 12:54:09 [boto] DEBUG: Retrieving credentials from metadata server. 
2016-04-14 12:54:10 [boto] ERROR: Caught exception reading instance data 
Traceback (most recent call last): 
  File "/home/michal/anaconda2/lib/python2.7/site-packages/boto/utils.py", line 210, in retry_url 
    r = opener.open(req, timeout=timeout) 
  File "/home/michal/anaconda2/lib/python2.7/urllib2.py", line 431, in open 
    response = self._open(req, data) 
  File "/home/michal/anaconda2/lib/python2.7/urllib2.py", line 449, in _open 
    '_open', req) 
  File "/home/michal/anaconda2/lib/python2.7/urllib2.py", line 409, in _call_chain 
    result = func(*args) 
  File "/home/michal/anaconda2/lib/python2.7/urllib2.py", line 1227, in http_open 
    return self.do_open(httplib.HTTPConnection, req) 
  File "/home/michal/anaconda2/lib/python2.7/urllib2.py", line 1197, in do_open 
    raise URLError(err) 
URLError: <urlopen error timed out> 
2016-04-14 12:54:10 [boto] ERROR: Unable to read instance data, giving up 
2016-04-14 12:54:10 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2016-04-14 12:54:10 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2016-04-14 12:54:10 [scrapy] INFO: Enabled item pipelines:  
2016-04-14 12:54:10 [scrapy] INFO: Spider opened 
2016-04-14 12:54:10 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2016-04-14 12:54:10 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 
2016-04-14 12:54:10 [scrapy] DEBUG: Crawled (200) <GET http://www.castorama.pl/bold_all/data/getProductPriceStockByStore/> (referer: None) 
2016-04-14 12:54:10 [scrapy] ERROR: Spider error processing <GET http://www.castorama.pl/bold_all/data/getProductPriceStockByStore/> (referer: None) 
Traceback (most recent call last): 
  File "/home/michal/anaconda2/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks 
    current.result = callback(current.result, *args, **kw) 
  File "/home/michal/anaconda2/lib/python2.7/site-packages/scrapy/spiders/__init__.py", line 76, in parse 
    raise NotImplementedError 
NotImplementedError 
2016-04-14 12:54:10 [scrapy] INFO: Closing spider (finished) 
2016-04-14 12:54:10 [scrapy] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 256, 
 'downloader/request_count': 1, 
 'downloader/request_method_count/GET': 1, 
 'downloader/response_bytes': 311, 
 'downloader/response_count': 1, 
 'downloader/response_status_count/200': 1, 
 'finish_reason': 'finished', 
 'finish_time': datetime.datetime(2016, 4, 14, 10, 54, 10, 776463), 
 'log_count/DEBUG': 3, 
 'log_count/ERROR': 3, 
 'log_count/INFO': 7, 
 'response_received_count': 1, 
 'scheduler/dequeued': 1, 
 'scheduler/dequeued/memory': 1, 
 'scheduler/enqueued': 1, 
 'scheduler/enqueued/memory': 1, 
 'spider_exceptions/NotImplementedError': 1, 
 'start_time': datetime.datetime(2016, 4, 14, 10, 54, 10, 477689)} 
2016-04-14 12:54:10 [scrapy] INFO: Spider closed (finished) 

这是我的最后一个问题。我的做法合适吗?我是否应该像上面的代码那样尝试将此 cookie 附加到请求中?或者我应该尝试以完全不同的方式来做。最后,如果我朝着正确的方向前进,我应该在我的代码中更改什么以创建正确的请求?

提前感谢您的帮助。


并在 Pawel Miech 更正后更新了蜘蛛版本。这更好,因为请求有效,但我仍然没有得到正确的响应。

import scrapy 
from Castorama.items import CastoramaItem 
 
class DmozSpider(scrapy.Spider): 
    name = "Castorama" 
    allowed_domains = ["castorama.pl"] 
    start_urls=['http://www.castorama.pl'] 
 
    def parse(self, response): 
        start_urls = ["http://www.castorama.pl/bold_all/data/getProductPriceStockByStore/"] 
        req=scrapy.Request(start_urls[0] 
            , method='POST' 
            , cookies ={'selected_shop_flag':3, 
                'CACHED_FRONT_FORM_KEY':'2MxQx5N1GeBOoDFl', 
                'selected_shop':1, 
                'selected_shop_flag':3, 
                'selected_shop_store_view':8002, 
                'VIEWED_PRODUCT_IDS':30052, 
                'frontend':'3lg7qtm3teba7s1sbfg77hi352', 
                'cSID_VM':1460626558358} 
            ,callback=self.rozkoduj 
            ) 
        yield req 
 
    def rozkoduj(self, response): 
        print '>>>>>>>>>' 
        print response.body 

请您参考如下方法:

Scrapy 请求是异步的。每个请求都必须有回调。如果没有回调,它被设置为 spider.parse 方法。如果没有 spider.parse 方法,您会得到 NotImplementedError,您会在此堆栈跟踪中看到:

   Traceback (most recent call last): 
      File "/home/michal/anaconda2/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks 
        current.result = callback(current.result, *args, **kw) 
      File "/home/michal/anaconda2/lib/python2.7/site-packages/scrapy/spiders/__init__.py", line 76, in parse 
        raise NotImplementedError 
    NotImplementedError 

因此,首先要向您的 POST 添加适当的回调(它必须引用蜘蛛方法而不是字符串,例如 self.rozkoduj 而不是“Rozkoduj”)。

Urlopen 错误是从 boto 发送的,如果你没有配置 s3 就会发出,它很难看但可能会被忽略,直到有人在 Scrapy 核心中修复这个票证。

And here is my final questions. Is my approach proper? Should I try to attach this cookie to the request as in above code?

这个问题的答案和往常一样:视情况而定。如果您只关心为一个请求发送 cookie,那么您的方法是正确的。 但是 如果您希望在从蜘蛛发送的所有 请求中包含这些cookie,包括从回调发出到您的POST 的请求,那么您必须将cookie 添加到cookiejar。不幸的是,在 cookiejar 中设置 cookie 不是很容易,这里有使这更简单的票:https://github.com/scrapy/scrapy/issues/1878

简而言之,将 cookie 设置为 cookiejar 就是按照以下方式做一些事情(这只是伪代码和指针):

# must be cookielib.Cookie object 
# must pass all kwargs for cookielib.Cookie 
cookie = Cookie(**kwargs) 
# get cookiejar object it is stored in cookie middleware 
all_mw = spider.crawler.engine.downloader.middleware.middlewares  
# find cookie middleware there 
cookiejar = cookie_middleware._cookiejars[None] 
cookiejar.set_cookie(cookie)