Skip to main content
 首页 » 编程设计

python之是否可以使用 Scrapy 通过代理访问 https 页面

2025年05月04日14davidwang456

我在Scrapy中通过代理可以成功访问http页面,但是无法访问https站点。我已经研究了这个话题,但我仍然不清楚。是否可以使用 Scrapy 通过代理访问 https 页面?我需要修补什么吗?或者添加一些自定义代码?如果可以确认这是一个标准功能,我可以跟进更多细节。希望这是一个简单的过程。

已编辑:

这是我添加到设置文件中的内容:

DOWNLOADER_MIDDLEWARES = {'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110, 'test_website.middlewares.ProxyMiddleware': 100} 
PROXIES = [{'ip_port': 'us-il.proxymesh.com:31280', 'user_pass': 'username:password'}] 

这是我的蜘蛛的代码:

import scrapy 
 
class TestSpider(scrapy.Spider): 
    name = "test_spider" 
    allowed_domains = "ipify.org" 
     start_urls = ["https://api.ipify.org"] 
 
    def parse(self, response): 
        with open('test.html', 'wb') as f: 
            f.write(response.body) 

中间件文件如下:

import base64 
import random 
from settings import PROXIES 
 
class ProxyMiddleware(object): 
    def process_request(self, request, spider): 
        proxy = random.choice(PROXIES) 
        if proxy['user_pass'] is not None: 
            request.meta['proxy'] = "http://%s" % proxy['ip_port'] 
            encoded_user_pass = base64.encodestring(proxy['user_pass']) 
            request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass             
        else: 
            request.meta['proxy'] = "http://%s" % proxy['ip_port'] 

这是日志文件:

2015-08-12 20:15:50 [scrapy] INFO: Scrapy 1.0.3 started (bot: test_website) 
2015-08-12 20:15:50 [scrapy] INFO: Optional features available: ssl, http11 
2015-08-12 20:15:50 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'test_website.spiders', 'SPIDER_MODULES': ['test_website.spiders'], 'LOG_STDOUT': True, 'LOG_FILE': 'log.txt', 'BOT_NAME': 'test_website'} 
2015-08-12 20:15:51 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState 
2015-08-12 20:15:53 [scrapy] INFO: Enabled downloader middlewares: ProxyMiddleware, HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2015-08-12 20:15:53 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2015-08-12 20:15:53 [scrapy] INFO: Enabled item pipelines:  
2015-08-12 20:15:53 [scrapy] INFO: Spider opened 
2015-08-12 20:15:53 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2015-08-12 20:15:53 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 
2015-08-12 20:15:53 [scrapy] DEBUG: Retrying <GET https://api.ipify.org> (failed 1 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>] 
2015-08-12 20:15:53 [scrapy] DEBUG: Retrying <GET https://api.ipify.org> (failed 2 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>] 
2015-08-12 20:15:53 [scrapy] DEBUG: Gave up retrying <GET https://api.ipify.org> (failed 3 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>] 
2015-08-12 20:15:53 [scrapy] ERROR: Error downloading <GET https://api.ipify.org>: [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>] 
2015-08-12 20:15:53 [scrapy] INFO: Closing spider (finished) 
2015-08-12 20:15:53 [scrapy] INFO: Dumping Scrapy stats: 
{'downloader/exception_count': 3, 
 'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 3, 
 'downloader/request_bytes': 819, 
 'downloader/request_count': 3, 
 'downloader/request_method_count/GET': 3, 
 'finish_reason': 'finished', 
 'finish_time': datetime.datetime(2015, 8, 13, 2, 15, 53, 943000), 
 'log_count/DEBUG': 4, 
 'log_count/ERROR': 1, 
 'log_count/INFO': 7, 
 'scheduler/dequeued': 3, 
 'scheduler/dequeued/memory': 3, 
 'scheduler/enqueued': 3, 
 'scheduler/enqueued/memory': 3, 
 'start_time': datetime.datetime(2015, 8, 13, 2, 15, 53, 38000)} 
2015-08-12 20:15:53 [scrapy] INFO: Spider closed (finished) 

如果删除“https”中的“s”或禁用代理,我的爬虫会成功运行。我可以通过我的浏览器通过代理访问https页面。

请您参考如下方法:

我得到这个是因为在代理中间件中使用 base64.encodestring 而不是 base64.b64encode 。尝试更改它。