Skip to main content
 首页 » 编程设计

Python之无法在 Scrapy 中动态旋转 userAgent

2025年05月04日12webabcd

我覆盖了 scrapy 模块 HttpProxyMiddleware 和 UserAgentMiddleware 的默认实现,我自己的 scrapy 实现轮换用户代理和 IP 地址,它从提供的 list 。 IP 会根据每个请求而变化,但用户代理不会。我无法弄清楚原因。

这是我的类实现

旋转用户代理中间件

    class RotateUserAgentMiddleware(UserAgentMiddleware): 
    def __init__(self, user_agent=''): 
        self.user_agent = user_agent 
 
    def process_request(self, request, spider): 
        ua = random.choice(self.user_agent_list) 
        if ua: 
            request.headers.setdefault('User-Agent', ua) 
            # Add desired logging message here. 
            spider.log( 
                u'User-Agent: {} {}'.format(request.headers.get('User-Agent'), request) 
                      ) 

代理中间件

class ProxyMiddleware(HttpProxyMiddleware): 
    def __init__(self, proxy_ip=''): 
        self.proxy_ip = proxy_ip 
 
    def process_request(self,request,spider): 
        ip = random.choice(self.proxy_list) 
        if ip: 
 
            request.meta['proxy'] = ip 
            print(request.meta) 
        return request 

settings.py 中 Downloader_Middleware 的更改是;

DOWNLOADER_MIDDLEWARES = { 
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None, 
    'IpRotation.ProxyMiddleware.ProxyMiddleware': 800, 
    'scrapy.downloadermiddleware.useragent.UserAgentMiddleware' : None, 
    'IpRotation.RotateUserAgentMiddleware.RotateUserAgentMiddleware':790 
} 

在我的控制台上为每个请求打印Ip 和user-agent 值:

    2015-10-09 15:51:46 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '198.*.*.*:80'} 
2015-10-09 15:51:46 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '195.*.*.*:3120'} 
2015-10-09 15:51:46 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '94.*.*.*:3128'} 
2015-10-09 15:51:47 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '94.*.*.*:3128'} 
2015-10-09 15:51:47 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '198.*.*.*:80'} 
2015-10-09 15:51:47 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '94.*.*.*:3128'} 
2015-10-09 15:51:47 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '94.*.*.*:3128'} 
2015-10-09 15:51:47 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '195.*.*.*:3120'} 
2015-10-09 15:51:47 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '200.*.*.*:80'} 
2015-10-09 15:51:47 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '198.*.*.*:80'} 
2015-10-09 15:51:47 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '213.*.*.*:80'} 
2015-10-09 15:51:47 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '198.*.*.*:80'} 
2015-10-09 15:51:47 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '200.*.*.*:80'} 
2015-10-09 15:51:48 [dmoz] DEBUG: User-Agent: Scrapy/1.0.3 (+http://scrapy.org) <GET http://www.imdb.com/chart/top> 
{'download_timeout': 180.0, 'proxy': '58.*.*.*:80'} 

没有更改 settings.py 中的 USER_AGENT,因为我必须随机分配值:

# Crawl responsibly by identifying yourself (and your website) on the user-agent 
#USER_AGENT = 'IPProxy (+http://www.yourdomain.com)' 

整个项目中,我不太清楚的地方就是给Downloader_Middleware赋值。没有人说 scrapy 忽略类,但 Integers 说什么?请有人从这里帮助我。

请您参考如下方法:

将Downloader_Middleware中'IpRotation.RotateUserAgent Middleware.RotateUserAgent Middleware'的值修改为小于400。