Skip to main content
 首页 » 编程设计

python之为什么 selenium 在执行这段代码之前要等待很长时间

2025年05月04日12sharpest

我正在尝试在此页面上进行无限滚动,这是我的代码:

from selenium import webdriver 
import time 
 
profile = webdriver.FirefoxProfile() 
profile.set_preference("general.useragent.override","Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0") 
driver = webdriver.Firefox(profile) 
 
driver.get("http://www.quora.com/Programming-Languages/followers") 
for n in range(0,5): # For testing I have capped this at 5, will handle this properly once things start to work. 
    driver.execute_script("window.scrollTo(0,1000000);") 
    time.sleep(2) 

所以当我运行它时,它会等待很多秒(有时超过 1 分钟),然后再进行任何滚动,然后在下一次滚动之前再次等待相同的时间。 该代码似乎在其他页面上运行良好。 关于如何解决此问题的任何想法?

当我尝试使用 Chrome 而不是 firefox 时,出现以下错误: driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver') 添加到 .py 文件。

Traceback (most recent call last): 
  File "ok.py", line 8, in <module> 
    driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver') 
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 65, in __init__ 
    keep_alive=True) 
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 73, in __init__ 
    self.start_session(desired_capabilities, browser_profile) 
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 121, in start_session 
    'desiredCapabilities': desired_capabilities, 
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute 
    response = self.command_executor.execute(driver_command, params) 
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute 
    return self._request(command_info[0], url, body=data) 
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 379, in _request 
    self._conn.request(method, parsed_url.path, body, headers) 
  File "/usr/lib/python2.7/httplib.py", line 973, in request 
    self._send_request(method, url, body, headers) 
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request 
    self.endheaders(body) 
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders 
    self._send_output(message_body) 
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output 
    self.send(msg) 
  File "/usr/lib/python2.7/httplib.py", line 791, in send 
    self.connect() 
  File "/usr/lib/python2.7/httplib.py", line 772, in connect 
    self.timeout, self.source_address) 
  File "/usr/lib/python2.7/socket.py", line 553, in create_connection 
    for res in getaddrinfo(host, port, 0, SOCK_STREAM): 
socket.gaierror: [Errno -2] Name or service not known 

请您参考如下方法:

切换到 Chrome()帮助我解决了问题:

import time 
from selenium import webdriver 
 
followers_per_page = 18 
 
driver = webdriver.Chrome() 
driver.get("http://www.quora.com/Programming-Languages/followers") 
 
# get the followers count 
element = driver.find_element_by_class_name('count') 
followers_count = int(element.text.replace('k', '000').replace('.', '')) 
print followers_count 
 
# scroll down the page iteratively with a delay 
for _ in xrange(0, followers_count/followers_per_page + 1): 
    driver.execute_script("window.scrollTo(0, 0,1000000);") 
    time.sleep(2) 

仅供引用,我使用了一种有点不同的方法:解析关注者的数量并计算每页的关注者,同时考虑到它一次加载 18 个关注者的事实。

我之前实际上处理过类似的 quora 问题,请参阅:


嗯,这不是我首先想到的。这是故事。

问题是有对 http://tch840195.tch.quora.com/up/chan5-8886/updates 的未决请求需要几分钟才能完成的 URL。这就是让 selenium 认为页面没有完全加载的原因。而且,情况越来越糟——这是每 X 秒发生一次的周期性事件。将其视为长池化。

我尝试了多种方法来解决使用 Firefox webdriver 的问题:

  • 设置webdriver.load.strategy偏好 unstable
  • 设置network.http.response.timeout、network.http.connection-timeout和network.http.keep-alive.timeout和 network.http.request.max-start-delay偏好
  • 设置页面加载超时:

    driver.set_page_load_timeout(3) 
    
  • 设置脚本超时:

    driver.set_script_timeout(3) 
    
  • 调用 window.stop(); 希望它能停止事件请求:

    driver.execute_script('window.stop();') 
    
  • 更新到最新的 Firefox 和 selenium 包版本

另一种可行的方法是以某种方式阻止对 "slow url" 的请求。使用代理服务器并将 firefox 指向它,或者,如果可能的话,让 Firefox 知道将 URL 列入黑名单(可能通过扩展)。

另请参阅内部包含多个解决方法的相关问题:

另见: