我正在尝试在此页面上进行无限滚动,这是我的代码:
from selenium import webdriver
import time
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override","Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0")
driver = webdriver.Firefox(profile)
driver.get("http://www.quora.com/Programming-Languages/followers")
for n in range(0,5): # For testing I have capped this at 5, will handle this properly once things start to work.
driver.execute_script("window.scrollTo(0,1000000);")
time.sleep(2)
所以当我运行它时,它会等待很多秒(有时超过 1 分钟),然后再进行任何滚动,然后在下一次滚动之前再次等待相同的时间。 该代码似乎在其他页面上运行良好。 关于如何解决此问题的任何想法?
当我尝试使用 Chrome 而不是 firefox 时,出现以下错误: driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver') 添加到 .py 文件。
Traceback (most recent call last):
File "ok.py", line 8, in <module>
driver = webdriver.Chrome('/home/asdf/apps/chromedrive/chromedriver')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 65, in __init__
keep_alive=True)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 73, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 121, in start_session
'desiredCapabilities': desired_capabilities,
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute
return self._request(command_info[0], url, body=data)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 379, in _request
self._conn.request(method, parsed_url.path, body, headers)
File "/usr/lib/python2.7/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 791, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 772, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known
请您参考如下方法:
切换到 Chrome()帮助我解决了问题:
import time
from selenium import webdriver
followers_per_page = 18
driver = webdriver.Chrome()
driver.get("http://www.quora.com/Programming-Languages/followers")
# get the followers count
element = driver.find_element_by_class_name('count')
followers_count = int(element.text.replace('k', '000').replace('.', ''))
print followers_count
# scroll down the page iteratively with a delay
for _ in xrange(0, followers_count/followers_per_page + 1):
driver.execute_script("window.scrollTo(0, 0,1000000);")
time.sleep(2)
仅供引用,我使用了一种有点不同的方法:解析关注者的数量并计算每页的关注者,同时考虑到它一次加载 18 个关注者的事实。
我之前实际上处理过类似的 quora 问题,请参阅:
嗯,这不是我首先想到的。这是故事。
问题是有对 http://tch840195.tch.quora.com/up/chan5-8886/updates 的未决请求需要几分钟才能完成的 URL。这就是让 selenium 认为页面没有完全加载的原因。而且,情况越来越糟——这是每 X 秒发生一次的周期性事件。将其视为长池化。
我尝试了多种方法来解决使用 Firefox webdriver 的问题:
- 设置
webdriver.load.strategy偏好unstable - 设置
network.http.response.timeout、network.http.connection-timeout和network.http.keep-alive.timeout和network.http.request.max-start-delay偏好 设置页面加载超时:
driver.set_page_load_timeout(3)设置脚本超时:
driver.set_script_timeout(3)调用
window.stop();希望它能停止事件请求:driver.execute_script('window.stop();')更新到最新的 Firefox 和 selenium 包版本
另一种可行的方法是以某种方式阻止对 "slow url" 的请求。使用代理服务器并将 firefox 指向它,或者,如果可能的话,让 Firefox 知道将 URL 列入黑名单(可能通过扩展)。
另请参阅内部包含多个解决方法的相关问题:
另见:
- How to Stop the page loading in firefox programmatically?
- Firefox DelayedCommand waits for pending XMLHttpRequests when it doesn't need to.
- FirefoxDriver webdriver.load.strategy unstable findelements getting elements from the wrong page
- Set up a real timeout for loading page in Selenium WebDriver?
- Selenium Firefox Open timeout
