Skip to main content
 首页 » 编程设计

python之使用 aiohttp 检测 http 响应编码

2025年05月04日11mengfanrong

我正在尝试学习如何使用 asyncio 构建异步网络爬虫。以下是用于测试框架的粗略爬虫:

import asyncio, aiohttp 
from bs4 import BeautifulSoup 
 
@asyncio.coroutine 
def fetch(url): 
    with (yield from sem): 
        print(url) 
        response = yield from aiohttp.request('GET',url) 
        response = yield from response.read_and_close() 
    return response.decode('utf-8') 
 
@asyncio.coroutine 
def get_links(url): 
    page = yield from fetch(url) 
    soup = BeautifulSoup(page) 
    links = soup.find_all('a',href=True) 
    return [link['href'] for link in links if link['href'].find('www') != -1] 
 
@asyncio.coroutine 
def crawler(seed, depth, max_depth=3): 
    while True: 
        if depth > max_depth: 
            break 
        links = yield from get_links(seed) 
        depth+=1 
        coros = [asyncio.Task(crawler(link,depth)) for link in links] 
        yield from asyncio.gather(*coros) 
 
sem = asyncio.Semaphore(5) 
loop = asyncio.get_event_loop() 
loop.run_until_complete(crawler("http://www.bloomberg.com",0)) 

虽然 asyncio 似乎有很好的文档记录,但 aiohttp 似乎只有很少的文档,所以我正在努力自己解决一些问题。

首先,我们有没有办法检测页面响应的编码? 其次,我们可以请求连接在 session 中保持事件状态吗?还是像 requests 中那样默认为 True?

请您参考如下方法:

您可以查看 response.headers['Content-Type'] 或使用 chardet 库来处理格式错误的 HTTP 响应。响应体是bytes字符串。

对于 keep-alive 连接,您应该使用 connector,例如:

connector = aiohttp.TCPConnector(share_cookies=True) 
 
response1 = yield from aiohttp.request('get', url1, connector=connector) 
body1 = yield from response1.read_and_close() 
response2 = aiohttp.request('get', url2, connector=connector) 
body2 = yield from response2.read_and_close()