异步接口外,其它使用方式完全一致。
完整代码
import asyncio
import httpx
from concurrent.futures import ThreadPoolExecutor, Future
import time
import contextvars
# 测试时将测试网址替换
urls = [
"http://www.bxxxx.com",
"http://www.aaaa.com",
"http://www.bbbb.com",
"http://www.cccc.com",
"http://www.sdddd.com",
"http://www.jdddd.com",
"http://www.zeeee.com",
"http://www.tffff.com",
"http://www.cgggg.com",
"http://www.zhhhhh.com.cn",
"http://www.google.com",
"https://www.yiiiii.com/",
]
async def check_one_ip(url):
headers = {
"user-ageng": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \
(KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 Edg/116.0.1938.69"
}
TIMEOUT = 3
result = ()
try:
async with httpx.AsyncClient() as client:
response = await client.get(url, headers=headers,timeout=TIMEOUT)
print(f"response from {url} is : {response.status_code}")
if 200 <= response.status_code < 300:
print(f"length of response body is {len(response.text)}")
result = (url, response.status_code)
except Exception as e:
print(f"{url} met timeout error")
return (url, 999)
return result
def task_callback(context):
# print response.status_code
url, code = context.result()
print(f"It is callback, got status_code: {code} of {url}")
async def main():
tasks=[]
for url in urls:
task = asyncio.create_task(check_one_ip(url))
task.add_done_callback(task_callback)
tasks.append(task)
await asyncio.gather(*tasks)
if __name__=="__main__":
t1 = time.time()
asyncio.run(main())
t2 = time.time()
print(f"total time: {t2-t1:.3f}s")
运行结果如下,可以看到,总耗时: 3.161s
,相比同步编程方式,耗时减少了1半。 随着发送请求量的增加,可以看到更加明显的效果。
response from url is : 302
It is callback, got status_code: 302 of url
response from url is : 302
It is callback, got status_code: 302 of url
response from url is : 200
length of response body is 23508
It is callback, got status_code: 200 of url
response from url is : 302
response from url is : 301
It is callback, got status_code: 302 of url
It is callback, got status_code: 301 of url
response from url is : 301
response from url is : 301
response from url is : 301
response from url is : 200
length of response body is 396837
It is callback, got status_code: 301 of url
It is callback, got status_code: 301 of url
It is callback, got status_code: 301 of url
It is callback, got status_code: 200 of url
response from url is : 404
It is callback, got status_code: 404 of url
response from url is : 200
length of response body is 1151330
It is callback, got status_code: 200 of url
url met timeout error
It is callback, got status_code: 999 of url
total time: 3.161s
五、异步编程注意事项
1)协程不应该执行耗时长的任务
异步event loop执行期间,虽然各个协程是在工作,但主线程是被阻塞的。本例中,异步耗时的总时长与访问google.com超时时长相同,那么意味着,如果协程中如果有1个是耗时很长的任务,那么主线程还将被阻塞,异步解决不了这个问题,这时耗时协程应该拿出来,用子线程、或者子进程来执行。
2) 协程应该汇集后并发执行
遇到一些开发者咨询,为什么采用了异步编程,但性能没有明显提升呢? 创建多个协程任务后,必须按第3步,用gather()方法来汇集创建的协程任务,然后用asyncio.run()方法并发运行。 另外官方文档要求 event loop要在主线程main() 方法中运行。
3)慎用底层编程接口
另外由于官方文档并未清晰说明 event loop、future对象等低层编程接口,除非你很了解异步低层的实现机制,否则不建议使用低层接口,
使用ayncio.run() 来启动evnetloop, 使用 task 对象,而非future 对象。