Python’s asynchronous programming functionality, or async for short, allows you to write programs that get more work done by not waiting for independent tasks to finish. The
asyncio library included with Python gives you the tools to use async for processing disk or network I/O without making everything else wait.
asyncio provides two kinds of APIs for dealing with asynchronous operations: high-level and low-level. The high-level APIs are the most generally useful, and they’re applicable to the widest variety of applications. The low-level APIs are powerful, but also complex, and used less frequently.
We’ll concentrate on the high-level APIs in this article. In the sections below, we’ll walk through the most commonly used high-level APIs in
asyncio, and show how they can be used for common operations involving asynchronous tasks.
If you’re completely new to async in Python, or you could use a refresher on how it works, read my introduction to Python async before diving in here.
Run coroutines and tasks in Python
Naturally, the most common use for
asyncio is to run the asynchronous parts of your Python script. This means learning to work with coroutines and tasks.
Python’s async components, including coroutines and tasks, can only be used with other async components, and not with conventional synchronous Python, so you need
asyncio to bridge the gap. To do this, you use the
async def main():
print ("Waiting 5 seconds. ")
for _ in range(5):
print ("Finished waiting.")
main(), along with any coroutines
main() fires off, and waits for a result to return.
As a general rule, a Python program should have only one
.run() statement, just as a Python program should have only one
main() function. Async, if used carelessly, can make the control flow of a program hard to read. Having a single entry point to a program’s async code keeps things from getting hairy.
Async functions can also be scheduled as tasks, or objects that wrap coroutines and help run them.
async def my_task():
task = asyncio.create_task(my_task())
my_task() is then run in the event loop, with its results stored in
If you have only one task you want to get results from, you can use
asyncio.wait_for(task) to wait for the task to finish, then use
task.result() to retrieve its result. But if you’ve scheduled a number of tasks to execute and you want to wait for all of them to finish, use
asyncio.wait([task1, task2]) to gather the results. (Note that you can set a timeout for the operations if you don’t want them to run past a certain length of time.)
Manage an async event loop in Python
Another common use for
asyncio is to manage the async event loop. The event loop is an object that runs async functions and callbacks; it’s created automatically when you use
asyncio.run(). You generally want to use only one async event loop per program, again to keep things manageable.
If you’re writing more advanced software, such as a server, you’ll need lower-level access to the event loop. To that end, you can “lift the hood” and work directly with the event loop’s internals. But for simple jobs you won’t need to.
Read and write data with streams in Python
The best scenarios for async are long-running network operations, where the application may block waiting for some other resource to return a result. To that end,
asyncio offers streams, which are high-level mechanisms for performing network I/O. This includes acting as a server for network requests.
asyncio uses two classes,
StreamWriter, to read and write from the network at a high level. If you want to read from the network, you would use
asyncio.open_connection() to open the connection. That function returns a tuple of
StreamWriter objects, and you would use
.write() methods on each to communicate.
To receive connections from remote hosts, use
asyncio.start_server() function takes as an argument a callback function,
client_connected_cb, which is called whenever it receives a request. That callback function takes instances of
StreamWriter as arguments, so you can handle the read/write logic for the server. (See here for an example of a simple HTTP server that uses the
Synchronize tasks in Python
Asynchronous tasks tend to run in isolation, but sometimes you will want them to communicate with each other.
asyncio provides queues and several other mechanisms for synchronizing between tasks:
asyncioqueues allow asynchronous functions to line up Python objects to be consumed by other async functions — for instance, to distribute workloads between different kinds of functions based on their behaviors.
- Synchronization primitives: Locks, events, conditions, and semaphores in
asynciowork like their conventional Python counterparts.
One thing to keep in mind about all of these methods is that they’re not thread-safe. This isn’t an issue for async tasks running in the same event loop. But if you’re trying to share information with tasks in a different event loop, OS thread, or process, you’ll need to use the
threading module and its objects to do that.
Further, if you want to launch coroutines across thread boundaries, use the
asyncio.run_coroutine_threadsafe() function, and pass the event loop to use with it as a parameter.
Pause a coroutine in Python
Another common use of
asyncio, and an under-discussed one, is waiting for some arbitrary length of time inside a coroutine. You can’t use
time.sleep() for this, or you’ll block the entire program. Instead, use
asyncio.sleep(), which allows other coroutines to continue running.
Use lower-level async in Python
Finally, if you think that the app you’re building may require
asyncio’s lower-level components, take a look around before you start coding: There’s a good chance someone has already built an async-powered Python library that does what you need.
For instance, if you need async DNS querying, check the
aiodns library, and for async SSH sessions, there’s
asyncSSH. Search PyPI by the keyword “async” (plus other task-related keywords), or check the hand-curated Awesome Asyncio list for ideas.