Today I Remember - Python HTTP Requests Optimisation

In today’s world, speed is everything. When it comes to HTTP requests, faster is always better. Fortunately, there are many advanced methods to making faster HTTP requests in Python, including async requests, concurrent requests, and streaming options.

One of the most powerful options for faster HTTP requests is async requests. With async requests, you can send multiple requests simultaneously and wait for them all to return, rather than waiting for each request to complete before sending the next one. This can significantly reduce the time it takes to complete a large number of requests.

Another option for faster HTTP requests is concurrent requests. With concurrent requests, you can send multiple requests at the same time, but unlike async requests, you do not have to wait for all of them to return before moving on to the next one. This can be especially useful if you need to perform many HTTP requests in parallel, but do not want to wait for all of them to complete before moving on to other tasks.

Finally, there are streaming options for faster HTTP requests. Streaming allows you to download large files or data streams without having to wait for the entire file to download before you can start processing it. This can be especially useful when dealing with very large files or data streams.

There are some very clean and succint descriptions of these techniques here: (Wayback machine archive link as original site is down) https://web.archive.org/web/20221225024307/https://julien.danjou.info/python-and-fast-http-clients/