httpx

httpx is a new generation HTTP client for Python3. It has actually the same API with the famous requests library.

The catch is that it also supports async requests natively. Tom Christie, the author of the Django Rest Framework, presented httpx in fosdem in this weekend.

I'm impressed.

It has by far the cleanest API available for a generic request client. I made a simple real world use case test to see the difference between sync and async calls, out of curiosity.

Parsing blocks in batch: Sync vs Async (100 blocks)

This is the sync version with the Requests library:

import requests
import time


def build_request_body(block_num):
    return {
        "jsonrpc": "2.0",
        "method": "block_api.get_block",
        "params": {"block_num": block_num}
    }


def get_blocks_sync(start, stop):
    start_time = time.time()
    total_tx_count = 0
    for block_num in range(start, stop + 1):
        response = requests.post(
            "https://api.steemit.com",
            json=build_request_body(block_num)
        ).json()
        total_tx_count += len(response['result']['block']['transactions'])

    execution_time = int(time.time() - start_time)
    print(f"[sync] {total_tx_count} transactions found in {execution_time} seconds.")


get_blocks_sync(40000000, 40000100)

Output:

(httpx) ➜  httpx_playground python parser_sync.py
[sync] 3015 transactions found in 49 seconds.

Async version with httpx:

import asyncio
import httpx
import time


def build_request_body(block_num):
    return {
        "jsonrpc": "2.0",
        "method": "block_api.get_block",
        "params": {"block_num": block_num}
    }


async def get_block_async(start, stop):
    total_tx_count = 0
    start_time = time.time()
    async with httpx.AsyncClient() as client:
        for block_num in range(start, stop + 1):
                response = await client.post(
                    "https://api.steemit.com",
                    json=build_request_body(block_num)
                )
                response = response.json()
                total_tx_count += len(response['result']['block']['transactions'])

    execution_time = int(time.time() - start_time)
    print(f"[async] {total_tx_count} transactions found in {execution_time} seconds.")

asyncio.run(get_block_async(40000000, 40000100))

Output:

(httpx) ➜  httpx_playground python parser_async.py
[async] 3015 transactions found in 13 seconds.

Parsing 100 blocks makes 36 seconds in difference. Async version completes in 13 seconds while the sync version completes in 49 seconds.

It's a huge performance benefit. I'll try to implement an "async" block streamer class for Lightsteem with httpx in the future.

Vote for my witness

If you didn't vote already, consider casting a vote for my witness on Steemconnect or on Steemit

Python community

This is originally posted on Python community in Steemit Beta. Check it out and subscribe if you're interested in Python related content.