Skip to main content

Production Benchmark

This end-to-end benchmark uses a production-shaped deployment: FastAPI on uvicorn, DLRM CPU inference in the request path, Aerospike CE, k6 HTTP load, and a serialized response body.

The measurement includes HTTP parsing, JSON serialization, and the model forward pass. These shared costs make the client difference smaller than in the Isolated Benchmark, but they produce the numbers to use for capacity planning.

Sources: internal benchmark runs, April 2026. Raw measurement artifacts are not committed to the repo (see git history before the benchmark/ rewrite for the original benchmark/results/*.md files). The headline numbers below are reproduced from those artifacts; a clean re-measurement on the new isolated harness is tracked in Bottleneck Analysis.

Where the gap lives​

Mean latency advantage of aerospike-py vs official (Python 3.11 + GIL)

A) Pure DB client       ████████████████████  −80%  (108 → 22 ms)    🔥
B) uvicorn ASGI         █████                 −21%  (290 → 228 ms)
C) uvicorn + DLRM       ███████████           −42% p95 (324→189 ms)  🔥

p95 advantage holds even when mean compresses (C): the GIL serialises
official-client result conversion at the tail.

Environment A — Pure DB Client​

What this isolates. This environment removes FastAPI, model inference, and HTTP load generation. A Python loop calls batch_read directly, so the client difference is largest here.

Setup​

ItemValue
SourceInternal run, April 2026 (pre-rewrite artifact; recoverable from git history)
Clientsofficial (sync C), official-async (sync wrapped via executor), py-async (aerospike-py)
Sets / batch sizes9 / 50, 200
Concurrency / iterations10 / 30

Aggregate result (avg over 9 sets × 2 batch sizes)​

Clientavg mean (ms)avg p99 (ms)avg TPS
official107.56195.34138.2
official-async110.64211.33125.7
py-async (aerospike-py)22.45120.67373.7
py-async vs official4.8× faster latency1.6×2.7× higher TPS
py-async vs official-async4.9× faster1.8×3.0×

Per-set speedup distribution (aerospike-py vs official)​

Batch 50:   mean speedup across 9 sets   4.4× ──────── 7.8×    (median ~5.8×)
Batch 200:  mean speedup across 9 sets   3.1× ──────── 6.6×    (median ~4.5×)

Outlier: set_8 (0% found rate) → fast not-found path, not real read latency.
set_8 is the outlier

0% found rate — the official client returns errors faster than success paths, so this row reflects "fast not-found" rather than real read latency. All other sets show 4–8× mean speedup.

official-async wraps the synchronous C client with loop.run_in_executor. It is slightly slower than the bare sync client for every set because each request crosses the thread pool. As a result, aerospike-py's lead over official-async is consistently larger than its lead over official.

Pattern across environments​

   layers added →            ratio compresses,      tail still wins
   ─────────────             ─────────────────      ───────────────
A) Pure DB (no HTTP/ML)      mean 4.8×              p99 1.6×
B) uvicorn ASGI              mean 1.27×             ≈ noise at C=5
C) uvicorn + DLRM            mean 1.24×             p95 −42%  🔥

Adding layers around the database call reduces the mean ratio from 4.8× to 1.24×. The upper-percentile advantage remains: aerospike-py releases the GIL during I/O, while the official client serializes GIL acquisition through run_in_executor, increasing tail latency.

Python 3.14t free-threaded — same pipeline, GIL removed​

Python 3.14t is the free-threaded build of CPython — the GIL is disabled (Py_GIL_DISABLED=1). The numbers below come from the same Environment C pipeline (FastAPI + DLRM + k6 10 VUs × 60s), with only the runtime swapped. No Rust or C source changes.

TL;DR​

p95 single-mode (k6 10 VUs × 60s, FastAPI + DLRM)

aerospike-py
  3.11 + GIL    ███████████████              189 ms
  3.14t         ████████                      97 ms   −49% 🔥

official (C extension)
  3.11 + GIL    █████████████████████████    324 ms
  3.14t         ██████████                   128 ms   −60% 🔥

Throughput (aerospike-py): 41.6 → 61.2 iter/s   +47%

The GIL was a shared bottleneck for both clients. aerospike-py's gain came without touching Rust.

What gets faster (and why)​

Internal stage timings under load. The two GIL-bound stages collapse to near-zero.

Stage                     3.11 + GIL              3.14t       Change
────────────────────────────────────────────────────────────────────
spawn_blocking_delay      234 ms     ████████    0.12 ms      −99.95% 🔥
event_loop_resume_delay   39.7 ms    ██           ≈ 0         ≈ −100%
io (Aerospike network)    7.51 ms    ▌           1.27 ms      −83%
merge_to_dict             4.48 ms    ▎           3.54 ms      −21%
key_parse                 967 μs     ·           1.06 ms      +10% (noise)
tokio_schedule_delay      83.1 μs                49.5 μs      −40%
limiter_wait              3.56 μs                0.96 μs      −73%

Two stages dominate the gain:

  • spawn_blocking_delay drops from 234 ms to 0.12 ms. Under GIL, when a Rust async future completes and needs to convert results into Py<...> objects, it dispatches that work to a spawn_blocking worker. That worker has to acquire the GIL — and under contention from the asyncio event loop and other workers, the queue stretches into hundreds of milliseconds. With no GIL, the worker runs immediately.
  • event_loop_resume_delay drops to effectively zero. Under GIL, after the future resolves and the event loop is woken up, the coroutine still has to wait its turn for the GIL before resuming. Free-threaded mode lets multiple coroutines resume in parallel.

io shrinking 8× is a second-order effect: Tokio workers can now parse Aerospike protocol responses without contending with Python code for the GIL.

Same-workload comparison on 3.14t​

On 3.14t, when both clients hit the same server load (alternating endpoints in the same pod, k6 10 VUs):

Clientp95 (single mode)Difference
aerospike-py126 msbaseline
official aerospike128 ms+2 ms (~1.5%, noise)

The 42% gap that existed under 3.11 + GIL collapses to ~2 ms under 3.14t.

This is the cleanest evidence that GIL contention — not architectural difference — was responsible for the bulk of the original gap.

Throughput (TPS)​

3.11 + GIL, stage OFF             ████████████        41.6  baseline
3.11 + GIL, stage ON              █████████████       44.1  +6% (noise)
3.14t, aerospike-py only          ███████████████████ 61.2  +47% 🔥
3.14t, both clients (split load)  ██████████████      47.3  +14%
Configiterations/shttp_reqs/s
3.11 + GIL, stage OFF41.650.8
3.11 + GIL, stage ON44.152.9
3.14t, aerospike-py only61.280.0
3.14t, both clients (split load)47.359.8

But aerospike-py still wins under solo load on 3.14t​

The "126 vs 128 ms" tie comes from a configuration where each client only sees ~5 effective VUs (10 VUs split across two endpoints). When the load is concentrated on one client, the gap reopens.

Metricaerospike-py soloofficial soloaerospike-py advantage
k6 single p9597 ms134 ms−28%
k6 gather p95 (9× fan-out)107 ms253 ms−58% 🔥
Server predict_duration_seconds p95100 ms138 ms−28%
Server batch_read_all p9564 ms67 ms−4%
Theoretical capacity (Little's Law: 10 VUs / p95)~103 req/s~75 req/s+37%

Why aerospike-py still leads under solo load on 3.14t:

  1. Native async vs threadpool wrapping. Even without GIL contention, run_in_executor adds a thread-pool hop per request. aerospike-py awaits directly on the asyncio loop.
  2. Lazy dict conversion. batch_read() returns a LazyBatchRecords handle. The Python-dict materialisation happens lazily when the caller invokes .to_dict(); .to_numpy(dtype) provides the GIL-released structured-array path.
  3. Single FFI boundary crossing. The aerospike-py Rust code completes a full batch_read inside one PyO3 call. The C extension crosses the Python ↔ C boundary multiple times per call.

These advantages compound as concurrency rises (the gather number — 107 vs 253 ms — is the clearest example).

Migration notes​

  1. Add a 3.14t row to CI matrix. Run unit + integration tests under python:3.14.2t-slim.
  2. Audit unsafe and shared mutable state in Rust before declaring gil_used = false.
  3. Wait for or build official client wheels. PyPI does not yet ship cp314t wheels for the official aerospike package; source build works.

Side-effect: inference also got faster. DLRM inference (PyTorch CPU, control variable) dropped from 43.5 ms → 20.7 ms (−52%) on 3.14t. Unrelated to aerospike-py — a free side benefit for any GIL-bound inference path running alongside async I/O.

What's next:

  • Isolated Benchmark — strip the FastAPI/JSON/inference cost out to see the client gap by itself
  • Bottleneck Analysis — per-stage profiling, the gather→single recipe (TODO: re-measure on the new harness)