Skip to main content

NumPy Integration Guide

Use NumPy structured arrays for high-performance batch reads and writes. The Rust client moves data directly between Aerospike and NumPy buffers without creating a Python object for each element.

Requirement

Requires numpy >= 2.0. Install with: pip install aerospike-py[numpy]

When to Use NumPy Batch​

ScenarioRegular batch_readNumPy batch_read
Records < 100PreferredOverhead not justified
Records 100–10KOK2–5x faster
Records > 10KSlow (dict allocation)5–10x faster
Non-numeric bins (strings, lists)RequiredNot supported
Vectorized analyticsManual conversionNative numpy arrays

Batch Read with NumPy​

Define a dtype​

Each field in the dtype maps to an Aerospike bin name:

import numpy as np

# Fields must be numeric (int/uint/float) or fixed-length bytes
dtype = np.dtype([
("score", "f8"), # float64
("count", "i4"), # int32
("level", "u2"), # uint16
("tag", "S8"), # 8-byte fixed string
])

Read into NumPy arrays​

keys = [("test", "demo", f"user_{i}") for i in range(1000)]

result = client.batch_read(keys, bins=["score", "count", "level", "tag"]).to_numpy(dtype)
# result is a NumpyBatchRecords instance

Access data​

# Vectorized operations on the full array
avg_score = result.batch_records["score"].mean()
high_scorers = result.batch_records[result.batch_records["score"] > 90]

# Individual record by primary key
record = result.get("user_42")
print(record["score"], record["count"])

# Metadata arrays
print(result.meta["gen"]) # generation numbers
print(result.meta["ttl"]) # TTL values

# Result codes (0 = success)
success_mask = result.result_codes == 0
valid_records = result.batch_records[success_mask]

Async batch read​

lazy_records = await async_client.batch_read(keys, bins=["score", "count"])
result = lazy_records.to_numpy(dtype)

Batch Write with NumPy​

Write records from a structured array. One designated field serves as the primary key.

import numpy as np

dtype = np.dtype([
("_key", "i4"), # primary key field (prefixed with _)
("score", "f8"),
("count", "i4"),
])

data = np.array([
(1, 95.5, 10),
(2, 87.3, 20),
(3, 92.1, 15),
], dtype=dtype)

results = client.batch_write_numpy(data, "test", "demo", dtype)
# Returns list of Record NamedTuples

Key field conventions​

  • Default key field: "_key" (configurable via key_field parameter)
  • Fields prefixed with _ are excluded from bins (only _key is used as the record key)
  • All other fields become Aerospike bins
# Custom key field name
dtype = np.dtype([("user_id", "i4"), ("score", "f8")])
data = np.array([(100, 95.5), (200, 87.3)], dtype=dtype)
results = client.batch_write_numpy(data, "test", "demo", dtype, key_field="user_id")

Async batch write​

results = await async_client.batch_write_numpy(data, "test", "demo", dtype)

Supported dtype Kinds​

NumPy kindCodeExamplesAerospike type
Signed intii1, i2, i4, i8Integer
Unsigned intuu1, u2, u4, u8Integer
Floatff2, f4, f8Float
Fixed bytesSS8, S16, S32String (truncated)
Void bytesVV8, V16Blob (truncated)
Unsupported types

Variable-length strings (U), objects (O), and datetime (M/m) are not supported. Convert to fixed-length types before writing.

Pandas Integration​

Read into DataFrame​

import pandas as pd

result = client.batch_read(keys, bins=["score", "count"]).to_numpy(dtype)

# Direct conversion — zero copy for numeric data
df = pd.DataFrame(result.batch_records)
df["success"] = result.result_codes == 0

Write from DataFrame​

# Convert DataFrame to structured array
dtype = np.dtype([("_key", "i4"), ("score", "f8"), ("count", "i4")])
data = np.array(list(df[["id", "score", "count"]].itertuples(index=False)), dtype=dtype)
client.batch_write_numpy(data, "test", "demo", dtype)

Strict Mode​

Enable warnings for missing or extra bins:

result = client.batch_read(keys, bins=["score", "count"]).to_numpy(dtype)
# No warnings — missing bins are zero-filled, extra bins are ignored

# With strict=True (internal API via _batch_records_to_numpy):
# Warns when dtype fields are missing from records
# Warns when record bins are not in dtype

NumpyBatchRecords API​

Method / AttributeDescription
batch_recordsStructured numpy array with bin data
meta(gen, ttl) structured array
result_codesint32 array (0 = success)
get(key)Retrieve single record by primary key
len(result)Number of records
key in resultCheck if primary key exists
for r in resultIterate over records

Performance Tips​

  1. Pre-allocate dtypes — Define dtype once and reuse across calls
  2. Match dtype to data — Use smallest sufficient type (i4 vs i8, f4 vs f8)
  3. Batch size — Optimal range: 500–5000 records per call
  4. Use fixed-length strings — S16 is much faster than variable-length alternatives
  5. Filter server-side — Combine with expression filters to reduce data transfer