Vaex

When to use

Use Vaex for columnar analysis on a single machine when data exceeds RAM, especially repeated reductions and histograms over local Vaex HDF5 or Arrow files. Expressions and virtual columns defer computation; reductions normally execute immediately. Out-of-core storage does not make every operation memory bounded: sorting, joins, large group dictionaries, materialization, and many estimator fits need substantial RAM.

Installation and verified scope

Use a separate environment; the repository's default Python is newer than this release supports:

uv venv --python 3.12 .venv-vaex
uv pip install --python .venv-vaex/bin/python "vaex-core==4.19.0" "vaex-hdf5==0.15.0" "vaex-viz==0.6.0"
# Optional ML (also installs its declared estimator dependencies):
uv pip install --python .venv-vaex/bin/python "vaex-ml==0.19.0"

On Windows use .venv-vaex\Scripts\python.exe as the interpreter path. The vaex 4.19.0 metapackage installs more integrations; it is not needed for the core workflow. Core 4.19.0 declares Python >=3.9,<3.13, pandas <3, Dask <2024.9, and NumPy <3. Do not upgrade these constraints independently. Arrow support is in core; FITS needs vaex-astro. Compatible binary wheels determine platform availability; compiling the optional annoy dependency requires a C++ toolchain, not just Python headers.

Native checks used Python 3.12, core 4.19.0, HDF5 0.15.0, viz 0.6.0, ML 0.19.0, NumPy 2.5.3, pandas 2.3.3, PyArrow 25.0.1 and Matplotlib 3.11.2 on macOS ARM. See review and verification for evidence and optional-integration limits. These are correctness checks on small synthetic inputs, not performance benchmarks.

Workflow

1. Establish row identity, units, schema, missing-value codes, and expected counts. Inspect CSV raw headers before parsers rename duplicates; supply explicit types for IDs and late-appearing values. Keep dates, time zones, and sampling cadence explicit.
2. Open files with vaex.open. HDF5 must use a compatible table layout; arbitrary HDF5 scientific arrays are not automatically a Vaex table. CSV opening performs indexing/schema work; Parquet must decode compressed data. Neither is an instant, zero-memory operation.
3. Select needed columns and use expressions for derived values. A virtual column avoids a full stored array but still needs expression metadata and evaluation buffers.
4. Record filters/selections and missingness before reductions. Batch independent statistics with delay=True, then df.execute() and each promise's .get().
5. Validate counts, units, join cardinality, and numerical results against a small independently computed subset. Binned or approximate summaries need explicit limits/resolution.
6. Plot aggregated grids or a bounded sample. A count heatmap and a mean heatmap answer different questions; show coverage and avoid hiding rare/extreme observations silently.
7. Export directly in chunks; exporting evaluates virtual columns without needing materialize() first. Reopen and check counts/schema/values before replacing source data.

Small executable example

Run in a writable working directory; output names must not refer to existing data.

from pathlib import Path
import numpy as np
import vaex

out = Path('vaex-example.hdf5')
if out.exists():
    raise FileExistsError(out)
df = vaex.from_arrays(
    x=np.arange(1., 7.), y=np.arange(6.) ** 2,
    category=np.array(['A', 'B', 'A', 'B', 'A', 'B']),
)
df['energy'] = df.x ** 2 + df.y
selected = df[df.x >= 3]
mean_task = selected.energy.mean(delay=True)
count_task = selected.count(delay=True)
selected.execute()
assert count_task.get() == 4
assert np.isclose(mean_task.get(), 35.0)
summary = df.groupby('category', agg={
    'rows': vaex.agg.count(), 'energy_sum': vaex.agg.sum('energy'),
})
assert int(summary.rows.sum()) == len(df)
df.export_hdf5(str(out), chunk_size=2)
reopened = vaex.open(str(out))
assert reopened.get_column_names() == df.get_column_names()
assert np.allclose(reopened.energy.to_numpy(), df.energy.to_numpy())

For a large real input, replace the in-memory fixture with vaex.open('input.hdf5'). The small .to_numpy() comparison above is a fixture check; do not apply it to a whole larger-than-RAM dataset. Compare sampled rows and streamed summaries instead.

Reference map

• Core DataFrames: loaders, expression/array distinctions, inspection and schema.

…