Columnar expressions¶
Expressions over pandas Series converge onto the columnar dialect of the IR and fuse into one kernel over the columns' memory, nulls included.
The example needs pandas (uv sync --group pandas). The runners skip it
where pandas is missing and say so.
Run it¶
Two expression trees, two fused loops¶
@ppy.pure
def blend(s: pd.Series, t: pd.Series) -> pd.Series:
return s * t + s.fillna(0.0)
@ppy.pure
def above(s: pd.Series, t: pd.Series) -> pd.Series:
return (s > t) & t.notna()
Arithmetic, comparison, fillna, isna/notna, and the boolean operators
are the surface pandas and PyArrow share. The plugin names them as the same
columnar operations PyArrow's compute lowers to.
Each tree becomes one loop, a columnar.map, that reads each input once and
writes the answer once. Nothing is materialized between the operators.
How nulls are handled¶
The nulls are whatever the Series' backing makes them:
- A NumPy-backed
float64Series runs under NumPy's convention. A NaN is the nullfillnafills andisnafinds, and a bool mask is a byte per row. The answer is written straight into the array the result Series wraps. - An Arrow-backed Series (
pd.ArrowDtype) is read in place with its validity bits through the Arrow C Data Interface, and the answer carries its own.
Either way the Series comes back over the callers' index with the same backing.
What stays with pandas¶
Anything the model does not capture exactly runs pandas itself, never an approximation. That includes:
- a Series of strings
- indexes that are not one index
- a nullable extension dtype
- a mix of backings
- an unknown method
Index alignment, copy-or-view, and dtype rules are pandas' own semantics,
and a frame is never treated as a 2-D tensor. The fused loop is one thread;
a polars plan runs across all of them.
Compared with pandas and polars¶
The two expressions over eight million rows, a NaN in every seventh row of
s and every eleventh of t, in compare/:
frames_bench.ppy,
frames_pandas.py,
frames_polars.py. Times are milliseconds, best
of five calls, over five processes.
- PPy is the pandas expression in a function.
- pandas is the same expression with no function around it, one pass and one temporary per operator (numexpr under it, where installed).
- polars is the expression rewritten in its own vocabulary, with the NaNs as nulls, planned and run across threads.
def blend(frame):
mixed = pl.col("s") * pl.col("t") + pl.col("s").fill_null(0.0)
return frame.select(mixed.alias("out"))["out"]
| PPy | pandas | polars | |
|---|---|---|---|
| blend | 22.47 ± 1.63 | 31.09 ± 1.11 | 13.24 ± 0.42 |
| above | 4.65 ± 0.37 | 6.78 ± 0.19 | 4.91 ± 0.19 |
The fused loop reads s and t once and writes the answer once, on one
thread, into the array the result Series wraps. pandas makes a temporary per
operator, three passes over 64 MB each, and numexpr under it splits the
arithmetic across threads to make that up. polars plans the expression and
runs it across every core, which is what its row is.
The point of the PPy column is the source: it is the pandas line, unchanged, with pandas' own NaN convention. The polars column is a different program.
Intel Core Ultra 9 386H (16 threads); pandas 3.0.5 with numexpr 2.14.2, polars 1.44.2 on CPython 3.12.13, PPy with pandas 3.0.5 on CPython 3.14.5, from a checkout on a native filesystem.
What it prints¶
python frames.ppy, ppy run frames.ppy
ppy inspect frames.ppy --stage columnar
(prints nothing; exits 0)
Read on: Plugins: pandas and PyArrow · The IR: the columnar and arrow dialects
frames.ppy is hand-written; there is no .py source and no conversion step.
41_columnar/frames.ppy¶
import pandas as pd
import ppy
@ppy.pure
def blend(s: pd.Series, t: pd.Series) -> pd.Series:
return s * t + s.fillna(0.0)
@ppy.pure
def above(s: pd.Series, t: pd.Series) -> pd.Series:
return (s > t) & t.notna()
def main() -> None:
s = pd.Series([1.0, float("nan"), 3.0, 4.0, 2.5], name="s")
t = pd.Series([2.0, 2.0, float("nan"), 0.5, 2.5], name="t")
print(blend(s, t).tolist())
print(above(s, t).tolist())
print(float(blend(s, t).sum()), int(above(s, t).sum()))
main()
Counterpart programs¶
The programs the comparison above measured, each written the way its tool expects. The PPy one is first.
frames_bench.ppy (PPy)
"""The two Series expressions of `frames.ppy` over eight million rows with NaNs, timed: PPY."""
import time
import numpy as np
import pandas as pd
import ppy
@ppy.pure
def blend(s: pd.Series, t: pd.Series) -> pd.Series:
return s * t + s.fillna(0.0)
@ppy.pure
def above(s: pd.Series, t: pd.Series) -> pd.Series:
return (s > t) & t.notna()
def main() -> None:
n = 8_000_000
left = np.linspace(0.0, 10.0, n)
right = np.linspace(5.0, 0.0, n)
left[::7] = np.nan
right[::11] = np.nan
s = pd.Series(left, name="s")
t = pd.Series(right, name="t")
blend(s, t)
above(s, t)
best = [1e9, 1e9]
for _ in range(5):
started = time.perf_counter()
mixed = blend(s, t)
best[0] = min(best[0], (time.perf_counter() - started) * 1000.0)
started = time.perf_counter()
mask = above(s, t)
best[1] = min(best[1], (time.perf_counter() - started) * 1000.0)
print(f"# blend: {best[0]:.2f} ms")
print(f"# above: {best[1]:.2f} ms")
print(f"{float(mixed.sum()):.3f} {int(mask.sum())} {int(mixed.isna().sum())}")
main()
frames_pandas.py (Python)
"""The same expressions as pandas evaluates them: one pass and one temporary per operator."""
import time
import numpy as np
import pandas as pd
def blend(s, t):
return s * t + s.fillna(0.0)
def above(s, t):
return (s > t) & t.notna()
def timed(label, run):
best = 1e9
answer = None
for _ in range(5):
started = time.perf_counter()
answer = run()
best = min(best, (time.perf_counter() - started) * 1000.0)
print(f"# {label}: {best:.2f} ms")
return answer
def main():
n = 8_000_000
left = np.linspace(0.0, 10.0, n)
right = np.linspace(5.0, 0.0, n)
left[::7] = np.nan
right[::11] = np.nan
s = pd.Series(left, name="s")
t = pd.Series(right, name="t")
mixed = timed("blend", lambda: blend(s, t))
mask = timed("above", lambda: above(s, t))
print(f"{float(mixed.sum()):.3f} {int(mask.sum())} {int(mixed.isna().sum())}")
main()
frames_polars.py (Python)
"""The same expressions in polars: lazy expressions the engine fuses and runs across threads."""
import time
import numpy as np
import polars as pl
def blend(frame):
mixed = pl.col("s") * pl.col("t") + pl.col("s").fill_null(0.0)
return frame.select(mixed.alias("out"))["out"]
def above(frame):
mask = (pl.col("s") > pl.col("t")) & pl.col("t").is_not_null()
return frame.select(mask.alias("out"))["out"]
def timed(label, run):
best = 1e9
answer = None
for _ in range(5):
started = time.perf_counter()
answer = run()
best = min(best, (time.perf_counter() - started) * 1000.0)
print(f"# {label}: {best:.2f} ms")
return answer
def main():
n = 8_000_000
left = np.linspace(0.0, 10.0, n)
right = np.linspace(5.0, 0.0, n)
left[::7] = np.nan
right[::11] = np.nan
# polars keeps nulls apart from NaN: the missing values arrive as nulls.
frame = pl.DataFrame(
{"s": pl.Series(left).fill_nan(None), "t": pl.Series(right).fill_nan(None)}
)
blend(frame)
above(frame)
mixed = timed("blend", lambda: blend(frame))
mask = timed("above", lambda: above(frame))
print(f"{float(mixed.sum()):.3f} {int(mask.sum())} {int(mixed.null_count())}")
main()
Source: examples/41_columnar.