Skip to main content

Expressions

Many methods take a function that runs once per row: filter((e) => e.get('cpu') > 0.5). An expression says the same thing as data instead: col('cpu').gt(0.5). Because pond can read an expression, it runs it over whole columns at once rather than calling a function for every row. That is usually faster, and an expression can be saved or sent as plain JSON, which a function can't.

Functions still work everywhere they did. Use an expression when the rule is simple (compare, combine, do arithmetic, pick between two values), and a function for anything else.

:::note Versions Expressions are new in pond-ts 0.76.0, and in the Python package (pond_rs) from 0.76.0. In Rust they are on main but not yet in a pond-rs release: crates.io's 0.1.0 doesn't have them, so build pond-rs from the repository to use them for now. :::

Building an expression​

Three functions start one:

  • col('cpu'): the value of column cpu in each row.
  • lit(1): a fixed value. Plain values are wrapped for you (col('cpu').gt(0.5)), so you only need lit when the fixed value comes first: lit(1).sub(col('cpu')).
  • when(condition, then, otherwise): then where the condition is true, otherwise where it isn't.

Then chain methods onto it:

groupmethodsgives
compareeq, ne, gt, ge, lt, le, in([...])true/false
combineand, or, nottrue/false
arithmeticadd, sub, mul, div, neg, absa number
gapsisMissing(), fill(value)as named
choicewhen(condition, then, otherwise)then's kind

There are no text functions, rounding or time-of-day functions yet, and an expression can't look at other rows (use shift, diff or rolling for that) or add up many rows (use aggregate or reduce).

In Rust the names are the same except is_in for in and is_missing for isMissing. Python has methods too (and_, or_, not_, is_in, is_missing, since and, or, not and in are Python keywords), but operators read better there:

PythonTypeScript / Rust
== !=eq ne
> >= < <=gt ge lt le
& | ~and or not
+ - * /add sub mul div
-x, abs(x)neg abs

Put brackets around each comparison when you combine them, (col("cpu") > 0.5) & (col("host") == "a"), because & binds tighter than > in Python. Python's own and, or and not don't work on expressions: they raise a TypeError rather than quietly dropping one side.

Where you can use one​

These methods take an expression wherever they take a function:

methodthe expression must give
filter, find, some, everytrue/false
mapColumns({ column: expr })the column's own kind
withColumn(name, expr)anything; the new column takes its kind
live filtertrue/false

filter, find, some, every​

import { TimeSeries, col } from 'pond-ts';

const cpu = new TimeSeries({
  name: 'cpu',
  schema: [
    { name: 'time', kind: 'time' },
    { name: 'cpu', kind: 'number', required: false },
    { name: 'cores', kind: 'number' },
    { name: 'host', kind: 'string' },
  ],
  rows: [
    [0, 0.2, 2, 'a'],
    [60_000, undefined, 4, 'b'],
    [120_000, 0.9, 4, 'a'],
    [180_000, 0.6, 0, 'c'],
    [240_000, 0.4, 2, 'b'],
  ],
});

const busy = cpu.filter(col('cpu').gt(0.5).and(col('host').eq('a')));
busy.length; // 1 (the row at 120_000)

cpu.find(col('cpu').gt(0.5))?.get('host'); // 'a'
cpu.some(col('host').in(['c', 'd'])); // true
cpu.every(col('cpu').isMissing().not()); // false: one row has a gap

find, some and every still stop early: they check a small stretch of rows first, then bigger ones, so a match near the start costs little.

mapColumns and withColumn​

withColumn(name, expr) adds a column. mapColumns({ column: expr }) replaces one; the expression may read other columns, and with several entries every expression reads the columns as they were before any of them changed.

import { col, lit, when } from 'pond-ts';

// A new column from two others. 0.6 / 0 gives a gap.
cpu.withColumn('load', col('cpu').div(col('cores')));
// load: 0.1, gap, 0.225, gap, 0.2

// Replace a column in place.
cpu.mapColumns({ cpu: col('cpu').mul(100) });
// cpu: 20, gap, 90, 60, 40

// Pick between two values. The gap row gets 'low'.
cpu.withColumn('level', when(col('cpu').gt(0.5), 'high', 'low'));
// level: 'low', 'low', 'high', 'high', 'low'

// lit() when the fixed value comes first.
cpu.withColumn('idle', lit(1).sub(col('cpu')));
// idle: 0.8, gap, 0.09999999999999998, 0.4, 0.6

The new column from withColumn is optional (it may have gaps) and takes the expression's kind: number, text or true/false.

A simple mapColumns like col('cpu').mul(100) is slower as an expression than as a function in pond-ts (see Speed). There the gain is reading other columns, not speed; (x) => x * 100 is still the quicker way to scale one column.

Live filter​

A live filter takes an expression too. It is checked against the columns when you build the view, then run on each event as it arrives.

import { LiveSeries, col } from 'pond-ts';

const live = new LiveSeries({
  name: 'cpu',
  schema: [
    { name: 'time', kind: 'time' },
    { name: 'cpu', kind: 'number', required: false },
    { name: 'host', kind: 'string' },
  ],
});
const hot = live.filter(col('cpu').gt(0.5));

live.push(
  [0, 0.9, 'a'],
  [1000, undefined, 'b'],
  [2000, 0.3, 'a'],
  [3000, 0.7, 'b'],
);
hot.length; // 2: the gap row is left out

Gaps​

An expression gives the same answer as the plain function it replaces, where a gap reads as undefined:

  • A gap fails every comparison. col('cpu').gt(0.5) and col('cpu').le(0.5) are both false for a gap row, just as undefined > 0.5 and undefined <= 0.5 are.
  • not flips the result, so col('cpu').gt(0.5).not() is true for a gap row. "Not above 0.5" keeps gaps; "at most 0.5" doesn't. If you come from SQL, where a gap stays unknown through NOT, this is the surprise. Ask for gaps by name with isMissing().
  • eq is false when either side is a gap, so ne is true there.
  • Arithmetic on a gap gives a gap, and so does a result that isn't a finite number, such as dividing by zero.
  • and, or and when read a gap as false.
  • In mapColumns a gap stays a gap, whatever the expression says, so fill changes nothing there. Use withColumn to fill.
// cpu: 0.2, gap, 0.9, 0.6, 0.4
cpu.filter(col('cpu').gt(0.5)).length; // 2
cpu.filter(col('cpu').le(0.5)).length; // 2: the gap fails this too
cpu.filter(col('cpu').gt(0.5).not()).length; // 3: not() brings the gap back

cpu.filter(col('cpu').eq(0.2)).length; // 1
cpu.filter(col('cpu').ne(0.2)).length; // 4, including the gap
cpu.filter(col('cpu').isMissing()).length; // 1

cpu.mapColumns({ cpu: col('cpu').fill(0) }); // cpu: 0.2, gap, 0.9, 0.6, 0.4
cpu.withColumn('cpu0', col('cpu').fill(0)); // cpu0: 0.2, 0, 0.9, 0.6, 0.4

Checks​

An expression is checked against the series' columns when a method receives it, before any row runs. You get an error for:

  • a column that doesn't exist (the message lists the ones that do);
  • values of the wrong kind, such as comparing a text column with a number or adding text;
  • filter, find, some, every or live filter given something that isn't true/false;
  • in([...]) values that don't match the column's kind.

col('name') takes a plain string, so TypeScript doesn't check the name while you type; the check happens when the code runs.

One difference: mapColumns in pond-ts skips a column name that doesn't exist (mapColumns({ nope: col('cpu') }) returns the series unchanged), as it does for functions. pond-rs (and Python) give an error instead.

cpu.filter(col('nope').gt(0));
// TypeError: filter: unknown column 'nope' (columns: cpu, cores, host)
cpu.filter(col('host').gt(0));
// TypeError: filter: gt needs two values of the same kind, got string and number
cpu.filter(col('cpu').mul(2));
// TypeError: filter: needs a true/false expression, got number

cpu.mapColumns({ nope: col('cpu') }); // no error: unknown names are skipped

Saving an expression as JSON​

expr.toJSON() gives a plain tree, and JSON.stringify uses it:

JSON.stringify(col('cpu').gt(0.5).and(col('host').eq('a')));
// {"op":"and","args":[{"op":"gt","args":[{"col":"cpu"},{"lit":0.5}]},
//  {"op":"eq","args":[{"col":"host"},{"lit":"a"}]}]}

pond-rs reads and writes the same tree (Expr::to_json, Expr::from_json, shown in the Rust tab above), and so does Python (expr.to_json(), pond_rs.Expr.from_json(text)). So a rule saved by one can run in another, or be written by a tool that never runs code.

Speed​

Measured on 1M rows when expressions were built (milliseconds, lower is better):

case (pond-ts)functionexpression
filter v > 05217
filter v / w > 10 and text4719
some, no match1234
withColumn v / w374.1
mapColumns v * 1.8 + 324.78.6

mapColumns is the one place an expression is slower in pond-ts: a function there already works straight on the column in one pass, while the expression takes one pass per operation (four here).

In Rust a closure is already compiled code, so an expression is a little slower (filter v > 0: 8 to 13 ms with a closure, 9.7 ms as an expression; replacing a column: 0.7 against 2.8 ms). Use one in Rust when the rule arrives as data, or to match a pond-ts rule exactly. In Python the gap is large the other way, because a function means a call into Python for every row: filter took 360 ms with a function and 9.7 ms as an expression, map 162 against 2.3 ms.