Learn · from docs/plan-v2.md in the eLucid8 repository

Contents

The eLucid8 execution plan, version 2

Status: Implemented 2026-10-04 (Architecture v1, Part A2b, stage 1; owner decision "After A1: A2 in three steps", item 1; ARCHITECTURE item 59). Updated 2026-10-04 for the owner's four clarifications (decision "A2b stage 1 accepted; four semantic clarifications"; ARCHITECTURE item 60): lists are not comparable (§6.1), neighbourhood aggregates need numbers (§6.6), the depth limit is a resource limit (§6.7) and upstream chains are serialised compressed (§9.1); §11 records them. Updated 2026-10-04 for A2b stage 2 (ARCHITECTURE item 61): alignment and effects are evaluated by the Rust runtime (§6.3, §7, §8), an effect may carry its config (§5, §10), and the run configuration binds clocks and services (§9.3, service-protocol.md). Version 2 replaces version 1 (plan-v1.md), which remains the reference for kernels and groups. It defines no new semantics: outcomes are those of semantics-v0.1.md, as the TypeScript reference evaluator computes them, and this document records how the reference computes them wherever two implementations could otherwise differ.

1. What changed from version 1

2. Numbers

As plan-v1 §2: a PlanNumber is a JSON number when finite and not −0, else { "special": "-0" | "NaN" | "Infinity" | "-Infinity" }. In equations it is used for number literals (num), which can be Infinity (an overlong literal). Every other number in an equation is a plain JSON integer: origins, offsets, ranges.

3. The top level

As plan-v1 §3, with planVersion 2. The fields are planVersion, engineVersion, mathLibrary ("fdlibm-5.3+exp1+msun-tanh", decision A5), sourceHash, dimensions, values, groups, clocks and services.

Nesting (owner decision "A2c part 2 accepted", 2; ARCHITECTURE item 66). Equations and kernel expressions nest one JSON level per operator, so a plan nests as deeply as its program. A reader may refuse a plan nested more than 2,000 levels of arrays and objects (the document counting as level 1), a resource limit like the depth limit (§6.7), never an outcome; the Rust runtime does, before reading anything else: "the plan nests more than 2,000 levels deep (line L, column C): …". A value's equation starts at level 4 and its kernel's expression at level 5, so a chain such as #t + 1 + … + 1 may have about 1,990 operators. The compiler accepts about 1,500 on Node's default stack, and a few hundred when each operator is parenthesised.

4. Dimensions

As plan-v1 §4: name, type (int or label), range (inclusive, optional), origin (integer dimensions: the range minimum, else 0), and clock (the clock a clocked dimension is on, C1).

5. Values

{ "name": "field", "kind": "source", "dims": ["h", "x", "y"] }
{ "name": "heat", "kind": "value", "dims": ["time", "x", "y"], "version": "…", "equation": { … }, "kernel": { … } }
{ "name": "warmest", "kind": "value", "dims": ["t"], "version": "…", "equation": { … }, "reason": "not supported by dense kernels: a text constant" }
{ "name": "score", "kind": "effect", "dims": ["time", "x", "y"], "version": "…", "service": "anomaly_detector",
  "args": [ … ], "policy": { "deadlineMs": 20, "retry": { "transient": ["overloaded"], "maxAttempts": 2 } }, "reason": "an effect: calls external service anomaly_detector" }
{ "name": "score", "kind": "effect", …, "service": "claude_scorer", "config": { "prompt": "unusual-v2", "maxTokens": 50 }, … }

6. Equations

An equation is a tree of nodes, each with an op. A node is evaluated at a context: an assignment of coordinates to dimensions. A value's equation is evaluated at the value's own context (its projected point); next, first, fby, shifts and aggregates over ranges evaluate their operands at changed contexts. In a context, an integer coordinate is a double (it can be −0, as -#x gives at x = 0, or beyond the safe integers after next); a label coordinate is text. The reference's interpreter is evaluate in src/syntax/compile.ts; Rust's is runtime/src/eduction/eval.rs.

6.1 Outcomes and values

An outcome is a value, Undefined or an Error (semantics §1). A value is a number (an IEEE double), a boolean, text, or a list (what a neighbourhood gives, §6.3). Two values are equal (==) when they are both numbers and equal as doubles (−0 equals 0; NaN is unequal to itself), both booleans and equal, or both text and equal; values of different types are unequal. Lists are not comparable (semantics §2, owner decision 2026-10-04): == and != with a list operand fail (§6.3).

6.2 Demand and projection

A reference demands its target at the current context projected onto the target's dimensions (semantics §2):

The outcome of a reference is the target's outcome at that point, except that an Error becomes an upstream Error naming the point:

{ "kind": "upstream", "from": { "name": "w1", "context": { "step": 199 } }, "cause": { … } }

context lists the target's dimensions in declared order. A chain of references gives a chain of upstream Errors ending at the Error's origin. This is the in-memory form, one link per point; as JSON a chain is written compressed (§9.1).

A point is computed at most once per query (memoised, owner decision A); a demand for a point already being computed in the same descent is a demand cycle (§6.7).

6.3 Nodes

"Strict" means: every operand is evaluated, in order, and then the first Error among them (in operand order) is the outcome, else the first Undefined, else the operation applies (Error over Undefined, semantics §2). "Fails with M" means the outcome is { "kind": "error", "error": { "kind": "failed", "message": M } }. "Finite(op, x)" means the value x if it is finite, else the non-finite Error (T4) { "kind": "non-finite", "op": op, "result": String(x) }, where String(x) is "Infinity", "-Infinity" or "NaN".

opfieldsoutcome
numvalue (PlanNumber)the number (Infinity is a value: T4 applies to operators, not literals)
textvaluethe text
boolvaluethe boolean
undefinedUndefined
coorddimthe context's coordinate of dim: a number for an integer dimension (possibly −0), text for a label dimension
refname, at (optional: [{dim, expr}])without at, the target demanded at the context (§6.2). With at, the coordinates are evaluated first (§6.4, shifts), then the target is demanded at the context overridden by them
shiftexpr, atexpr evaluated at the context overridden by at (§6.4)
nextdim, argarg at the context with dim increased by 1 (a double addition)
firstdim, origin, argarg at the context with dim set to origin
fbydim, origin, left, rightwith c the coordinate of dim: c < origin, Undefined; c = origin, left at the same context; c > origin, right at the context with dim = c − 1. Only the selected operand is evaluated
ifcond, then, elsecond first; a non-value is the outcome; a non-boolean fails with if needs a boolean condition; else only the branch taken is evaluated (semantics §2, MDP p. 90)
binary and, orleft, rightleft first; a non-value is the outcome; a non-boolean fails with and needs booleans (or needs booleans). false and … is false, true or … is true, without evaluating right. Otherwise right: a non-value is the outcome; a non-boolean fails as above; else right's value
binary ==, !=strict; then, if either operand is a list, fails with == needs values, not lists (!= needs values, not lists); else equality of §6.1 (values of different types are never equal)
binary <, <=, >, >=strict; both numbers, else fails with < needs numbers (the operator's symbol); IEEE comparison
binary +, -, *, /strict; both numbers, else fails with + needs numbers; / by 0 or −0 fails with division by zero (checked first); else Finite(+, a + b), and so on
unary -arga non-value is the outcome; a non-number fails with - needs a number; else Finite(unary -, −a)
unary notarga non-value is the outcome; a non-boolean fails with not needs a boolean
mathfn, argsstrict; every argument a number, else fails with exp needs a number (two arguments: min needs numbers); else Finite(fn, result) from mathLibrary (N1, A5); min and max are ECMAScript's (−0 below +0; NaN wins)
neighbourhoodname, offsets (dim, from, to, range)the list of name's outcomes at every combination of offsets (the first offset outermost, each ascending), each coordinate c + k kept only if inside that offset's range (truncation). Every neighbour is demanded (strict); an Error (upstream) or Undefined among them is the outcome; else the list of their values
aggregatefn, argan aggregate of a list (sum(neighbourhood(…))): arg first; a non-value is the outcome; not a list fails with sum needs a list. Then §6.6
reducefn, arg, over (dim, from, to)an aggregate over declared ranges (A1–A5): §6.6
alignname, policyan alignment (C3–C6): §7.1. Never inside a shift or an aggregate (the compiler's checks), so its context is the value's own point

6.4 Shifts

A shift @ {d₁ = e₁, …} (in ref or shift) evaluates every eᵢ at the current context, in order (strict: the first Error, else the first Undefined, is the outcome of the whole node). Then each value is checked in order: for an integer dimension it must be a number that is a safe integer, for a label dimension text; the first that is not fails with coordinate d = V is not a valid int (or label), with V as JSON.stringify writes the value (§6.5). A coordinate can therefore be −0 (from -#x), which coord then returns as −0, and which projection turns into 0.

6.5 Text in messages

Messages print numbers and values exactly as JavaScript does:

6.6 Aggregates

Over a list (aggregate) (a neighbourhood, or a window alignment):

Over declared ranges (reduce) (A1–A5):

  1. Every binding's from and to are evaluated at the current context, in order (strict). Then each binding's bounds, in order, must be numbers that are safe integers, else fails with sum: bound d = V is not a valid int (V as in §6.5).
  2. The number of elements is the product of max(0, to − from + 1) over the bindings, in doubles. If it exceeds the query's limit (maxAggregate, default 1,000,000), the query stops with the program error X at {CONTEXT}: sum(…, d in a..b, …) would combine N elements, over the limit of L (QueryOptions.maxAggregate), before any element is demanded. X is the value whose equation this is, CONTEXT its context as JSON in declared order, N and L as toLocaleString("en").
  3. The elements: if any binding has from > to, none. Otherwise every combination, the first binding outermost, the last fastest, each ascending, with −0 bounds as 0; arg is evaluated at the context overridden by the bound coordinates. Every element is evaluated (strict).
  4. The fold: the first element's number starts the accumulator; each later number is added (sum, mean), or combined with ECMAScript min or max.
  5. The outcome: the first Error among the elements; else Undefined if any element is Undefined; else for count, the number of elements (of any type); else, if an element is not a number, fails with sum needs numbers (the function's name); else for no elements, 0 for sum and Undefined for the others; else Finite(fn, accumulator), divided by the number of elements for mean.

6.7 Program errors and the driver

A program error stops the query; it is never an outcome (semantics §2). Besides projection (§6.2) and the aggregate limit (§6.6), the reference raises:

A point is written as the reference's id: the value's name followed by its context key, the dimensions sorted by UTF-16 code units, each as d=iN; for an integer and d=s"…"; for a label (the label as JSON), a dimension name containing =, ; or " written as JSON.

Logical depth is what the limit counts, and the reference defines it by its driver (ARCHITECTURE items 18 and 47), which a backend must follow to give the same program errors:

Which points are evaluated per element, and so which descents can reach the limit, also depends on the runtime's use of kernels (§8). The depth limit is a resource limit (semantics §2, owner decision 2026-10-04): whether a query stops with it may depend on runtime choices (group levels, single recurrences by levels, retention, suspension); outcomes never do. A backend that follows this section gives the reference's program errors exactly; one that evaluates more by levels may compute deeper histories, and is still correct.

7. Effects and alignment

Effects (kind: "effect") carry their external service, their arguments (args, equations evaluated strictly in order at the effect's context; the call's input is the list of their values), their call-site policy (policy: deadlineMs, retry with transient error codes and maxAttempts, reuseAcrossQueries), as EffectDefinition.policy (semantics §5, R4), and optionally their config (§5). An align node carries its target and policy (exact, asof, hold with optional maxAgeMs, window with widthMs; C4). Their meaning is semantics §3 and §5; clock tables and services are bound outside the plan, by name (C2, §9.3). The reference computes them as follows (Run.align, alignSelect and callEffect in src/evaluate.ts; Rust: runtime/src/eduction/clocks.rs and effects.rs).

Clocks. A dimension's clock names its clock (C1). A clock's table is pinned per query by its version in the query's snapshot (C2). Looking one up is a program error if no table is bound for the clock (no clock table for clock C), if a source has the clock's name (clock C has the same name as a source; sources and clocks are pinned by name in the snapshot) or if the snapshot does not pin it (clock C is not pinned by the query's snapshot); if the table does not have the pinned version, the outcome that needed it is the Error { "kind": "source-version-unavailable", "source": C, "version": V } (not upstream). A tick the table has no timestamp for makes it Undefined. The clocked dimension of a value is the one of its dimensions on a clock: more than one is a program error (X has more than one clocked dimension: a, b), as is a clock on a label dimension.

7.1 Alignment

align(T, policy) in the equation of X, at X's point P:

  1. X's clocked dimension d on clock A, and T's clocked dimension e on clock B; if either has none, a program error align(T) from X needs a clocked dimension on both sides.
  2. The timestamp of P's tick on A (above): an Error or Undefined there is the outcome.
  3. B's pinned table (above); latest is the latest tick of B whose timestamp is at most that timestamp τ.
  4. Each selected tick k is demanded as T at P's context with d removed and e = k (projected onto T's dimensions); an Error becomes an upstream Error naming T's point (§6.2).
    • exact: latest, if its timestamp is τ; else Undefined.
    • asof: latest; Undefined if there is none.
    • hold (maxAgeMs m, optional): from latest down to tick 0, stopping before a tick whose timestamp is below τ − m; the first outcome that is not Undefined (a value or an Error: Errors are never skipped); Undefined if none.
    • window (widthMs w): every tick from latest down to tick 0 whose timestamp is above τ − w, demanded in ascending tick order; the first Error among them, else the list of the values (Undefined ticks omitted; possibly empty). Lists are not comparable (§6.1) and their aggregates need numbers (§6.6).

7.2 Effects

An effect E at its point P, when demanded (memoised like a value: §6.2, so one call per query and point):

  1. Arguments in order (strict): the first Error, else the first Undefined, is the outcome, and no call is made. The input is the list of their values.
  2. The service: not bound, a program error no adapter S for effect E.
  3. Start (C5): if E has a clocked dimension, the timestamp of P's tick (an Error or Undefined there is the outcome); otherwise 0. The deadline, if deadlineMs is given, is start + deadlineMs.
  4. Attempts: maxAttempts (default 1; an integer from 1 to 10, the compiler's limit, and a plan with any other is refused) if the service is idempotent, else 1. Attempt n (from 1) starts at the previous attempt's completion. Its answer:
    • a value completing at c: late ({ "kind": "late", "deadlineMs": the deadline, "completedMs": c }) if c is above the deadline, else the value. A service's value is never checked for being finite (T4 applies to operators; a service is external, like a source);
    • an error code completing at c: retried if the code is in transient, n < maxAttempts and c is not above the deadline; otherwise late if c is above the deadline, else { "kind": "effect-failed", "code", "message" }.
  5. No speculation (R4): a call is never made for a point nobody demanded. In the Rust runtime the only eager evaluation is of elements a group kernel did not give (§8), and a call reached there leaves that element to per-element evaluation where it is demanded.

A service's request carries the service's name, config, the input, the point (ref), the attempt and its start; its answer, a value or an error code, and when it completed (service-protocol.md).

8. Kernels and groups (the fast path)

Kernels and groups are as plan-v1 §6–§7. In version 2 they are an optimisation of what the equations already define: an element a kernel gives is exactly the equation's outcome, and a backend may compute any element by eduction instead.

How the reference uses them, which a backend should follow to give the same program errors (§6.7):

9. Outside the plan: the boundary of the Rust runtime

These formats are not part of the plan; they stand in for connectors (F2). Services use the protocol of service-protocol.md.

9.1 Outcomes as JSON

runtime/src/outcome.rs and src/outcome-json.ts (outcomeToJson, outcomeFromJson) write and read the same form: the reference's Outcome, field for field and in its field order, with every number inside a value that JSON cannot carry exactly (−0, NaN, ±Infinity) as a PlanNumber object. Error payloads are written as the reference constructs them: failed (message), non-finite (op, result), source-version-unavailable (source, version), effect-failed (code, message), late (deadlineMs, completedMs). Two outcomes are the same exactly when their JSON texts are the same.

Upstream chains are compressed (owner decision, 2026-10-04). In memory a chain is linked, one upstream link per point (§6.2), and can be as long as a history; nested JSON of that form overflows parsers (Node's JSON.stringify near 10,000 links, serde_json at 128 levels by default). As JSON an Error whose payload is upstream is written flat:

{ "kind": "upstream", "length": 12,
  "path": [ { "name": "err", "context": { "step": 209, "h": 0, "x": 0, "y": 0 } },
            { "name": "w1", "context": { "step": 209 }, "count": 10, "last": { "step": 200 } },
            { "name": "g1", "context": { "step": 199 } } ],
  "origin": { "kind": "non-finite", "op": "mean", "result": "Infinity" } }

The compression is a format, not a semantic change: outcomes are the same, errorOrigin and traceOrigin follow the linked chain in memory, and the playground and viewer display chains as before. A 200,000-link chain is written and read in both engines (tested).

9.2 Source data and jobs

Source data (elucid8-source-data/1, runtime/src/eduction/sources.rs): per source, its points as [coordinates in declared order, outcome]. A demanded point the data does not hold is a program error (source S has no data at {…}); in a kernel's array it is not ok.

Source versions (stage 3): a source may instead list versions, "field": { "versions": { "a": { "points": […] }, "b": { "points": […] } } }, read as the reference reads them: each query pins one version in its snapshot (§9.3); an unpinned source is the program error source S is not pinned by the query's snapshot where it is read; a pinned version the data does not list gives the Error source-version-unavailable; and unchangedAt (R3) is exact: a point is unchanged between two versions when both list it with the same outcome (numbers as Object.is compares them, anything else by its JSON form). "unchangedAt": false declares a source without one. The version-less form above is one version that every pin names, and needs no pin.

A job (runtime/src/eduction/job.rs): maxDepth, maxAggregate, groups, singleRecurrences (single recurrences by levels; off by default, §8), and queries, each a value and its points. Each query is one run with one memo; its points are demanded in order, as the reference's queryTile demands a tile's elements. elucid8-runtime query <plan.json> <sources.json> <job.json> prints each query's outcomes, or its program error, with timings and counts.

Caches across a job's queries (stage 3): "pointCache": true keeps a point cache (R0–R4: entries with dependency records, usable while R3 holds; a service result only under reuse successes, served as a replay without a call; a failure only in its own query), and "sourceCache": true a source cache by pinned version. Queries are run in order, so a job is a sequence of queries with snapshots changing between them. With a point cache the queries are evaluated per element: "groups": true is refused (group levels with a point cache are not yet built). Each query's counts include computed (values evaluated to completion, the reference's CostMeter.evalCount), sourceReads (read afresh), cacheHits, cacheReplays and sourceCacheHits.

Retention (stage 3): "retain", as QueryOptions.retain: "auto" (default), "all", "static" or "retire", with "autoRetainAt" and "sweepFloor" (65,536 each). Static counts come from the plan's usage (§10); a plan without it keeps everything under static, and only retires under retire and auto. Retention is off with a point cache, as in the reference. "checkRetention": true keeps the outcome of every point a count drops, and counts (recomputed) and compares (recomputedDifferently) any that is computed again: never, unless a count is too low. The counts per query add maxHeld, dropped, droppedAtSwitch, switches, sweeps, retired and retiredRecomputed.

Tiles (stage 3): a query with "tile": true is the reference's queryTile: a group member's group is evaluated over all its points before they are demanded (seedGroup), and retention knows every point as demanded from the start.

Root nodes (stage 3, minimal traces): each query's roots, one per point, as the reference's untraced root (rootNode): ref (the demanded point, projected) and how (computed, memoized, called, cache-hit, replayed, vectorised, source-read, source-memo, source-cache); for an Error, origin as errorOrigin gives it: hops (links in its upstream chain), origin (the point where it arose, or null at the point itself) and cause (the error there, in the form of §9.1). "trace": true (a full trace) is refused.

9.3 Clocks, the snapshot and services (stage 2)

10. Versioning

As plan-v1 §8. Version 2 was required because a version-1 reader would misread a version-2 plan (it would treat values without kernels as uncomputable). Information-only fields may still be added without a new version.

The usage sites (stage 3) did not need a new version. The plan may carry usage: readers (per value and effect, exact: its sites are exactly what it demands; release: it has a usage footprint, so its completions release what it read) and sites, every usage site in program order as src/usage.ts defines them (reader, target, at per target dimension: { "lin": { "k", "terms" } }, { "label" } or "top"; guards; optional within and bound; −0 written as 0). It is information only: a reader that ignores it keeps every point, which changes memory, never an outcome.

An effect's config (stage 2) did not need a new version. It changes what a call is sent, so a reader that ignored it could misread a plan that has one; but no version-2 reader before stage 2 evaluates effects (the stage-1 runtime refuses them with a program error when demanded), so none can misread it, and the compiler writes it for no program. Any later field a version-2 reader that evaluates effects could misread needs version 3.

11. Questions found by the Rust implementation (decided 2026-10-04)

The second implementation exposed four places where the reference's behaviour was accidental. The owner decided all four (decision "A2b stage 1 accepted; four semantic clarifications"; ARCHITECTURE item 60), and both engines implement the decisions.

  1. List equality was identity. == on two lists was true only when they were the same list object: n == n for a list-valued n was true (the second demand served from the memo), but neighbourhood(f, …) == neighbourhood(f, …) was false, and so would n == n be if retention dropped and recomputed n between the two demands. Decided: lists are not comparable; == and != with a list operand fail (§6.1, §6.3; semantics §2).
  2. The depth limit depends on runtime policy. Whether a deep, well-founded history reaches maxDepth depends on whether it is evaluated per element or by levels. In the reference itself: with a @ (step, x) = 1 fby.step (mean(neighbourhood(b, x in -1..1, boundary truncate)) * 0.5 + #x), b @ (step, x) = a + 1, x in 0..9 and maxDepth 1,000, a at step 2,000, x = 1 is a value (3.1495427649273804) with group levels (the default) and the depth-limit program error with groups: false. Decided: the depth limit is a resource limit, like memory: whether a ProgramError from it occurs may depend on runtime choices (group levels, retention, suspension); outcomes never do (§6.7; semantics §2). No code changed; the Rust runtime still follows the reference's choices (§8), so it gives the same result in each mode.
  3. Neighbourhood aggregates coerced non-numbers as JavaScript does (Q9): sum of text concatenated ("0ab", reported as a non-finite Error whose result was that text), mean and min converted text with JavaScript's number syntax (" 0x1F " was 31), booleans counted as 0 or 1. Decided (Q9 closed): a non-number element is an Error, as for aggregates over ranges; count counts any values (§6.6; semantics §2). Both engines' coercions are removed.
  4. Long upstream chains could not be serialised. Node's JSON.stringify overflowed near 10,000 links, and serde_json reads at most 128 levels of nesting. Decided: chains are serialised compressed (§9.1).