Changelog¶

All notable changes to acid will be recorded here. The format follows Keep a Changelog; the project follows Semantic Versioning within the constraints of its alpha posture (see CLAUDE.md).

[0.5.0a1] — 2026-06-15¶

Added¶

acid list — the local twin of acid search. Lists the catalogs in the catalog path ($ACID_PATH → config path → built-in default ~/datasets) — the ones acid open / acid query resolve a bare name against — instead of the download path. Same discovery engine, transports (local / ssh:// / http(s)://), flags (PATTERN, --cache, --timeout, --insecure, --no-color), and output contract as acid search (aligned table on a TTY, name⇥margins⇥root⇥marker TSV when piped, margin radii, namespace/child names, shadowing); the only divergence is the shadowing footnote, which names acid open (which resolves the first match) rather than acid download. So acid search answers what can I download and acid list answers what's already here that I can open.

Changed¶

acid.list_catalogs() / Connection.list_catalogs() now return the same results as the acid list CLI. Previously they did a shallow local-only os.listdir walk that returned a list[str] of basenames — which silently dropped ssh:// / http(s):// roots in ACID_PATH, missed catalogs under namespace directories, and listed margin-cache siblings (e.g. object_2arcsec) as if they were catalogs. They now go through the same discovery engine as acid list and return a list[acid.tools.download.CatalogInfo] — rows (name, margins_arcsec, root, shadowed) — over local / ssh / http roots, with namespace/child names, margin-cache siblings attributed to their parent, and cross-root shadowing flagged. Explicitly registered catalogs (YAML / register_catalog) are still included (a superset of the CLI). This is a breaking change: callers using the name-membership idiom ("gaia" in acid.list_catalogs()) must use {c.name for c in acid.list_catalogs()}.
Renamed the catalog-discovery row type FoundCatalog → CatalogInfo. It's now returned by both acid list / acid.list_catalogs() (catalog path) and acid search / acid.archives.search() (download path), so the download-flavored FoundCatalog name no longer fit. Same fields (name, margins_arcsec, root, shadowed); import as from acid.tools.download import CatalogInfo.

[0.4.0a4] — 2026-06-14¶

Fixed¶

acid search no longer blocks or hangs on an unreachable download root. The default download path includes an ssh:// root, so a user who hasn't set up that host (e.g. slacd not in their ~/.ssh/config) previously had acid search (and acid.archives.search()) either sleep ~19 s through the download path's 6-attempt MaxStartups backoff or — for a resolvable but firewalled host — hang indefinitely (no connect timeout), and in both cases the failure aborted the whole search, discarding catalogs already found at the other roots. Now:
ssh runs with BatchMode=yes (fail instead of blocking on a password / passphrase prompt) and ConnectTimeout (bound the TCP connect), so a dead host fails in seconds, not minutes;
discovery probes use a single attempt (retries=1) instead of the download hot loop's backoff schedule;
a download path is treated as a search path — a root that can't be crawled is skipped (reported via search_downloads(on_error=…) / a UserWarning, and as a ! skipped <root> line by the CLI), never fatal, so the other roots' catalogs still come back;
connection readiness is now a positive handshake rather than a 150 ms timing heuristic: the remote server emits a one-byte ready marker once ssh has authenticated and the remote python has started, and SshSession blocks on it. A failed connect / auth / host-key check closes the pipe (EOF), so the failure surfaces at connect time (__init__) instead of being deferred to the first later read. This also fixes a latent misclassification where a remote key rejection was reported as "catalog not found" (absent) instead of "unreachable" (error) by SshFetcher.fetch_text_ex, and makes the retry/backoff actually apply to auth/slow-kex failures (the old heuristic returned "connected" before they happened);
the skip is classified into an actionable hint (ssh_failure_hint): a rejected key prints → the host rejected authentication — set up key-based SSH access (ssh-copy-id <host> / ssh-add), with distinct hints for an untrusted host key, an unresolvable name, a connect timeout, and a refused connection. The hint is derived from the ssh stderr (every ssh failure exits 255), and is detected at both the connect (__init__) and first-read (scan_dir) sites, since a remote auth rejection surfaces only after the connection looks established.

[0.4.0a3] — 2026-06-14¶

Changed¶

The HATS spatial index (_healpix_29) is no longer surfaced in query output. It's an internal HATS format detail, not data the user asked for, so it no longer appears in SELECT * / a.*, Catalog.columns / Result.column_names, or any materialized output (to_polars / to_arrow / to_astropy / to_pandas / print / show):
the root catalog's index is hidden but kept physically — an explicit select("…, _healpix_29") / WHERE _healpix_29 … resolves it, save / --output hats still write it to disk (the index is required for a valid, re-queryable catalog), and acid inspect still reveals it in the on-disk schema;
a joined right table's index (_healpix_29_<alias>) is dropped entirely after a join/crossmatch — it's meaningless in the result (which is partitioned by the root's index) and is neither selectable nor written.
print(result) now renders the result as a Polars DataFrame (via Result.__str__) — its shape: (rows, cols) header plus Polars's own head/tail row truncation — instead of the previous fixed-width first-20-rows ASCII table. Result.show(n) (and the acid query CLI) keep the fixed-width renderer. Note print materializes the full result; use show(n) for a bounded peek at a large on-disk result.

[0.4.0a2] — 2026-06-14¶

Removed¶

Result.save() is gone (breaking, no shim). A Result is already-materialized data — it has left the partitioned system — and its in-memory branch could only write a degenerate single-partition catalog (all rows mislabelled into pixel (0,0), no spatial index). HATS output is a stays-in-the-system operation written from the lazy handle that can stream it correctly: Catalog.save(path, name=...), or acid.sql.query(query, output="dir/") / acid query --output dir/ for the SQL surface. Result keeps export(path) (one flat file) and the to_* converters. This sharpens the terminal split — save is Catalog-only; Result only leaves the system (amends the API-DESIGN V4/V5 Catalog/Result symmetry deliberately).

Fixed¶

A saved catalog now always keeps its HEALPix spatial index, even when a projection dropped it. db.open("x").select("source_id, designation").save(...) used to write a partition-only catalog with no _healpix_29 column; re-opening it and running a query under many workers crashed (TypeError: unsupported operand type(s) for -: 'NoneType' and 'int') when the work-tuple planner tried to sub-partition a catalog it couldn't row-restrict. The HATS spatial index is now treated as a required column of a HATS catalog: a query whose result stays in the system (Catalog.save / acid query --output hats) retains _healpix_29 in its terminal projection regardless of the select, so the written catalog stays valid and re-queryable. The leaving-the-system terminals (export, to_polars / to_arrow / to_astropy / to_pandas, display) are unchanged — a flat extract omits the index unless you select it explicitly.
Catalog.save / acid query --output hats now declare hats_col_healpix / hats_col_healpix_order in the output catalog's properties, so re-registration is HATS-spec-compliant and no longer relies on the reader's _healpix_<order> column-name auto-detection.
The catalog registry no longer stores the literal string 'None' for hpix_col when a catalog has no HEALPix column (it now stores None).

[0.4.0a1] — 2026-06-13¶

Changed¶

Result now mirrors Catalog's terminal verbs (API-DESIGN V4/V5 — one concept, one name, on both nouns; breaking, no shims): Result.arrow() → Result.to_arrow(); Result.write(path, format=) → Result.export(path, format=) (same contract as Catalog.export, including the ValidationError-pointing-at-save destination checks); Result.write_parquet(layout="hats") → Result.save(path) (a HATS tree, without Catalog.save's name registration). The per-format writers (write_parquet(layout="single") / write_csv / write_fits) and the df() pandas alias are removed — use export(path) (format by extension) and to_pandas().
add_catalog → register_catalog (Connection method + module-level delegate): one prefix for "make a name known to the connection" (register_catalog / register_file / register_moc). acid.register_file gained the missing module-level delegate (API-DESIGN O3).
agg.bool_and / agg.bool_or → agg.all / agg.any — the numpy idiom, like mean/std/var (API-DESIGN A1). SQL-string BOOL_AND(...)/BOOL_OR(...) in db.sql are unchanged.
acid query --out → --output — one flag spelling for one concept across subcommands, mirroring the Connection.sql(output=) kwarg (API-DESIGN P2). The abbreviation --out still parses (argparse prefix matching).
Connection(workers=None) is the new default spelling of "resolve for me" (was workers="auto"; "auto" is still accepted) — None is the one inherit sentinel across the API (API-DESIGN S2).
Result.to_arrow() no longer imports DuckDB for disk-backed results — the partition union now streams through the same manifest-driven PyArrow path as Result.batches(). DuckDB is again strictly a test-only dependency ("one engine").
acid search / acid.archives.search cache control is one keyword: cache="use"|"refresh"|"off" (CLI --cache) replaces the overlapping refresh=/use_cache= boolean pair and the --refresh/--no-cache flags; archives.search also gained the timeout=/insecure=/workers= crawl knobs the CLI already exposed (API-DESIGN P3 — no CLI-only capability).
build_margin_cache now fails loudly on the first partition failure (parallel scan): the run aborts with the failing partition named instead of printing per-partition FAILED lines and exiting 0 over an incomplete margin cache (API-DESIGN E1/E5).

Added¶

Work-tuple subdivision gate (autosize, decision #20) — a query over a handful of partitions (e.g. a small cone) used to coalesce to a handful of work tuples and leave most workers idle, because the parallelism floor is clamped to physical-partition granularity. When the layout yields fewer than workers/2 tuples, the driver now re-enumerates with the floor allowed below partition granularity, aiming for ~workers tuples so idle cores get fed. Fires only in the under-subscribed regime (the billion-row/thousands-of-partitions path is byte-for-byte unchanged); the cost is duplicated central+margin reads per sub-cell (the OS page cache keeps the physical I/O ~1×).
acid hats build-margin --tmpdir (and build_margin_cache(tmpdir=...)) — redirect the accumulator's spill scratch to fast local storage when --output lives on a slow networked filesystem (a unique subdir is created and removed; the base must already exist, fail-loud).
API-DESIGN.md — the API design language (new root authoritative doc). The prescriptive principles governing the public Python surface — the two-noun object model, the composition / materialization / introspection verb taxonomy, fail-loud error rules, astronomer idiom + explicit units, signature and config-knob conventions — plus the CLI-as-projection corollary. Normative, not descriptive: new or changed public API must pass its §13 checklist (citable rule IDs) or amend the rule in the same change. Added to CLAUDE.md's trusted-docs list.
acid search [pattern] — discover catalogs available to download. Enumerates the catalogs under the download path over every transport (local / ssh:// / http(s)://) and prints one line per catalog with the margin-cache radii available for it (gaia_dr3 margins: 10, 300 arcsec). Catalogs nested under namespace directories surface as namespace/child (wise/allwise); HATS collections are presented as plain catalogs (the "collection" concept is never shown). Roots are merged in download-resolution order: a same-named catalog at a later root is still listed but flagged (a trailing * / a shadowed TSV column), since acid download resolves first-wins. Output is an aligned, colored table on a TTY (with a live spinner while crawling) and parseable TSV when piped. Remote listings are cached under $XDG_CACHE_HOME/acid/downloads for ~1h (--refresh re-crawls, --no-cache bypasses); local roots are always live. The Python sibling is acid.archives.search(pattern=None, *, cache="use", timeout=300.0, insecure=False, workers=16) → list[FoundCatalog] (the same knobs as the CLI flags; cache is "use" / "refresh" / "off").
acid download accepts nested namespace/child names. A name like wise/allwise (as shown by acid search) now resolves against the download path and lands locally under its leaf (<ACID_PATH>/allwise). Only a leading ./ / / / ~ path or a URL is treated as an explicit source now (an internal / no longer forces verbatim use — prefix ./ to copy a local relative dir).
The built-in default download path gained a second root. It is now https://data.lsdb.io/hats/ then ssh://slacd/sdf/home/m/mjuric/datasets, searched first-wins. (default_download_path() now returns list[str].)
ACID_QUERY_LOG=<file.csv> — a per-tuple query execution log (debug aid; acid/engine/querylog.py). When set, every executed work tuple appends one CSV row — timestamp,worker_id,norder,npix,nrows — with the originating query written as #-prefixed comment lines before the header (SQL verbatim; the fluent path logs a rendered pipeline summary, now also attached to OpPlan.query so fluent error messages and manifests carry a description). The file is truncated per query and written only by the parent process (workers stamp worker_id/exec_time onto the PartitionResult they already return; the single-threaded result collector writes the row), so it is correct on local disk, NFS, and Weka alike — no concurrent-O_APPEND race. Read once at import; a no-op when unset. In-process (workers<=1) rows carry worker_id=-1.
Catalog.export(path, *, format=None, progress=None) -> Path — a flat-file terminal verb, the leaves-the-system counterpart to save's stays-in-the-system HATS write (spec provenance docs/archive/EXPORT-API.md). Sugar over execute().write(...): it gathers the full result in RAM and writes one CSV / parquet / FITS file (format by extension or format=), returning the written Path. A no-extension / unknown-extension / format="hats" call is a ValidationError pointing at save — export never writes HATS.
save() now joins the catalog library for a bare name. A bare destination (no /, e.g. save("gxt")) lands under the first writable local ACID_PATH root, so the name is durably re-openable in a later session (acid.open("gxt") / ... FROM gxt) with no path bookkeeping — the same model acid download <name> uses. Explicit paths (./x, /abs/x, ~/x) stay verbatim/cwd-relative. A bare name shadowed by an existing catalog earlier on ACID_PATH is a hard RegistryError (overwrite=True does not override it). acid query --out <bare name> (hats) shares the rule. The destination machinery lives in acid/api/_dest.py (shared by Catalog.save, acid query, and acid download).
RAM-budget work-tuple sizing (autosize) — the new default enumeration (key decision #20; spec provenance docs/archive/WORK-AUTOSIZE.md). Work tuples are no longer dictated by the on-disk HATS layout: a byte-budget quadtree over point_map.fits row-count maps (engine/autosize.py) picks the coarsest HEALPix cells whose estimated resident bytes fit ram_budget / workers — coalescing small physical partitions into one tuple (concatenated scans) and splitting oversized ones (cursor row filters), with count-based pruning (cells without root rows or INNER partners within the locality band emit nothing) and a two-level threshold (the parallelism floor never splits below physical partition granularity; only the RAM ceiling does — the OOM protection). Verified result-identical to the legacy enumeration across a 10-shape oracle sweep at three granularities and 23 exact-equality real-data validation tests; the test suite's wall clock halved under it.
ram_budget — the one new knob: acid.conf key, ACID_RAM_BUDGET, acid.init(ram_budget="64GB"), CLI --ram-budget. Bytes or human sizes (64GB, 512MiB, 32g); default 0.25 × available RAM (cgroup-aware).
ACID_WORK_AUTOSIZE=0 selects the legacy layout-driven enumeration (A/B + verification switch; slated for deletion once autosize is validated at scale).
Every acid HATS output now writes point_map.fits (exact counts when the healpix column survives the projection; partition-painted counts otherwise), so outputs can always be re-registered and queried.

Fixed¶

A catalog-resolution probe error no longer masquerades as "not found." Resolving a bare catalog name probes each download/catalog-path root; a probe that errored (an unreachable SSH host, a connection timeout, a 5xx) was previously swallowed and indistinguishable from "the catalog isn't here," so a catalog that genuinely lives on a momentarily-unreachable root could be reported as a flat "not found." The fetcher probe is now tri-state (fetch_text_ex → FetchResult(text, error)): cleanly absent (404/403/410, missing local file, connected-but-no-file SSH) is distinguished from a transport failure, and resolution reports the per-root outcome instead of flattening an unreachable root to "absent."

Changed¶

Friendlier "catalog not found" errors across acid download, acid inspect, and acid query / acid.open. The error now shows a per-root trail (which roots had no match, and which were unreachable — with the reason), a did you mean '<closest>'? suggestion drawn from already-cached/local listings (never a fresh crawl), and an actionable next step (acid search to find a downloadable catalog, acid download <name> to fetch one). For query/open, a name that's available to download is called out as such.
acid download over SSH no longer shells out to rsync. The whole-file SSH download path now streams over the existing SshSession transport (the ssh subprocess, so ~/.ssh/config aliases / ProxyJump are honored — unlike paramiko/fsspec), the same backend the column-subset path already used. Three consequences: (1) the download progress bar fills smoothly within each file (byte-level, with a real ETA) instead of jumping one whole file at a time; (2) rsync is no longer required on the client for SSH downloads; (3) each transfer streams to a .tmp sidecar, verifies the full byte count landed, then atomically renames into place — so an interrupted transfer (Ctrl-C, dropped connection) never leaves a passing-but-truncated partition under its final name and a re-run cleanly re-fetches it (closing the rsync --partial + size-only-check hole that let a half-downloaded catalog look complete and fail later at metadata rebuild, decision #13). RsyncFetcher is removed; make_fetcher returns the new SshFetcher for ssh:// sources.
SQL-string entry points moved under the acid.sql submodule (alpha rename, no shim). acid.sql(query) is now acid.sql.query(query); acid.validate / acid.explain are now acid.sql.validate / acid.sql.explain. acid.sql is a real module, not a callable — the top-level acid.sql(...) / acid.validate(...) / acid.explain(...) functions are gone. The fluent Catalog API and the Connection.sql / .validate / .explain methods are unchanged.
Connection.map_partitions_sql is now private (_map_partitions_sql). The phase-1-only inspection hook is an internal power-user/debug tool, not part of the public surface; the top-level acid.map_partitions_sql delegate is removed with no replacement.
save() rejects a single-file extension. save("out.csv") (any recognized flat-file extension) is now a ValidationError pointing at .export(...), instead of silently writing a HATS directory literally named out.csv. Pass a trailing slash (save("out.csv/")) to force a HATS tree genuinely named that. acid query --out x.csv --format hats (the explicit-conflict case) is the aligned CLI error.
copartitioned= → localized= on group_by() (alpha rename, no shim): the assertion and the equi-join contract are the same spatial statement — rows sharing a key are localized to within the margin-cache radius — and now share one word.
Equi joins assert locality and require the RHS to carry _healpix_29 and a declared margin cache (radius 0 = exact-pixel); position-less lookup tables use the in-memory broadcast join. Margin completeness validation (_band_budgets) now counts equi edges at their margin radius for deeper right-subtree leaves.
point_map.fits (with real row counts) is required for every HATS catalog participating in a query under autosize; a missing map is a clear ValidationError naming the catalog.
Plan validation (validate_ops) runs at compile time in both frontends (the lowering re-validates as a backstop), so structural/contract errors surface as ValidationError at composition, not worker-side ExecutionError.
WorkTuple is now (n_cur, p_cur, leaf_scan) with the root folded in as slot 0 — one LeafScan contract for every leaf.
acid download text UI reworked to the CLI design language (cli_text_ui_design_language.html). The command renders a calm, scannable pipeline: a one-time ACID hello banner, one primary progress indicator at a time (an indeterminate ⠋ braille spinner, or a ▰▱ progress bar once a denominator is known), aligned ✓ step lines ([state] [object ~28 cols] [metric] [detail]), semantic color, and a final aligned Summary block (partitions / output size / elapsed / throughput). A partial download renders a ✗ reason+next-action error block before failing hard (downloads still never exit 0 with a half-built catalog). Pretty on a TTY; durable, parseable committed lines when piped (no animation); ACID_PROGRESS=off silences it. New flags: --quiet (only the summary), --verbose (per-item detail), and --progress {auto,on,off,plain} (mirrors acid query). The download bar advances by the fraction of the current file streamed — not one jump per completed file — so a large partition file fills smoothly (HTTP whole-file via a chunked read + Content-Length; SSH column subsets via per-chunk bulk_read progress); each file still contributes exactly one unit (retry-safe), so the k/N files readout stays exact. The HTTP column-subset path (pyarrow-driven Range reads, no byte hook) keeps one advance per completed file. Driven by a reusable PipelineReporter / Step in acid.io.progress — a sibling of the query path's RichReporter, with spec-exact glyphs (the two frontends deliberately don't share a look).

[0.3.0a1] — 2026-06-10¶

Added¶

Module-level API (singleton-by-default). A headline surface mirroring Ray / DuckDB / Polars — no with-block teardown per use:

import acid
acid.init("./data", workers=8)          # optional — first acid.open() lazy-inits
df  = acid.open("gaia").head(100).to_polars()
df2 = acid.sql("SELECT ... FROM a JOIN b ON XMATCH(...)").df()
acid.shutdown()                          # optional — atexit handles it

init() is fingerprint-matched (same config → no-op; a different config → ConfigError unless reuse_existing=True). Module-level open/sql/map_partitions_sql/add_catalog/register_moc/in_cone/ list_catalogs/validate/explain/status all delegate to one shared default Connection (lazy-built on first use, torn down at exit). acid.configure(progress=...) sets process-wide display defaults without rebuilding the pool. acid.Connection(...) is the explicit-isolation escape hatch (two simultaneous connections / two configs in one process); use it as a context manager. (ACID-MODULE-API.md.)

Catalog.join(<frame>, on=...) — broadcast equi-join against an in-memory table. join's operand may now be a polars / pandas / numpy-structured / pyarrow / astropy frame, not just another Catalog — for attaching a flat id→value lookup whose key-matching rows aren't spatially co-partitioned:

labels = pl.DataFrame({"source_id": [...], "class": [...]})
db.open("gaia_dr3").where("phot_g_mean_mag < 18").join(labels, on="source_id")

The frame is spilled once to a broadcast (non-spatial) virtual catalog — one memory-mapped Arrow IPC file, no coordinates / no _healpix_29 / no partitions — and read whole into every work tuple, then Polars-hash-joined on the integer key: partition-local, no Exchange, no reshuffle; INNER and LEFT both supported; the result stays partitioned by the root. A broadcast table is tuple-independent, so the engine short-circuits it to a whole-file scan (_exec_source), bypassing the per-tuple scope machinery. A frame has no position, so it's a .join() RHS only — .crossmatch(<frame>) errors (open it with db.open(frame, ra=, dec=) for a spatial match). Deferred: string keys, nested=True over a frame, and a db.sql/CLI surface for broadcast tables.

acid query --open uses a raw file as a named table. A raw data file (.parquet / .csv / .fits / .arrow / …) can now be referenced by name in a CLI query while every other table still resolves from --db / acid_path — it is spilled once to a virtual catalog and registered. Two forms, both requiring the ra/dec column names (never guessed): positional PATH,RA,DEC (table name = file basename) or named NAME=PATH,ra=RA,dec=DEC. Repeatable:

acid query "SELECT t.id, g.source_id FROM t JOIN gaia_dr3 ON XMATCH(radius_arcsec => 1.0)" \
    --db /data/hats --open t=candidates.csv,ra=RA,dec=DEC

The Python counterpart is Connection.register_file(name, path, ra=, dec=) — the registering sibling of db.open(<file>) (which returns a fluent Catalog but does not register a name, so the SQL escape hatch can't see it).

db.open(...) opens raw files and in-memory frames as catalogs. Beyond a HATS directory, open() now accepts a raw data file (.parquet / .csv / .tsv / .arrow / .feather / .fits / .votable) or an in-memory frame (numpy structured ndarray / pandas / polars / pyarrow / astropy Table) — the "bring my own RA/Dec target list" on-ramp, no offline HATS import:

db.open("targets.parquet", ra="ra", dec="dec").crossmatch(db.open("gaia_dr3"), radius=1*u.arcsec)
db.open(my_ndarray, ra="RA", dec="DEC")        # …pandas / polars / astropy too

The source is spilled once at open() to a single memory-mapped, uncompressed Arrow IPC file under the Connection's scratch dir (cleaned up on close()): ra=/dec= name the coordinate columns and are required (no column-name guessing); _healpix_29 computed; NULL/NaN-coord rows dropped with a warning; the data adaptively partitioned (budget-first, area-first) so a sparse target list doesn't over-read the crossmatch RHS. To the rest of the pipeline it is an ordinary (if coarse) HATS catalog — a virtual catalog, distinguished by TableSpec.backing ("hats" vs "ipc"); a virtual RHS's coverage reuses the general central ∪ ring mechanism with the ring computed on the fly. Usable as crossmatch root or operand, INNER or LEFT. Deferred (see docs/archive/EXTERNAL-SOURCES.md §6): non-spatial id↔id broadcast, a size cost-guard.

Virtual catalogs read per-partition, not whole-file (≈12–90× on the read path). The virtual backing file is now written one Arrow IPC record batch per partition, sorted by _healpix_29, and the engine reads only the batch(es) a cursor / margin band overlaps (TableSpec.batches_for_ranges + pyarrow.ipc.get_batch over the mmap) instead of scanning the whole file and filtering per partition (Arrow IPC has no min/max pushdown). Because the exact _healpix_29 predicate still runs over whatever is read, batch selection is a pure, over-approximating read-pruning — it cannot change results. Measured (synthetic, 2048 partitions): the whole-file path grows O(P×|file|) (13.6 s → 34 s → 85 s at 100k → 1M → 4M rows) while the per-batch path stays ≈1 s — 11× / 29× / 92×. Gated by ACID_VIRTUAL_BATCH_SLICING (default on; set 0/false/no/off for the whole-file path, to A/B the speedup on real catalogs). (docs/archive/EXTERNAL-SOURCES.md §2.2.)
Single-aggregate reduction shortcuts on Catalog — count/sum/mean/min/max/std/var. Each is sugar for a one-aggregate .aggregate(...) and is polymorphic in the preceding group_by: global (no group-by) materializes and returns a bare Python scalar (db.open("a") .where("mag<18").count() → int), while grouped returns the lazy chainable Catalog with the stat in a count / mean_<col> / … column (so a following .where(...) is HAVING). No engine changes — it rides the existing global/grouped aggregate path. count() is COUNT(*), count(col) the non-null count; the rest take one column. (FLUENT-REFERENCE.md §5.2; CLAUDE.md "What acid is".)
Composable (bushy) crossmatches & joins. A crossmatch / join operand may now itself be a full join/crossmatch sub-spine, not just a single catalog — a.crossmatch(b.join(c, on="objectId"), radius=1*u.arcsec), or the deeper-leaf association shape gaia.join(ztf.join(ztf2gaia, on="ztf_id"), on="gaia_id") (the outer equi key may bind any leaf of the operand). It stays one per-partition program — no shuffle — because partitioning is compositional: every sub-expression is HEALPix-partitioned by its leftmost leaf (anchor_source(join.right)). The fluent compiler is reentrant (_compile_spine + a _SpineCtx threading slot ids / aliases / regions); the per-leaf scan contract is slot-keyed (WorkTuple.leaf_scan, one right_rowid(slot) for spatial and equi RHS leaves — the positional xmatch-index machinery is gone); enumeration gathers every leaf. Margin completeness (§7): a leaf whose declared margin-cache radius is below the sum of the spatial match radii on its path to root is a hard error in validate_ops — LEFT and INNER alike (a LEFT-path shortfall fabricates false "no counterpart" rows); this generalizes the single-join margin assumption (it fires on a flat crossmatch whose radius exceeds the right's margin too). v1 covers spatially-bounded RHS sub-spines; the broadcast path for non-spatial id↔id mappings is deferred. CLAUDE.md key decision #17; the design study is docs/archive/COMPOSABLE-JOINS.md.
FLUENT-REFERENCE.md — the canonical definition of the fluent language. A root, authoritative, present-tense reference for the "acid fluent language": concepts + the abstract Op algebra + a formal grammar (verb chain + the embedded-SQL fragment) + the full verb reference + composition/ordering semantics + an exhaustive limitations table + a worked gallery. Aimed at both a third-party re-implementor and an agent writing fluent queries. Added to CLAUDE.md's authoritative-docs list; docs/archive/CATALOG-API.md is now provenance.
Feature A — Python partition functions (UDFs). Catalog.with_columns(name, fn, *, columns=, schema=, mode="numpy") (single- or multi-column) and Catalog.map_partitions(fn, *, schema=, columns=None) (whole-table) run a user callable per partition as a Map operator in the OpPlan, so NumPy / SciPy / Astropy work that doesn't fit a Polars/SQL expression (calibrations, period fits, SED matching, custom stats) is a first-class fluent step with the engine's parallelism and partition layout. columns= and schema= are required (no signature inference) — columns= drives projection pushdown across the UDF boundary (a function reading 3 of 100 columns reads only 3 from parquet, lazy map_batches, verified via .explain()), and the declared schema= makes output names + dtypes known at compile time so .columns / downstream verbs compose with zero engine I/O. NumPy is the default input mode (one .to_numpy() at the worker boundary); mode="polars" passes pl.Series through (for pl.col(...).list.* on nested lists). map_partitions is the table form (collect barrier; receives the whole pl.DataFrame); it is rejected before a later crossmatch/join (it can rewrite ra/dec/_healpix_29). @acid.function (api/function.py) declares a reusable function's metadata (acid_columns/acid_schema/acid_mode) once at the definition site, and on a class turns it into a deferred-construction (_Factory/_Deferred) stateful UDF: the heavy __init__ runs once per worker, the instance lives in a bounded per-process cache keyed by (cls, args) and never rides the pickle (so a 100 MB model can't leak into the payload). cloudpickle is a new hard dependency and is forkserver-preloaded, so closures / lambdas / decorated classes ship to workers; the OpPlan carries the cloudpickled UserFnSpec bytes once per query, never per task. Spec/plan archived at docs/archive/FLUENT-EXTENSIONS-IMPL-PLAN.md; as-built in CLAUDE.md / ARCHITECTURE.md. (The docs/cookbook.md recipe prose is the one deferred remainder — see FLUENT-FUTURE-EXTENSIONS.md.)
Feature B — nested aggregation (nested crossmatch + db.sql LIST). A nested=True (+ order_by=) kwarg on Catalog.crossmatch folds each anchor object's matches into per-object list<T> columns — one row per anchor, the anchor's own columns staying scalar — instead of duplicating the wide LHS for every match (the canonical DP1-style lightcurve shape, which otherwise OOMs on the output shape, not the join work). It compiles to a partition-local LIST Aggregate grouped on the synthetic anchor rowid (Path A), so it runs phase-1-only with no phase-2 reduce — a hard guarantee (needs_global_reduce is False, asserted, never hand-set), the difference between a ~1-hour and a ~10-hour full-sky save. order_by= co-sorts every list column by one key (tie-stable maintain_order=True), so element i is the same match across columns. how="left" unmatched anchors get a one-element [null] list (not []). The general cross-partition form is db.sql: LIST(...) / ARRAY_AGG(...) ... GROUP BY <col> is recognized by the analyzer and decomposed through the shared plan.aggregates recipe with needs_global_reduce = True (Path B) — phase-1 partial implode, phase-2 concat (or, with an in-aggregate ORDER BY, a key-carrying ordered merge). Both arms ride a typed AggExpr (NativeAggExpr for LIST, ScalarExpr for the SQL aggregates). The single-table fluent siblings (agg.list, collect_lists) and the nested equi-join (join(nested=True)) ship under their own entries below. Spec/plan archived at docs/archive/FLUENT-EXTENSIONS-IMPL-PLAN.md; as-built in CLAUDE.md / ARCHITECTURE.md.
Features A × B compose. A .with_columns / .map_partitions / .where after a crossmatch(nested=True) runs on the nested per-object list columns (the list-in → scalar-out lightcurve shape: numpy mode → an object-array of per-object sub-arrays, polars mode → a pl.Series of List with .list.*), staying partition-local (the Map runs in phase 1, no reduce). The schema fold derives the post-aggregate Map's output columns zero-I/O. A list-output dtype and a direct post-nest .select() narrowing remain deferred (FLUENT-FUTURE-EXTENSIONS.md).
Fluent Catalog.collect_lists(*cols, order_by=…, descending=…). A terminal fold verb after group_by — the single-table convenience sugar over agg.list, sibling of the nested join's all-RHS-columns default. It folds every column (or a named subset) except the group key(s) and the HEALPix index column into per-group list<T> columns named after their source column, so the headline "group diaSource by diaObjectId, collect every column into per-object lists" light-curve shape no longer makes you enumerate agg.list(...) by hand. Naming the columns narrows (only those agg.lists are built, so projection pushdown reads only them + the key + order_by); omitting them folds all the rest. Pure frontend sugar in api/catalog.py — it desugars to an AggregateStep of one agg.list(col, order_by=…) per column and flows through the existing _compile_aggregation / list_aggspec path, so there are no _fluent / _optree / engine changes. It inherits the cross-partition default and the group_by(copartitioned=True) partition-local form (and that path's agg.list-only / plain-key / no-having-sort-limit restrictions). order_by sorts within every list consistently (via the kwarg or a trailing "<col> DESC" suffix; the two compose by OR). Single-catalog only for now (a preceding crossmatch/join is rejected — the merged-frame fold is a follow-up); duplicate fold columns raise an actionable ValidationError. Spec archived at docs/archive/COLLECT-LISTS.md; as-built in CLAUDE.md / ARCHITECTURE.md.
Fluent agg.list + co-partitioned (partition-local) single-table list fold. agg.list("col", order_by=…, descending=…) is the fluent LIST/ARRAY_AGG constructor, so single-table list folding no longer requires dropping to db.sql — cross-partition by default (one row per key, full list, correct for any partitioning, equivalent to db.sql LIST(...) GROUP BY). group_by(*keys, copartitioned=True) is the opt-in partition-local form: it asserts the keys are co-partitioned (every row sharing a key lives in one HEALPix partition — the HATS nested-association layout) so the LIST fold runs phase-1 only with no cross-partition combine, the single-table sibling of the nested equi-join. copartitioned is a derivation input to the Aggregate's partition_local (threaded through _optree.terminal_cluster, re-derived by the engine.lower runtime guard which now reads the new Aggregate.copartitioned field, constant False in the db.sql analyzer) — the "always derived, never hand-set" invariant (§II.5.1) is preserved. A wrong copartitioned assertion makes a key spanning N partitions appear in N rows with split lists, which is why cross-partition is the default and the flag is off unless asked. Currently copartitioned requires every aggregate be agg.list and plain column group keys, and forbids .having()/.sort()/.limit() (each needs a cross-partition combine) — per-partition decomposable aggregates, expression keys, and a catalog-level co-partition declaration are follow-ups. Spec archived at docs/archive/FLUENT-LIST-AGGREGATE.md; as-built in CLAUDE.md / ARCHITECTURE.md.
Nested equi-join — Catalog.join(..., nested=True, order_by=…). The object ⋈ source ON objectId light-curve shape now folds each left row's equi-join partners into per-row list<T> columns (one row per object; the left row's own columns stay scalar), matching the nested crossmatch (.crossmatch(nested=True)) that landed in M1.3. order_by= sorts the elements within each list, consistently across every list column. Frontend-only — three fields on OrdinaryJoinIR, two join kwargs, and one detection line in _compile_components; _fluent._compile_nested was already join-kind-agnostic, so there are zero engine changes. The aggregation is partition-local (phase-1 only), grouped on __anchor_rowid, so the right catalog must be co-partitioned with the left by the left object's HEALPix pixel (the HATS nested-association layout); a non-co-partitioned association silently under-fills lists exactly as the flat .join() drops rows — this is a documented precondition, not verified. The cross-partition variant is a follow-up. Spec archived at docs/archive/NESTED-EQUI-JOIN.md; as-built in CLAUDE.md / ARCHITECTURE.md §3.
acid download <name> resolves a bare catalog name. A single bare name (no /, not a URL) now resolves its source against a new download search path — ACID_DOWNLOAD_PATH → the download_path config key → the built-in https://data.lsdb.io/hats/ — and, when no destination is given, its destination against ACID_PATH (the same search path acid query uses). Source resolution is collection-aware: for each <root>/<name>, a directory holding collection.properties is treated as a collection and its hats_primary_table_url child is downloaded (so acid download two_mass → https://data.lsdb.io/hats/two_mass/two_mass). The destination follows the same bare-vs-path rule as the source: omitted → <first local writable ACID_PATH root>/<catalog name> (URL entries skipped, auto-created with a notice); a bare token → <ACID_PATH root>/<token>; a path with a / (./x, /data/x) → verbatim. An explicit source path/URL is used verbatim and still requires an explicit destination. download_path is a first-class config key (acid config set/get/unset/show download_path) and resolves through the usual explicit → env → config → built-in precedence.
acid inspect <name> resolves a bare catalog name against ACID_PATH (the same local search path acid query uses) → config path → ~/datasets, collection-aware (a collection.properties directory resolves to its hats_primary_table_url child). So acid inspect two_mass / acid inspect schema two_mass work on a downloaded catalog without typing the full path. An explicit path or URL is used verbatim; a remote catalog needs its full URL (the remote download mirror is not searched).

Changed¶

acid.connect() removed; use acid.init() (singleton) or acid.Connection() (explicit). Per the alpha no-backcompat policy there is no deprecation shim — acid.connect(...) is gone. acid.Connection(...) is the same object it always returned (the explicit, context-manager Connection); acid.init(...) is the new module-level singleton entry. The CLI is unaffected (it builds its own Connection).
db.in_cone(...) is now an execution-time scope, not a construction-time capture. A Catalog no longer records the cone-block context at the moment it's built; the cone is read when the query is compiled/executed (which is what already happened for db.sql). So a query can be built once and run scoped inside an in_cone block and full-sky outside it — and a query built inside a block but executed outside it is full-sky (previously a StaleCatalogError):

q = db.open("gaia").where("phot_g_mean_mag < 18")   # built anywhere
with db.in_cone((180, 0), radius=2*u.deg):
    near = q.to_polars()    # scoped to the cone
allsky = q.to_polars()      # full sky — same query object

Consequences: the cone is no longer part of Catalog identity (two identical queries built under different cones compare equal); Catalog._captured_cone_stack and the cone-prefix freshness check are gone. StaleCatalogError is removed (it only ever signalled the cone-context mismatch this change eliminates; a closed/GC'd Connection still raises ConnectionClosedError).

acid.agg constructors renamed to the astronomer/numpy idiom — agg.avg → agg.mean, agg.stddev → agg.std, agg.variance → agg.var (the other constructors are unchanged). The old SQL-spelled names are removed (alpha: no aliases). This is the Python surface only: the internal AggExpr.func decomposition token stays SQL-standard ("avg"/"stddev"/ "variance") — it is the key shared with the db.sql analyzer, so SQL queries still spell it AVG/STDDEV/VARIANCE. (FLUENT-REFERENCE.md §4.3.)
Nested joins execute by pre-imploding the RHS (no LHS-duplicated wide product). A nested=True crossmatch/join is now a first-class Join property (Join.nested), not a synthesized Aggregate above the join, and the engine collapses the RHS to per-anchor list<T> columns before the join — so the anchor (often a 1000+-column object row) is never replicated across its partners. At 2000 objects × 500 epochs × 300 columns the peak RSS drops ~20× (2469 MB → 122 MB) with identical output. Behavior change: a LEFT-unmatched anchor's lists are now an empty list [] (list.len() == 0), not a one-element [null]. (Engine: engine.lower._exec_nested_fold; as-built in ARCHITECTURE.md §5 / CLAUDE.md #14–#16; design in docs/archive/NESTED-PREIMPLODE.md + docs/archive/NESTED-JOIN-NODE.md.)
group_by(copartitioned=True) is now correct at partition boundaries. The single-table co-partitioned LIST fold scans the catalog's margin band (so a key's boundary-spilled rows land in the partition) and deduplicates with a local min(_healpix_29) ownership filter — each group emitted by the one partition whose cursor pixel owns its minimum index, no cross-partition communication, still phase-1-final. A boundary-straddling object now folds into one complete row (matching the cross-partition default) instead of splitting. Correct under the precondition object extent ≤ margin radius. It now requires a spatial index (_healpix_29) and a configured margin cache — each missing piece is a compile-time ValidationError (no silent boundary-wrong fallback). The cross-partition default is unchanged and needs neither. (Design in docs/archive/COPARTITIONED-MARGIN.md; as-built in ARCHITECTURE.md §5 / FLUENT-REFERENCE.md §6.5.)
Symmetric fluent→IR→Polars redesign (the IR mirrors the fluent composition; both join operands are lowered the same way). The cKDTree spatial match is now treated as the only operation Polars can't do natively, so a Join's left and right operands are real Op subtrees lowered through the same engine/lower._exec_subtree. This dissolves four asymmetries that were artifacts of the old left/right split:
No more inline_where on the Op tree. A pre-filter on either operand is just a Filter node in that subtree, applied before the matcher — so nearest-surviving-match and LEFT-NULL semantics fall out of node ordering rather than a special right-keys filter slot. (db.sql's Relation.inline_where is now a value-type intermediate that _optree._leaf lowers to a Filter.)
The fluent compiler is a fully-direct _steps→Op walk (frontend/_fluent.compile_catalog_ops) — no relational value types, no components dict, no shared assembler. _optree (assemble_opplan / build_spine / terminal_cluster) is now the db.sql analyzer's path only; assemble_fluent_opplan / _build_interleaved_spine / _fold_above_nested_joins and the transitional _fluent_direct.py + equivalence test are deleted.
Nested joins compose. a.crossmatch(b, nested=True).crossmatch(c, nested=True) (and the equi-join equivalent) now work — each nested=True join folds inline as a partition-local Aggregate above its Join, grouping on the current row's full identity (all live source rowids except the folded one), listing the folded RHS columns and carrying the rest via a first/carry aggregate. The "nested must be the first join" and "exactly one nested join" guards + the special nested dispatch are gone.
The equi-join RHS keeps its scan rowid (equi_right_rowid) and its _healpix_29 just like a spatial-join RHS — "RHS in a join" and "RHS in a crossmatch" now carry the same surviving identity (which is what lets nested joins compose). Pruning the RHS _healpix_29 was a bug; it is a normal catalog column now, kept and collision-named like any other.
db.sql collision naming is left-bare / right-prefixed (HATS-clean). In a db.sql join, the left side keeps its natural names and only a colliding right column is dotted (id, b.id) — the previous scheme prefixed both sides. Output is clean for HATS round-tripping.
Operand .select() is honored (not rejected). a.crossmatch(b.select(…)) projects the operand subtree via a plain Project node at its top — no special right_select field, no hidden coord retention. The operand is projected before the matcher/key-join, so the select must keep the coords / join key; dropping one is a clear compile-time ValidationError rather than a silent miss. Bare/renamed columns only — computed operand projections are deferred.
Verbs after .aggregate() compose in written order; .having() is removed. The aggregate is a barrier, and select / where / with_columns / sort / limit after it form the post-aggregate chain (_fold_post_aggregate → Project / Filter / Map / TopK / Limit), composing by position over the aggregate output. A post-aggregate .where() is the exact equivalent of the old .having() (deleted), and ordering is honored — .sort().limit(5).where(…) filters the top-5 (fluent ⊋ SQL). Engine: reduce.reduce_ops = _combine_aggregate (group_by/agg + the db.sql HAVING field + rename) + _lower_post_aggregate (lower the chain over the combined frame); phase-1 _lower passes through everything above a combining (cross-partition) aggregate. db.sql keeps SQL HAVING via the retained Aggregate.having field (analyzer/optree untouched). Partition-requiring verbs after an aggregate (crossmatch / join / in_region) stay rejected — the reduced result isn't HEALPix-partitioned and the aggregate consumes the coordinates (issue #101). Spec archived at docs/archive/FLUENT-IR-REDESIGN.md; as-built in CLAUDE.md (decisions #14–#16) / ARCHITECTURE.md §3–§6.
Fluent composition is now one ordered step list (the fluent tree builder). A Catalog's entire composition is a single ordered _steps tuple (api/catalog.py) — every verb appends exactly one Step. The role-segregated slots (_joins/_pre_where/_post_steps/_select/ _limit/_group_keys/_aggs/_having/_order/_regions) are deleted; _fold_steps folds _steps into slot-equivalents and _fluent._StepView wraps a Catalog so the compile tail runs unchanged. Verb placement is now structural — the pre/post-where barrier (first join or Map) and the last-of-kind terminal verbs are decided in _fold_steps, not at verb time. __eq__/__hash__/__repr__/describe fold _steps, so identity can't drift the way parallel slots could; _columns_override is now part of identity (a db.open(..., columns=…) subset reports distinct .columns, so it must compare/hash distinctly — a latent collision bug, fixed). The post_steps_post_terminal flag is gone in favor of two explicit ordered chains in assemble_fluent_opplan (post_steps on the spine, post_terminal_steps + post_terminal_projection above the terminal). db.sql (parser/analyzer) is byte-untouched. The design is archived at docs/archive/FLUENT-TREE-BUILDER-DESIGN.md; the as-built is CLAUDE.md (decision #14) / ARCHITECTURE.md §3.
Post-Map nested .select() unlock. A .select() after a .with_columns() / .where() on a crossmatch(nested=True) now projects the post-Map columns — it becomes a Project above the post-terminal Map chain (Project(Map(Project(Aggregate)))). A .select() directly after the nested crossmatch still prunes which RHS columns are listed.
validate_ops is the structural backstop for Map/Filter placement. A Map or Filter inside a Join's subtree (the operand-subtree / between-joins shape the builder can now represent but the engine cannot yet execute) is rejected there; the thin verb-time guards (_reject_if_mapped/_reject_mapped_operand) supply the friendly, verb-keyed message. The schema fold, engine.lower._spine, and connection._explain_plan walk through such a node so a builder-produced-but-rejected tree folds and explains cleanly.
ParquetSink flushes on bytes, not rows. The buffer threshold is now byte_target (default 256 MiB), replacing the previous row_group_target (1M rows) entirely. The row-count threshold left the in-memory buffer free to balloon on wide schemas: 1M rows × long string / struct / list columns (e.g. the Rubin DP1 object catalog at ~1250 columns) can be many GB before the flush fires, turning the buffer into the OOM vector the flush was meant to prevent. pa.Table.nbytes (a C-level sum of Arrow buffer sizes) is the cheap estimate we accumulate against byte_target. Per-flush row count is now variable; the row_group_size pin still ensures one Parquet row group per flush. Breaking config change (alpha; no shim) — the row_group_target= kwarg is gone. Fixes #33.
acid never auto-creates a user-supplied tmpdir. A non-existent path passed to acid query --tmpdir <path> or to acid.connect(tmpdir=<path>) now fails fast (with AcidError from the CLI / ConfigError from the library) instead of being silently materialized via mkdir(parents=True, exist_ok=True). The hazard the auto-create masked was real on shared systems: a mistyped path or one whose intended mount isn't present landed scratch data on the wrong volume (often under $HOME), with no warning until the disk filled up. acid download already had this fail-fast discipline; the CLI and the Python API now match. If you need the auto-create, do it explicitly (mkdir -p <path> / os.makedirs(...)) before calling acid. Fixes #72.
The CLI moved to its own top-level package, and import acid no longer tunes the host. The CLI source (acid.cli) is now the new top-level acid_cli package; the console script is acid = "acid_cli:main" and python -m acid_cli also works. acid_cli owns its process and stages worker tuning (jemalloc _RJEM_MALLOC_CONF + the BLAS/OMP family) before importing the heavy stack. The acid library itself sets no allocator/thread env at import — so import acid in Jupyter (or any host app) no longer mutates that interpreter's allocator or thread pools, the silent side effect that motivated this work. acid/__init__.py is PEP 562 lazy (heavy names resolve on first attribute access); acid --help / --version / acid config complete without loading numpy / polars / pyarrow. Worker tuning is staged on-demand via a numpy-free preload shim (acid.engine._worker_env) listed first in set_forkserver_preload, fed by inert ACID_WORKERS_* env vars the parent exports through the new mandatory start_pool_with_env helper both pool builders route through. The forkserver preload list collapses to two entries (the shim + the acid.engine._preload fat-import anchor) — adding a new heavy worker-side dependency means adding an import to _preload.py (a drift test polices this). ACID_CAP_BLAS is dropped (OMP is now always managed in acid-owned processes); the ACID_FORKSERVER_PRELOAD=0 opt-out is dropped (preload is mandatory — the one airtight worker-tuning hook). User-facing knob is now workers_jemalloc_conf / ACID_WORKERS_JEMALLOC_CONF on the standard explicit → env → config → built-in chain. One user-visible behaviour change: an imported acid.connect(workers=1) runs the in-process per-partition collect with the host's stock jemalloc (the change deliberately leaves a host untouched — the cross-process madvise contention the tuning kills is largest at high worker counts and negligible at workers=1). Spec archived at docs/archive/WORKER-ENV-TUNING.md; user-facing guidance in MEMORY-TUNING.md.
Operator-tree engine: the flat Plan IR is gone. The one executed plan shape is now the left-deep operator tree (OpPlan, plan/ops.py); both frontends emit it directly (analyzer.analyze_ops, _fluent.compile_catalog_ops, via the shared _optree.assemble_opplan), the engine lowers it to one Polars LazyFrame per work-tuple (engine/lower.py), and the phase-2 reduce reads off its root nodes (reduce.reduce_ops). The flat Plan dataclass and its lowering path (analyzer.analyze, _fluent.compile_catalog, Catalog._compile_plan, reduce.reduce, executor.phase1_agg/aggregate_output_columns, schema.merged_schema/output_columns/column_origin_map, the lower_legacy_plan_to_ops bridge) are deleted; acid.plan.Plan is now a kept alias of OpPlan. Connection.validate(query) returns an OpPlan. The engine refactor ADR is archived at docs/archive/ARCHITECTURE-ENGINE.md; the as-built model is ARCHITECTURE.md §4–§6.
Memory-first phase-2 reduce. Global-reduce queries (decomposable aggregates / top-K) now combine their per-partition partials in memory instead of always round-tripping through a Parquet tempdir — a latency win on the interactive COUNT(*) / GROUP BY path. The disk reduce remains the automatic fallback when partials spill past inmem_row_limit, so billion-row behavior is unchanged. Applies to .df(), the CLI display and single-file --out, and Catalog.save. See docs/archive/REDUCE-INMEMORY.md.
acid query with no --out streams the full result. The implicit 100-row display cap is gone: output is a type-driven fixed-width table on a TTY (columns trimmed to terminal width) and TSV when piped/redirected, emitting every row. Fixes silent truncation of piped output (acid query … | wc -l previously returned 100, not the true count). A user-written LIMIT N is still honored.
db.sql(query, output=<path>) for a global reduce is now rejected (ValidationError) instead of writing phase-1 partials into the directory (which was neither the answer nor a valid catalog). Omit output= for the result, or use Catalog.save for single-partition HATS.
Empty catalogs are rejected at registration. A HATS catalog path that exists but enumerates zero partitions now raises RegistryError at connect() time, instead of silently yielding an empty query result — a valid HATS catalog must have at least one partition. A non-existent path is still tolerated for offline config validation (with hpix_order set explicitly). This removes the now-dead empty-catalog special-casing in reduce_global.

Fixed¶

Virtual-catalog batch slicing now works under multiple workers. Per-partition IPC batch slicing maps a cursor to batch indices via TableSpec.partitions, but the plan shipped to workers strips partitions (parent-only enumeration bulk) — so with workers > 1 a virtual root found no batches and read 0 rows (a workers=1 run, which doesn't strip, hid it). _strip_for_workers now keeps partitions for a virtual spec (bounded by max_partitions, cheap to ship), while still stripping a real HATS catalog's potentially-huge list. Regression test runs with workers=2.
Virtual-catalog rows are read once, not once per partition. A virtual catalog (db.open(<file | frame>)) is backed by a single file that holds the whole catalog, so — unlike a real HATS partition, whose file is that one partition — each partition's root scan must be filtered to its cursor pixel. _root_filters only applied that filter when refining below the root partition, so a virtual root read all rows in every partition: db.open(f).count() / .to_polars() returned N×P (rows duplicated per partition), and a virtual-root crossmatch did O(P×N) wasted work. Fixed by always applying the cursor-pixel filter to a virtual root. (The crossmatch oracle tests missed it because they compare sets of pairs, which dedup the duplicates; a non-deduping count() / row-count regression test now guards it.) Found while benchmarking the per-partition read path.
Crossmatches no longer miss matches at the outer edge of an RHS footprint. A root point just outside the right catalog's footprint, within the match radius of a right point just inside (across a partition seam into an empty pixel), was silently dropped (INNER) or returned a false NULL (LEFT) — a real correctness gap wherever two catalogs' footprints differ (common with disjoint survey footprints). Root cause: the margin cache only bridged central↔central partition seams, and the enumeration coverage was central-only, so a cursor in an empty pixel had no coverage and no staged data. Fix, in two pre-staged halves (no query-time inter-tile communication): the margin builder now also emits a "ring" of margin files at the empty pixels just outside the footprint (uniform at the catalog's finest partition order), holding the bordering central rows; and the spatial-crossmatch enumeration coverage is now central ∪ ring (TableSpec.coverage_partitions), so a cursor in a ring pixel is provisioned with the ring's margin file and the matcher finds the across-edge partner. Margin builds gain a perimeter-bounded set of small ring files; query enumeration gains only the boundary-shell work tuples that were previously skipped (scales with the populated footprint perimeter, not area). Equi joins are unchanged (co-partitioned / key-based, no ring). Validated: a new synthetic edge test (INNER + LEFT) plus the bench/validation real-data suite at exact equality with ring-inclusive margins. Existing margin caches should be rebuilt (acid hats build-margin) to pick up the ring; a ring-less cache simply keeps the old central-only coverage (no regression, no edge fix).
FixedWidthSink accepts an explicit width= constructor kwarg. The no---out TTY display path consulted shutil.get_terminal_size((80, 24)) unconditionally, so layout depended on the calling terminal — useful for the actual TTY display but a hazard for tests and library callers that wanted a deterministic layout. The new width= (default None) pins an explicit column-drop budget when given; None keeps the existing ambient-terminal behavior (which honours $COLUMNS first via shutil.get_terminal_size, so users can still override at runtime). Fixes #87. The test that bit the bug (test_fixed_width_sink_type_driven_null_and_nested) now uses width=120 and passes deterministically regardless of the calling terminal width.
Streaming sinks raise a clear OutputError on schema drift (instead of a confusing pyarrow stack trace mid-stream). Each of ParquetSink / CsvSink / FitsSink now compares every incoming partition table's schema against the first one it pinned the writer to; on drift it raises with a diff (added / removed columns, type changes) and explicitly says this is an acid bug, pointing at the issue tracker — because the engine is supposed to produce a consistent output schema across partitions for a given query plan (see plan/schema.py's schema fold + Polars's null-fill on LEFT outer joins). The original reproducer in #35 (LEFT XMATCH null-fill drift to null<>) no longer fires on current main, but the safety net catches any future regression with a useful message rather than a truncated file at --out. Closes #35.
Download: flat columns whose name contains a dot are no longer mistaken for struct leaves. acid.tools.download mapped a parquet physical column back to its top-level Arrow field by splitting path_in_schema on . — but a flat column literally named a.foo and a struct a's leaf foo produce identical path_in_schema='a.foo', so the naive split collapsed them. Now _physical_col_indices walks the actual Arrow schema (counting leaves per top-level field via the new _physical_col_ranges helper) to keep the two cases distinct. This matters because the query-lowering redesign lets db.sql emit dotted output column names on collision (SELECT a.foo, b.foo → a.foo, b.foo), so flat dotted names can now appear in HATS trees acid itself writes. The CLAUDE.md "Parquet physical vs logical column indices" gotcha was updated; it had been endorsing the buggy split recipe. Fixes #45.
Scratch HEALPix-count mmaps no longer land in the output directory. Both point_map.fits accumulators are pure scratch (mkstemp'd, unlinked after use) but were created inside the output tree, where a full read-back of the ~805 MB order-12 map dominates on a slow/networked store (~30 s on /sdf NFS vs ~1 s on local scratch). They now go to a scratch dir (None ⇒ system temp, honours $TMPDIR): the engine skymap via execute(..., scratch_dir=) — the CLI HATS path passes --tmpdir, the Connection its owned scratch dir (#68) — and the download-time catalog footprint via download_catalog(..., tmpdir=), exposed as a new acid download --tmpdir flag (#71). The download path's accumulator is (workers, npix), so the win there scales with worker count. A download --tmpdir that doesn't exist now errors before any bytes are fetched (it is never auto-created), so a mistyped path fails loudly rather than leaving a half-built catalog.

Deviation from the spec¶

Operand .select() is rejected, not honored. The fluent tree builder design (FLUENT-TREE-BUILDER §1.6/§7) listed an operand's .select() (a.crossmatch(b.select(...))) as a "honored for free" unlock. Honoring it requires projecting the operand subtree before the merge — changing collision suffixing in op_merged_schema and the matcher's right frame in _exec_merged, i.e. the high-risk operand-subtree execution the design defers. It is now rejected loudly (_reject_select_operand), turning the prior silent drop into a clear error; honoring it folds into the deferred operand-subtree-execution follow-up (CLAUDE.md "Things explicitly NOT done"; FLUENT-FUTURE-EXTENSIONS.md §2.4).

[0.2.0a3] — 2026-06-03¶

Worker-startup performance, a dependency-light MOC implementation, a unified progress UI, and a self-configuring acid.conf settings layer. Includes public Python API and CLI changes (see Changed).

Added¶

Memory-aware workers="auto" — the automatic worker count is now min(cpu_cap, mem_cap, 24): in addition to the existing CPU cap (affinity / cgroup quota / cpu_count), a memory cap gives each worker at least mem_per_worker_gb of RAM (default 4 GB), so a high-core but memory-modest box (or a tight cgroup memory limit) no longer spins up one worker per core and OOMs. The auto count is also capped at 24 (parallel efficiency flattens before the core count on big nodes); the cap is auto-only — an explicit workers / ACID_WORKERS / config / --workers is never capped. The memory figure is min(physical RAM, cgroup memory limit); a small tolerance absorbs the kernel's RAM reservation. New mem_per_worker_gb config key / ACID_MEM_PER_WORKER_GB env / acid.connect(mem_per_worker_gb=) arg. cgroup CPU- and memory-limit detection now honors the mountinfo root mapping, so a limit on a container whose own cgroup is the mount root (typical Docker/k8s) is read at the mountpoint instead of a nonexistent nested path — previously such limits were silently missed.
acid.conf settings layer (config.py; see docs/archive/CONFIG-SYSTEM.md) — per-machine configuration of the catalog search path, query workers, the per-worker RAM budget mem_per_worker_gb, the tmpdir base, and inmem_row_limit, resolved explicit → env → config file → built-in. Discovery is first-found-wins over ~/.config/acid/acid.conf, two /sdf/.../etc paths, $XDG_CONFIG_DIRS, and /etc/acid/acid.conf; --config FILE / ACID_CONFIG point at a specific file. Env overrides: ACID_PATH, ACID_WORKERS, ACID_MEM_PER_WORKER_GB, ACID_TMPDIR, ACID_INMEM_ROW_LIMIT. New acid config show|get|set|unset subcommand (file values by default, --effective for resolved). acid query gains --tmpdir; --db/--workers/--tmpdir --help shows the effective default + provenance. acid.connect(...) gains a config= argument.
rich-based progress UI (io/progress.py + the engine-neutral engine/_reporter.py Reporter protocol). One self-overwriting status line on a TTY — an arrow3 spinner through the setup stages, switching to a full-width block bar with percent / row count / ETA during execution. Writes to stderr only, so piped stdout results stay clean. New acid query --progress {auto,on,off,plain} flag, fronting the ACID_PROGRESS env var (0/1/plain); plain commits one line per stage for logs/debugging.
acid query prints a final elapsed: <dur> runtime line (interactive only — silent when piped or --progress off). Anchored at process start, so it spans imports → parse → worker launch → execution → metadata write (≈ /usr/bin/time minus the unmeasurable interpreter launch + import acid).
Startup banner for acid query — a one-time cyan boxed ACID logo (● ▪ A C I D, tagline, version, project URL) printed to stderr the moment the CLI is alive, with a compact fallback on narrow terminals. Gated like the progress UI (TTY / ACID_PROGRESS), so it's silent when piped or off. The box also carries a runtime/resource panel — worker count, threads per worker, and total memory, then the temp directory and (with --out) the output path, each with their filesystem's free disk space; long paths are tail-ellipsized to fit.
hats/rangemoc.py — a minimal MOC (order-29 [lo, hi) ranges + vectorized set algebra) covering the mocpy.MOC slice acid uses, so mocpy is no longer needed at runtime. Validated bit-for-bit against mocpy (now a test-only oracle) in tests/test_rangemoc.py.
Worker-startup tuning knobs, all on by default (opt out with =0): ACID_CAP_BLAS (cap native BLAS/OMP thread pools), ACID_FORKSERVER_PRELOAD (preload the native stack into the forkserver server for COW inheritance), ACID_PREWARM (spawn all workers up front). Documented in MEMORY-TUNING.md §Worker startup.

Changed¶

acid.connect(...) settings now resolve through acid.conf (alpha, no back-compat shims). cache_dir= is renamed tmpdir= and is now a base directory — a unique owned scratch subdir is created under it and removed on close() (previously an explicit cache_dir was used directly and never cleaned). workers/inmem_row_limit default to None (resolve from env/config/built-in) instead of "auto"/50_000_000, and source=None now resolves (ACID_PATH → config path → ~/datasets) rather than yielding an empty connection — pass source=[] for an explicitly empty one.
acid query / validate require the SQL query. Pass it as the positional argument, - to read stdin, or -f FILE; omitting it is now an error (it no longer silently reads stdin). acid query's built-in --workers default is now auto (cgroup-aware), not 1.
Worker startup optimized for high partition counts. The plan shipped to workers is stripped of parent-only bulk (TableSpec.partitions / partition_index / margin_partitions, MocSpec.moc) — ~2 MB → ~1 KB on a 100k-partition catalog; a single-catalog enumeration fast path skips the dense partition-ID map; workers capture parquet FileMetaData at write time so the master assembles dataset/_metadata by concatenation instead of re-reading every footer. Several-fold faster cold start at high worker/partition counts.
point_map.fits accumulation is now a single lock-free shared mmap (was per-worker rows summed at the end), written sparsely (O(nonzero)). Work tuples own disjoint footprint pixels, so workers scatter-add without locks; a runtime row-count cross-check raises loudly if that invariant is ever violated.
CLI progress is now TTY-auto — an animated bar on a stderr TTY, silent when piped. Previously the tqdm bar was effectively always shown.

Fixed¶

CTRL-C during acid query now exits cleanly instead of flooding stderr with per-worker tracebacks. A terminal SIGINT reaches the whole foreground process group; pool workers now ignore it (so only the parent unwinds), and the parent forcibly reaps a one-shot pool (SIGTERM → brief grace → SIGKILL) before exiting with code 130. The reap ignores further SIGINTs so repeated CTRL-C can't orphan workers. A Connection-owned persistent pool is left untouched (its workers stay healthy and reinitialize on the next query).
CTRL-C during worker-pool startup no longer leaks a traceback from a process the user never sees. Two startup-window leaks are closed: (1) the forkserver bootstrap preloads the native stack (numpy/pyarrow/polars/…) before it installs its own SIGINT: SIG_IGN, so a CTRL-C mid-preload printed a KeyboardInterrupt from deep inside an import — the forkserver is now started while the parent holds SIGINT: SIG_IGN (which survives exec, and a fresh Python honors an inherited SIG_IGN), so its preload runs uninterrupted; and (2) the ACID_PREWARM barrier's transient multiprocessing.Manager raced its own teardown on interrupt, printing a FileNotFoundError/BrokenPipeError — the prewarm now defers SIGINT across the Manager's lifecycle and re-raises it cleanly once the Manager is down.
Misleading "unknown column" error for a table alias passed to an unrecognized function. A typo like WHERE xIN_MOC(d, 'object') previously raised a self-contradictory unknown column 'd' … known tables: d, o whose caret pointed at an unrelated d.* in the SELECT. The analyzer now diagnoses it as 'd' is a table alias, not a column, hints toward IN_MOC (the only acid function taking a table alias), and the error-span finder points the caret at the offending alias argument rather than the first textual match.

Removed¶

mocpy runtime dependency — moved to a test-only extra (see hats/rangemoc.py).
tqdm dependency — replaced by rich (the tools' progress bars now use the acid.io.progress.bar shim).

[0.2.0a2] — 2026-06-01¶

A performance + tooling release on top of the native-Polars engine. No API or on-disk changes.

Fixed¶

--workers 16 native-engine regression (the headline fix). A wide LEFT-XMATCH anti-join ran ~30% slower than the old --engine=polars build. Root cause was the bundled-jemalloc dirty-page purge (madvise(MADV_DONTNEED)) serializing across workers on the kernel mmap_lock, not acid code. acid now sets _RJEM_MALLOC_CONF=dirty_decay_ms:-1,muzzy_decay_ms:-1 by default (via setdefault, so it's overridable) before Polars is imported — ~2× faster wall at high worker counts, at ~20% higher peak RSS. Full analysis in bench/W16-EFCOLLECT-REGRESSION.md.
O(C²) per-partition schema fold on wide catalogs. _relation_columns now indexes column_types positionally instead of the per-name TableSpec.column_type linear scan, which dominated worker CPU on a 1250-column light-curve catalog.
coalesce=False on the XMATCH join made the LEFT join ~2× slower per partition on a wide right catalog. Now coalesce=True — invisible to output (the coalesced key is an engine-internal __ column).
ACID_PROFILE=1 with multiple workers produced no output — forkserver workers don't run atexit hooks. Rewritten as a per-worker shared mmap (one row per worker, summed by the master), mirroring the skymap mechanism. (#50)

Added¶

MEMORY-TUNING.md — user-facing guide to the jemalloc allocator knob and the related workers/threads memory levers; linked from the README and listed as an authoritative doc.
engine/profiling.py — per-worker, per-step profiling module (anchor_setup / right_setup / xmatch / execute_final / write), with a stderr summary table and a JSON per-worker matrix (ACID_PROFILE_OUT). (#50)

[0.2.0a1] — 2026-05-31¶

Lands the fluent Catalog API (CATALOG-API.md). The old Session / acid.sql / acid.run surface is removed without aliases — 0.1.0a* code won't run unchanged. See "Removed" for the migration sketch.

Polars-native engine (breaking)¶

acid now has one execution engine: native Polars. A query lowers to a single engine-neutral Plan executed as polars.LazyFrame ops, including the phase-2 reduce (ARCHITECTURE.md).

Removed (no aliases, alpha):

The engine= keyword and --engine flag — Polars is implied.
The duckdb_threads= keyword / --duckdb-threads flag — renamed to threads= / --threads.
The DuckDB engine (acid.engines.duckdb), the Engine / PartitionContext ABCs, and acid.engine.resolve.
The per-tuple SQL generator (acid.rewriter) and the SQL phase-2 reducer (acid.reducer).
The QueryPlan IR container and the acid.planner adapter — acid.analyzer.analyze now emits acid.plan.Plan directly.
DISTINCT / COUNT(DISTINCT) / bare GROUP BY / unbounded ORDER BY — the full-materialization fallback now raises ValidationError. Decomposable aggregates, top-K, SELECT *, and bare LIMIT are unchanged.
duckdb is no longer a runtime dependency (test/bench only).

Changed — scalar SQL semantics are now Polars-SQL semantics (via pl.sql_expr):

ROUND rounds half-to-even (2.5 → 2), was half-away.
CAST(float AS INT) truncates (2.7 → 2), was rounding.
Math-domain edges — SQRT(-x) / LN(0) yield NaN / -inf instead of raising.

Preserved: XMATCH (SciPy cKDTree), margin handling, HATS-output validity, aggregate decomposition, top-K / cone / MOC pushdown. validate() returns the Plan; explain() returns a Plan summary.

Query lowering redesign (breaking)¶

A rewrite of the lowering layer (docs/archive/QUERY-LOWERING-REDESIGN.md) collapses the two-layer IR and the column-name mangling into a single flat-named Plan both frontends emit directly. No perf change at scale — the win is a smaller correctness surface.

Output column names change (user-visible). Join collisions keep the anchor's natural name and get a _<alias> suffix on the right (id / id_b); db.sql output collisions are SQL-dotted (a.id / b.id). Names are now final from the start — identical on disk, at the Result boundary, and in .columns — so saved queries referencing the old mangled/demangled names will differ.

Removed (no aliases, alpha):

XMATCH_DISTANCE(<alias>) — replaced by the opt-in crossmatch(dist_col="sep") / XMATCH(..., dist_col => 'sep') column (un-named ⇒ not emitted).
IN_MOC(...) outside a top-level conjunctive WHERE — it's a footprint restriction only (AND-ed WHERE, optional NOT, or .in_region(...)); anywhere else raises ValidationError.
The <alias>__<col> mangle, executor.canonicalize_sql, and output.demangle_columns.
The fluent → SQL-string → parser.parse round-trip — the fluent Catalog now compiles directly to a Plan (acid._fluent).
The ir.py scratch IR (RelationRef / XMatchJoin / AggregatePlan / PartialAgg).
scalar.RangeMembership / scalar.BoolOp — Predicate = ScalarExpr.
acid.rewriter and acid.engine (both deleted) — the PartitionFilters contract now lives in acid.ir.

Fluent join surface. Catalog.join(other, on=...) takes a single on= — on="id" or on=(left, right), integer-ID equi-joins only.

Added¶

acid.connect(source, *, workers="auto", threads=None, inmem_row_limit=50_000_000, cache_dir=None, progress="auto") — the single top-level entry point, replacing acid.sql / acid.run.
Connection (renamed from Session) — cgroup-aware workers="auto", lazy pool, sticky settings, __del__ / atexit cleanup, pickling rejected.
Connection.open(name_or_path, *, alias=None, columns=None) -> Catalog — fluent entry; eager metadata read, lazy pool start.
Connection.in_cone(center, *, radius) — context manager scoping a cone over every query in the block (replaces the old cone= kwarg); nested blocks rejected.
Connection.list_catalogs() — walk the registered roots for HATS-catalog basenames.
Connection.add_catalog(name, **spec_kwargs) -> Catalog — lower-level alternative to open.
Connection.register_moc(name, source) — unchanged from Session.register_moc.
Connection.map_partitions_sql(query, *, output=None, progress=None) — renamed from Session.run.
Catalog — frozen lazy-query handle. Composition verbs (where/select/limit/in_region/crossmatch/join/group_by/ aggregate/having/sort) return new Catalogs; materialization verbs (head/execute/to_pandas/to_polars/to_astropy/ to_arrow/save) run through the Connection.
Catalog.crossmatch(other, *, radius, how="nearest") — spatial XMATCH; radius is an astropy Quantity (bare floats rejected).
Catalog.join(other, *, on, how="inner") — integer-ID equi-join; bare-name + tuple forms; double-join / cross-Connection rejected.
Catalog.group_by(*keys) / aggregate(**named) / having(pred) — fluent aggregation via acid.agg constructors; keys appear output-first. Aggregation is the outer query — only having/sort/limit may follow.
Catalog.sort(*keys, descending=False, nulls_last=False) — ORDER BY; with .limit(n) it's top-K (pushed to phase-1). A standalone unbounded sort is rejected.
acid.agg (and acid.AggExpr) — the aggregate-constructor namespace for Catalog.aggregate.
Catalog.in_region(region) — a registered name, peer Catalog, FITS path/URL, mocpy.MOC, MocSpec, or HATS directory. Named/handle forms compile to IN_MOC verbatim; anonymous sources are content-hashed.
Inline subquery / CTE pre-filter (db.sql) — anchor FROM, XMATCH-RHS, and ordinary-JOIN-RHS accept (SELECT * FROM <catalog> [WHERE <pred>]) AS alias, folded into the per-partition filter plumbing. Projection narrowing / joins / aggregates / DISTINCT / GROUP BY / ORDER BY / LIMIT inside are rejected; see docs/archive/SUBQUERY-RHS.md.
Catalog.save(path, *, name=None, overwrite=False, progress=None) -> Catalog — atomic-on-success HATS write (stages to a sibling <name>.acid-save-tmp, renames only on success). Returns a registered handle.
Result.show(n=20, *, width=10_000) — terminal pretty-print (same look as acid query); Result.__str__ returns it so print(r) works.
Result.write_csv(path) / write_fits(path) / write(path, format=None) — single-file writers alongside write_parquet; write infers format from the extension.
output.format_table_text / output.print_table — exported Arrow-table formatting helpers.
progress kwarg on every materialization method plus a Connection-level default; "auto" enables tqdm under a TTY / Jupyter and stays silent under pytest.
ConnectionClosedError, StaleCatalogError — typed errors; StaleCatalogError carries captured_cones / current_cones.
ConeSpec, ConnectionStatus — public dataclasses for cone geometry and Connection.status().
Astropy adapter (acid._coerce) — converts Quantity / SkyCoord / (ra, dec) to engine types. Astropy is now a hard runtime dependency (imported lazily).

Changed¶

Default workers is "auto" (was 1) — resolves to min(sched_getaffinity, cgroup CPU quota) on Linux, else os.cpu_count(). Tests opt back to workers=1.
CLI _print_table is now a thin wrapper around output.print_table.
Catalog.save staging dir is now visible to ls (<name>.acid-save-tmp, no leading dot).
Runtime dependencies. astropy>=5 and fsspec[http]>=2023.1 are now hard deps; duckdb is test/bench-only.

Removed¶

acid.sql(query, catalogs=C, ...) — use with acid.connect(C) as db: db.sql(query).
acid.run(query, catalogs=C, output=O, ...) — use with acid.connect(C) as db: db.map_partitions_sql(query, output=O).
The Session class / SessionClosedError — renamed to Connection / ConnectionClosedError; Session.run → map_partitions_sql.
Session.materialize(name, query, ...) — use Catalog.save(path, name=name), or db.sql(query, output=path) + db.add_catalog(name, path=...).
The cone=(ra, dec, r_deg) kwarg on sql / run — use with db.in_cone((ra, dec), radius=r_deg*u.deg):.
acid.Registry is no longer in __all__ (still importable).

Fixed¶

A senior-engineering pass after M1–M3 surfaced several bugs (CATALOG-API.md §14.4):

Nested in_cone blocks silently dropped the inner cone — now rejected.
Auto-MOC names defaulted to basenames, silently aliasing — now content-hashed.
Connection.open(absolute_path) rebuilt the TableSpec each call, breaking Catalog.__eq__ — now reuses the cache hit.
Connection.add_catalog returned a TableSpec — now returns a Catalog.
Catalog names with special chars produced invalid SQL — identifiers now quoted (mostly moot post-redesign).
Catalog.save(overwrite=True) deleted the target before running, so a failure lost data — now stages and atomic-renames.
Catalog.in_region(<MOC>) emitted IN_MOC inside a filtered subquery (rejected by the engine) — moved to outer WHERE.

Performance¶

Persistent worker pool with Manager-dict plan delivery. The pool stays alive across queries, and the Plan is delivered once per query via a multiprocessing.Manager dict-proxy keyed by version instead of as a per-task argument. This lifts Catalog.save throughput to parity with acid query (was capped at ~7-8 tasks/s) while avoiding fork() of heavy-RAM notebook parents.

Notes for downstream users¶

CLI subcommands are unchanged in name/shape, with flag changes: acid query gains --format (hats/parquet/csv/fits) and drops --reduced-out; acid download gains --insecure; --engine / --duckdb-threads are gone (--duckdb-threads → --threads).
The on-disk HATS catalog format is unchanged.
Flat-on-disk column names resolve #41; round-tripping a db.sql dotted name (a.foo) through the loader is tracked as #45.

[0.1.0a3] — earlier release¶

See git history (git log v0.1.0a2..v0.1.0a3). This CHANGELOG starts at 0.2.0; prior releases are reconstructable from commit messages.