CLI reference¶
The acid command-line tool runs the same engine as the Python API.
Use it for one-off queries from a shell prompt, for catalog downloads
and inspection, and for building margin caches.
This page lists every subcommand and every flag, with defaults read
straight from src/acid_cli/. For the design rationale behind
configuration discovery and --db resolution, see
docs/archive/CONFIG-SYSTEM.md.
Synopsis¶
Global options¶
| Flag | Description |
|---|---|
--version |
Print version, copyright, and license; exit. |
--config FILE |
Path to an acid.conf (overrides ACID_CONFIG and the default search). |
acid query — run a SQL query¶
acid query [QUERY|-] [-f FILE]
[--db ENTRIES] [--open PATH,RA,DEC]
[--output PATH] [--format {hats,parquet,csv,fits}]
[--workers N] [--threads N] [--ram-budget SIZE] [--tmpdir DIR]
[--cone RA,DEC,RADIUS_DEG]
[--progress {auto,on,off,plain}]
Reads the SQL query from a positional argument, from stdin (when
the positional is -), or from a file (-f FILE). Exactly one of
these is required.
Where to write results¶
--output |
--format |
Effect |
|---|---|---|
| omitted | (rejected if given) | Pretty-print to stdout (fixed-width on a TTY, TSV when piped). |
| directory or path with no extension | inferred / hats |
Write a partitioned HATS tree at --output. |
*.parquet / *.pq |
inferred / parquet |
Single Parquet file. |
*.csv |
inferred / csv |
Single CSV file. |
*.fits / *.fit |
inferred / fits |
Single FITS binary table. |
If you set --format explicitly, it wins over the extension; an
unrecognized extension with no --format is an error rather than a
silent fallback to HATS. A global-reduce query (aggregate / top-K)
cannot be written as a HATS tree — use single-file format, or omit
--output to pretty-print.
Flags¶
| Flag | Default | Description |
|---|---|---|
-f, --file FILE |
— | Read the SQL query from a file. |
--db ENTRIES |
$ACID_PATH → config path → ~/datasets |
':'-separated list of HATS directories, registry YAMLs, or both. Leftmost entry wins on name collisions. |
--open PATH,RA,DEC |
— | Use a raw data file (.parquet/.csv/.tsv/.fits/.arrow/VOTable) as a named table in the query, alongside the --db catalogs (a virtual catalog: spilled once, queried like HATS). Two forms: positional PATH,RA,DEC (table name = file basename), or named NAME=PATH,ra=RA,dec=DEC. The RA/DEC coordinate column names are required (never guessed). Repeatable. |
--output PATH |
— | Output path; omit to pretty-print to stdout. |
--format FMT |
inferred from --output (no ext → hats) |
One of hats, parquet, csv, fits. Rejected without --output. |
--workers N |
resolved from config / ACID_WORKERS / "auto" (cgroup-aware) |
Process-pool size. |
--threads N |
cpu_cap // workers (cgroup-aware) |
Per-worker Polars thread budget. |
--ram-budget SIZE |
25 % of available RAM (cgroup-aware) | Total RAM the planner budgets for; work tuples are sized to fit SIZE / workers each. Bytes or a human size (64GB, 512MiB, 32g). See Performance — RAM budget. |
--tmpdir DIR |
config / $TMPDIR |
Base scratch directory; a per-run subdir is created and removed on exit. |
--cone RA,DEC,RADIUS_DEG |
— | Restrict the query to a sky cone (degrees, ICRS). |
--progress MODE |
auto |
auto (animated bar on a TTY, silent when piped), on (force bar), off (silent), plain (one committed line per stage; good for logs). |
Examples¶
# Pretty-print to stdout — no --output.
acid query "SELECT COUNT(*) FROM gaia_dr3" --db /data/hats
# Write a HATS tree (default --format when --output has no extension).
acid query -f xmatch.sql --db catalogs.yaml --output results/ --workers 32
# Write a single CSV — extension picks the format.
acid query "SELECT id, ra, dec FROM object LIMIT 1000" \
--db /data/hats --output sample.csv
# Read SQL from stdin.
echo "SELECT * FROM object LIMIT 10" | acid query - --db /data/hats
# Restrict to a 2 deg cone around (50, -50).
acid query "SELECT g.id, t.designation FROM gaia g JOIN two_mass t \
ON XMATCH(radius_arcsec => 1.0)" \
--db /data/hats --cone 50,-50,2
# Crossmatch a raw CSV target list (named --open form) against a catalog.
acid query "SELECT t.id, g.source_id FROM t \
JOIN gaia_dr3 ON XMATCH(radius_arcsec => 1.0)" \
--db /data/hats --open t=candidates.csv,ra=RA,dec=DEC
# Cap the RAM the planner budgets for (work tuples shrink to fit).
acid query -f big.sql --db /data/hats --workers 32 --ram-budget 64GB
acid validate — parse + analyze, don't run¶
Same --db / query-source rules as acid query. Prints the analyzed
plan summary (anchor, joins, projection, aggregation, ordering,
footprint filters) and exits. Use it to catch ParseError /
ValidationError before kicking off a long run.
acid config — show or edit acid.conf¶
acid config show [--effective]
acid config get KEY [--effective]
acid config set KEY VALUE
acid config unset KEY
show lists the file's settings. --effective shows the resolved
values (with provenance: env > config > built-in) plus the search
order ACID walked.
get KEY prints the file value (exits 1 silently if unset) — use
--effective to print the resolved value instead.
set and unset rewrite the target file in place; comments and
formatting are not preserved.
Valid keys: path, download_path, workers, reader_workers,
mem_per_worker_gb, tmpdir, inmem_row_limit, workers_jemalloc_conf,
ram_budget, margin_arcsec, skymap_style, skymap_cmap.
acid search — list downloadable catalogs¶
Crawls every root on the download path ($ACID_DOWNLOAD_PATH →
config download_path → built-in default) over its native transport —
local directories, ssh:// hosts, and http(s):// mirrors alike — and
lists the HATS catalogs available to acid download. One line per
catalog; the name is the token to hand to acid download.
| Flag | Default | Description |
|---|---|---|
PATTERN |
— | Show only catalogs whose name contains this text (case-insensitive substring). |
--cache {use,refresh,off} |
use |
Remote-listing cache mode: use serves a fresh-enough cached listing (remote roots are cached ~1h under $XDG_CACHE_HOME/acid/downloads), refresh forces a re-crawl and rewrites the cache, off bypasses it entirely (neither read nor write). |
--timeout SEC |
300 |
Per-request timeout in seconds (HTTP / SSH). |
--insecure |
off | Skip TLS cert verification (HTTPS only — self-signed mirrors). |
--no-color |
off | Disable ANSI color. |
Output¶
Catalog rows go to stdout; the searched roots, the count, and any
shadowing footnote go to stderr as quiet context (so acid search | …
pipes only the data).
- On a terminal — an aligned, colored table, with a live braille
spinner and a running catalog count while the crawl is in flight. Each
row reads
NAME margins: <widths> arcsec(ormargins: none); the margin widths are the margin-cache radii (arcsec) available for the catalog. With more than one download root, the row also shows a compact root label; a catalog shadowed by an earlier root carries a trailing*. - When piped — clean tab-separated lines,
name⇥margins⇥root⇥marker, wheremarkerisshadowed(the resolved copy leaves it empty). This is stable for scripting.
Catalogs grouped under a namespace directory surface as namespace/child
(e.g. wise/allwise) — the exact token acid download accepts. HATS
collections are listed as plain catalog names; the collection structure
is read internally only to find their margins.
Shadowing. Every occurrence of a name across the roots is shown, but
acid download <name> resolves first-wins, so a same-named catalog at
a later root is flagged shadowed (trailing * on a TTY; the
shadowed 4th TSV column when piped). The summary reports the count, e.g.
✓ 12 catalogs (2 shadowed). Download a shadowed copy by its explicit
URL/path, not by name.
Examples¶
# Everything available on the download path.
acid search
# Only catalogs whose name contains "gaia".
acid search gaia
# Force a fresh crawl past the ~1h cache.
acid search --cache refresh
# Names of only the catalogs `acid download` would actually fetch
# (drop the shadowed ones, whose 4th column is non-empty).
acid search | awk -F'\t' '$4 == "" { print $1 }'
acid list — list catalogs you can open¶
The local twin of acid search.
It uses the same discovery engine, transports, flags, and output contract
— but crawls the catalog path ($ACID_PATH → config path →
built-in default ~/datasets) instead of the download path. So acid
search answers what can I download, while acid list answers what's
already here that I can open by name — the catalogs acid open /
acid query resolve a bare name against, first-wins across the roots.
The flags, the TTY-table-vs-piped-TSV output, the namespace/child names,
the margin-radii column, and the shadowing rules are all exactly as
documented under acid search,
with one wording change: the shadowing footnote names acid open (which
resolves the first match) rather than acid download.
Examples¶
# Every catalog you can open by name, with margin radii.
acid list
# Only the ones whose name contains "gaia".
acid list gaia
# Just the names (piped → TSV).
acid list | cut -f1
acid download — fetch a HATS catalog¶
acid download SOURCE [DEST]
[--cone RA,DEC,RADIUS_DEG]
[--columns COL,COL,...]
[--workers N] [--timeout SEC]
[--estimate] [--prefetch-metadata]
[--skip-margin] [--insecure]
[--tmpdir DIR]
| Flag | Default | Description |
|---|---|---|
SOURCE |
required | A catalog name to resolve against the download path, or an explicit source. A name is a bare token (gaia_dr3) or a nested namespace/child (wise/allwise, as acid search surfaces). An explicit source — a URL, an SSH user@host:path, or a local path with a leading ./ / / / ~ — is used verbatim. (To copy from a local relative directory, prefix it with ./; a bare token is always a name lookup.) |
DEST |
optional for a name | Destination directory. A path (containing a /) or URL is used as-is; a bare token resolves to <ACID_PATH-root>/<token>. Omit it for a name source — it defaults to <ACID_PATH-root>/<leaf> (a nested name's last segment, so wise/allwise → <ACID_PATH>/allwise). An explicit source requires an explicit dest. |
--cone RA,DEC,RADIUS_DEG |
— | Download only partitions overlapping this cone (degrees, ICRS). |
--columns COL,COL,... |
all | Download only the named columns (HATS-required columns like RA/Dec/_healpix_29 are always included). |
--workers N |
8 |
Parallel download workers. |
--estimate |
off | Print estimated bytes/files and exit without downloading. |
--prefetch-metadata |
off | Force fetching _metadata for exact sizes / bulk byte ranges. |
--skip-margin |
off | Don't download the catalog's margin cache. |
--timeout SEC |
300 |
Per-file timeout (HTTP only). |
--insecure |
off | Skip TLS cert verification (HTTPS only — self-signed certs, testing). |
--tmpdir DIR |
$TMPDIR |
Scratch directory for the point_map.fits mmap build. Keep on fast local storage when DEST is networked. |
Downloaded subsets are valid HATS catalogs with a rebuilt _metadata
and a regenerated point_map.fits footprint. If any file fails after
retries, the download exits non-zero — there is no "partial success"
mode, on purpose (a half-downloaded catalog looks structurally valid).
The built-in download path is two roots, searched first-wins:
https://data.lsdb.io/hats/ then ssh://slacd/sdf/home/m/mjuric/datasets.
Override it with $ACID_DOWNLOAD_PATH or an acid.conf download_path =
line. Use acid search to see
what each root offers and which copy a name resolves to.
Examples¶
# By name — resolved against the download path; lands under <ACID_PATH>/gaia_dr3.
acid download gaia_dr3
# Nested name — lands under its leaf, <ACID_PATH>/allwise.
acid download wise/allwise
# Full catalog over an explicit HTTP URL (explicit source needs an explicit dest).
acid download https://data.lsdb.io/hats/two_mass/two_mass /data/two_mass
# 5-deg cone, column subset, over SSH.
acid download user@server:/hats/gaia /data/gaia \
--cone 180,0,5 --columns ra,dec,phot_g_mean_mag
# Copy from a LOCAL directory: prefix ./ or / so it isn't a name lookup.
acid download ./mirror/two_mass /data/two_mass
# Estimate first.
acid download https://data.lsdb.io/hats/two_mass/two_mass /data/two_mass \
--cone 50,-50,2 --estimate
acid import — build a HATS catalog¶
# Pick the source kind: a file format, a Butler dataset, or peek.
acid import {parquet,csv,fits,arrow} SOURCES... --out PATH|NAME --ra COL --dec COL
[--schema FILE] [--columns A,B,...]
[--storage-option KEY=VALUE]... [--insecure] [--from-list URL|PATH]
[--mode {auto,inmem,shuffle}]
[--margin-arcsec R] [--no-margin]
[--workers N] [--reader-workers N] [--ram-budget BYTES] [--tmpdir DIR]
[--order K] [--max-order M] [--rows-per-partition N]
[--skymap-style {quadrant,blocks,braille,off}]
[--skymap-cmap {auto,gray,viridis,inferno,magma,plasma,none}]
[--overwrite]
acid import csv SOURCES... [--no-header] [--comment-char CHAR] [--delimiter CHAR] …
acid import butler --repo REPO --collection COLL --dataset DS [--where EXPR] --out … --ra … --dec … …
acid import peek SOURCES... [--list] [--storage-option …] [--from-list URL|PATH] [--no-header] [--delimiter …]
Reads tabular files (FITS, CSV/TSV, parquet, Arrow) — local or on a
remote fsspec store — or a Rubin Butler dataset, and writes a HATS
catalog, building a margin cache afterward by default. The first word
names the source kind: a file format (parquet/csv/fits/arrow), a
Rubin butler dataset, or peek (inspect without importing). Naming the
format keeps each format's options under its own verb and lets a file
whose extension doesn't say its type still import (acid import csv
survey.dat). The import path is chosen from the input size: in memory
when it fits, else an out-of-core shuffle (remote inputs always shuffle).
See the importing guide for the task-shaped
walkthrough.
Flags shared by every file-format verb (parquet/csv/fits/arrow)
and butler:
| Flag | Default | Description |
|---|---|---|
SOURCES |
required (or --from-list) |
One or more files, directories, or quoted globs — local paths or remote scheme:// URLs (http(s):// out of the box; s3:// / gs:// / az:// with the backend package installed). Read as the chosen format, with the same columns (order may differ). A directory/glob imports every data file inside; over http(s):// the server's directory index (autoindex) is scraped, non-data links ignored — for a mirror with no index, use --from-list. (Not for butler.) |
--out PATH\|NAME |
required | Output catalog. A bare NAME joins the catalog library (<first writable ACID_PATH root>/NAME); a path (has a /) is used verbatim. |
--ra COL / --dec COL |
required | RA / Dec column names. Never guessed — use acid import peek to find them. |
--schema FILE |
inferred | YAML/JSON declaring column types (and order). Recommended for CSV. Positional form (name: {at: N, type: …}) names headerless columns. |
--columns A,B,... |
all | Output columns + order. RA/Dec kept automatically. Slims wide catalogs (e.g. dropping FITS array columns). |
--storage-option KEY=VALUE |
— | fsspec backend option for a remote SOURCES (repeatable; true/false/null/integers coerced). E.g. anon=true, endpoint_url=https://…, key=…/secret=…. (Not for butler.) |
--insecure |
off | Skip TLS cert verification for a remote HTTPS SOURCES (self-signed certs / private mirrors only). (Not for butler.) |
--from-list URL\|PATH |
— | Read the input file list from a manifest instead of a directory index (for HTTP mirrors that serve none). A newline-delimited list of names/URLs, or a SHA256SUMS-style checksums file (auto-detected); # comments/blank lines skipped; relative names resolve against the manifest's dir. SOURCES may then only be glob pattern(s) that filter the list (or be omitted to import every listed file). (Not for butler.) |
--mode {auto,inmem,shuffle} |
auto |
auto picks in-memory vs out-of-core by input size (remote inputs always shuffle). Force a path for testing. (File verbs only; butler is always shuffle.) |
--margin-arcsec R |
10.0 |
Radius for the auto-built margin cache. Make it ≥ the widest XMATCH radius you'll run against the catalog. |
--no-margin |
off | Skip the margin build (do it later with acid hats build-margin). |
--workers N |
auto | Parallel workers (from RAM/cores). |
--reader-workers N |
same as --workers |
Workers for the read + shuffle phase (Pass A) only; the merge + write phase stays at --workers. Raise for a read-bound source, lower to cap reader memory. Resolves --reader-workers → $ACID_READER_WORKERS → reader_workers (config) → --workers. Shuffle mode only. |
--ram-budget BYTES |
0.5 × available |
RAM for the in-core shuffle buffer (e.g. 64GB). |
--tmpdir DIR |
$TMPDIR |
Scratch base for out-of-core spills. |
--order K / --max-order M / --rows-per-partition N |
6 / 10 / 1e6 |
Shuffle-mode HEALPix-order and partition-size knobs. |
--skymap-style STYLE |
quadrant |
Style of the end-of-import sky map (drawn on a TTY): quadrant (area-faithful), blocks (classic ░▒▓█ density ramp), braille (finer), or off. Resolves flag → $ACID_SKYMAP_STYLE → skymap_style (config). |
--skymap-cmap CMAP |
auto |
Density colormap for the sky map: auto (magma on a dark terminal, gray on a light one — detected via OSC 11), explicit gray/viridis/inferno/magma/plasma, or none for monochrome. Resolves flag → $ACID_SKYMAP_CMAP → skymap_cmap (config). |
--overwrite |
off | Replace an existing catalog at --out. |
acid import csv adds the CSV/TSV read options:
| Flag | Default | Description |
|---|---|---|
--no-header |
off | CSV/TSV has no header row; needs a positional --schema. |
--comment-char CHAR |
— | Skip CSV/TSV lines starting with CHAR (e.g. #). |
--delimiter CHAR |
, / tab |
CSV/TSV field separator (e.g. \|). Defaults: comma for .csv, tab for .tsv. |
acid import butler (requires the LSST stack; always uses the shuffle):
| Flag | Default | Description |
|---|---|---|
--repo / --collection / --dataset |
required (together) | Rubin Butler repo path/alias, collection, and dataset type to import. A glob --collection imports the UNION of the dataset type across all matching collections (deduped by data ID, newest wins); --dataset must be a single exact type. |
--where EXPR |
— | Butler query expression limiting which datasets are pulled (e.g. "band='r'"). |
--from-list PATH\|- |
— | Import only the datarefs in a acid import peek butler … --list JSONL manifest (- = stdin). Each line is self-describing, so --collection/--dataset/--where don't apply (and are rejected); only --repo is needed. The manifest must list a single dataset type. |
acid import peek SOURCES... summarizes the input (format, rows, column
schema with each column's estimated byte size, a Bytes/row total,
candidate RA/Dec, and the first 10 matched files) and exits without
writing — auto-detecting the format, reading metadata + a small sample only.
The byte sizes are measured where it's cheap (FITS column formats, the
parquet footer, the CSV/Arrow sample) and a bucket default otherwise, so a
list/array column shows its real per-row size; multiply Bytes/row by the
row count to size --workers / --ram-budget.
With --list it instead prints the full matched-file list (one per line —
local paths relative to the cwd, remote URLs verbatim; plain when redirected,
a count header on a terminal), for checking a fat glob / remote listing before
committing. It takes --storage-option / --insecure / --from-list for
remote sources and the CSV read options so its sample matches a real CSV import.
Like acid download, an import either completes or fails loudly — it
never leaves a half-written catalog. A bare-name --out lands under
ACID_PATH in both the in-memory and out-of-core paths.
Examples¶
# Inspect an input before importing — what are the coordinate columns?
acid import peek tractor-1126p222.fits
# Check exactly which files a directory / glob matched (scriptable when piped).
acid import peek '/data/legacy/tractor/**/*.fits' --list | wc -l
# One FITS file → a named catalog in the library (margin built by default).
acid import fits tractor-1126p222.fits --out legacy_dr10 --ra ra --dec dec
# A directory of FITS bricks → an explicit path, slimmed to a few columns.
acid import fits /data/legacy/tractor/ --out /data/hats/legacy --ra ra --dec dec \
--columns ra,dec,type,flux_g,flux_r,flux_z
# Gzipped, '#'-commented CSV with a declared schema.
acid import csv gaia_chunk.csv.gz --out gaia_chunk --ra ra --dec dec \
--comment-char '#' --schema gaia.schema.yaml
# Pipe-delimited catalog, no header, positional schema.
acid import csv psc.tbl --out two_mass_psc --ra ra --dec dec \
--delimiter '|' --no-header --schema psc.schema.yaml
# Remote: a quoted S3 glob from a public bucket (peek it first, then import).
acid import peek 's3://open-data/cat/*.parquet' --storage-option anon=true
acid import parquet 's3://open-data/cat/*.parquet' --out cat --ra ra --dec dec \
--storage-option anon=true
# Remote http directory (autoindex scraped) and, for an index-less mirror, a manifest.
acid import parquet https://example.org/dr1/ --out dr1 --ra ra --dec dec
acid import parquet --from-list https://example.org/dr1/SHA256SUMS --out dr1 --ra ra --dec dec
# From a Rubin Butler.
acid import butler --repo /repo/main \
--collection LSSTComCam/runs/DRP/DP1 --dataset object \
--out dp1_object --ra coord_ra --dec coord_dec --where "band='r'"
# Curate a set of datarefs with `peek … --list`, then import exactly those.
acid import peek butler --repo /repo/main \
--collection 'LSSTComCam/runs/DRP/*' --dataset object --list \
| acid import butler --repo /repo/main --from-list - \
--out dp1_object --ra coord_ra --dec coord_dec
acid inspect — show catalog info¶
MODE is one of summary (default), schema, or properties.
SOURCE is a local path, HTTP URL, or user@host:path SSH path.
acid inspect /data/two_mass # summary
acid inspect schema /data/two_mass # column schema
acid inspect properties /data/two_mass # HATS properties
acid inspect https://data.lsdb.io/hats/two_mass/two_mass
acid hats — HATS catalog operations¶
acid hats build-margin CATALOG
[--margin-arcsec ARCSEC]
[--workers N] [--output DIR]
[--overwrite] [--mem-limit GB]
[--tmpdir DIR]
Builds a HATS margin cache for CATALOG. Output passes
hats.read_hats(...) and matches the upstream hats-import tool.
| Flag | Default | Description |
|---|---|---|
CATALOG |
required | Path to a local HATS catalog. |
--margin-arcsec ARCSEC |
10.0 |
Margin threshold in arcseconds. Must be at least as large as the largest XMATCH radius you'll ever run against this catalog (see margin caches). |
--workers N |
cgroup-aware cpu_cap |
Parallel workers. |
--output DIR |
<catalog>_<margin>arcsec |
Output directory. |
--overwrite |
off | Overwrite an existing margin cache at --output. |
--mem-limit GB |
10 % of RAM | Memory limit before the accumulator spills to disk. |
--tmpdir DIR |
spill under the output directory | Base directory for the spill scratch (must already exist). Point at fast local storage when --output is on a slow networked filesystem. |
Example¶
Environment variables¶
A few env vars influence CLI behavior; the full list (including the
allocator and worker-startup knobs) is in
MEMORY-TUNING.md.
The ones a user is most likely to set:
| Variable | Effect |
|---|---|
ACID_PATH |
Default value for --db (overrides config path). |
ACID_WORKERS |
Default for --workers. |
ACID_MEM_PER_WORKER_GB |
RAM/worker bound for --workers auto. |
ACID_RAM_BUDGET |
Default for --ram-budget (bytes or 64GB/512MiB). |
ACID_TMPDIR |
Default for --tmpdir. |
ACID_INMEM_ROW_LIMIT |
Spill threshold (rows). |
ACID_CONFIG |
Path to a specific acid.conf. |
ACID_PROGRESS |
0 to silence, 1 to force the bar, plain for committed lines per stage. |
ACID_PROFILE |
1 to print a per-step profile to stderr. |
ACID_PROFILE_OUT |
Path for the full per-worker profile JSON. |