Opening remote catalogs¶

acid.open(<url>) opens a HATS catalog (or a single raw file) that lives on a remote store — a public HTTPS mirror, an object store, an SSH host — and queries it without downloading the whole thing first. The catalog's small metadata is fetched once; the partition data is pulled lazily, only for the columns and partitions a query actually touches, and cached locally so repeat queries (and later sessions) read from disk.

Use this when you want to explore or query a catalog you don't have locally, and your queries touch only a fraction of it — a cone, a few columns, a crossmatch against a small target list. When you'd rather pull the whole catalog (or an explicit cone / column slice) to disk up front, use acid download instead — see When to use this vs acid download.

This page covers:

The shortest path: open and query a public catalog over HTTPS.
Where the cache lands (ACID_PATH), the .acid.remote marker, and re-opening by name / working offline.
Backends and credentials (storage_options=).
Choosing a cache mode (cache="blocks" | "none" | "whole").
Remote single files (raw parquet/CSV/FITS over a URL).

Quick start¶

data.lsdb.io hosts public HATS catalogs over anonymous HTTPS. Point acid.open straight at one — no credentials, no prior download:

open_remote.py

import acid

cat = acid.open("https://data.lsdb.io/hats/vsx")   # a public HATS catalog
print(cat.columns)                                  # works from cached metadata

acid.open(<url>) fetches the catalog's metadata (its properties, partition_info.csv, schema, and point_map.fits footprint) into a local cache directory, then returns a Catalog handle. No partition data has been read yet — cat.columns, cat.describe(), and cat.explain() all answer from the cached metadata alone.

A query downloads only what it needs. Here, a tight cone selecting two columns pulls just those columns' bytes, and only for the partitions the cone overlaps:

open_remote.py

import astropy.units as u

with acid.in_cone((305.0, -10.0), radius=0.2 * u.deg):
    rows = (
        cat.select("RAdeg", "DEdeg", "Type")
           .limit(20)
           .to_polars()
    )

print(rows)

The cone (ICRS, here a 0.2° radius) prunes the catalog to the handful of partitions it overlaps; the .select(...) narrows the read to three columns. Only those columns of those partitions are fetched over the network — a few MB, not the whole catalog. The fetched bytes are cached, so re-running the cell (or a different query over the same region) reads from local disk.

Lead with a cone and a select on a remote catalog

The whole point of acid.open(<url>) is to transport only the bytes a query touches. A query with no spatial restriction over all columns will fetch the whole catalog on first run — at which point you may as well have used acid download. Scope by cone (or .in_region(...)) and project the columns you need.

The same catalog works with the SQL escape hatch. acid.open(<url>) registers the catalog under its leaf name (here vsx), so the SQL parser can see it:

FluentSQL

with acid.in_cone((305.0, -10.0), radius=0.2 * u.deg):
    rows = cat.select("RAdeg", "DEdeg", "Type").limit(20).to_polars()

acid.open("https://data.lsdb.io/hats/vsx")   # registers it as "vsx"

with acid.in_cone((305.0, -10.0), radius=0.2 * u.deg):
    r = acid.sql.query(
        "SELECT RAdeg, DEdeg, Type FROM vsx LIMIT 20"
    )
    rows = r.to_polars()

Where the cache goes¶

A remote HATS catalog is mirrored into a cache directory under the first writable ACID_PATH root, named after the catalog. Opening https://data.lsdb.io/hats/gaia_dr3 caches into <ACID_PATH>/gaia_dr3/. That directory is a real HATS catalog tree:

the catalog metadata, fetched whole on first open (properties, partition_info.csv, dataset/_common_metadata, point_map.fits, plus collection and margin-cache metadata);
the partition data fetched so far — column-sparse, accumulated as queries touch new partitions and columns;
a small marker file, .acid.remote, recording the source URL (and a non-secret subset of storage_options, e.g. anon — see Credentials).

A remote collection is flattened, exactly like a full download: the primary table lands directly in the cache directory (a plain catalog), and its margin cache becomes a sibling <name>_<width>arcsec (e.g. gaia_dr3_10arcsec/), each with its own .acid.remote marker. There is no nested collection wrapper — <ACID_PATH>/gaia_dr3/ is the catalog.

You choose where it lands — there is no silent fallback¶

If no ACID_PATH root is writable, acid.open(<url>) fails loudly and tells you to add a writable directory to ACID_PATH. This is deliberate: a remote pull can grow to many gigabytes as you query, and ACID will not guess a location for that on your behalf.

export ACID_PATH=/data/hats        # a directory you can write to

acid.init("/data/hats")            # or rely on $ACID_PATH
cat = acid.open("https://data.lsdb.io/hats/gaia_dr3")

Re-opening by name, and working offline¶

Because the cache is a real catalog directory carrying a .acid.remote marker, you can later open it by its bare name once it resolves under your ACID_PATH:

acid.init("/data/hats")
cat = acid.open("gaia_dr3")        # transparently reuses the cache dir

A bare-name re-open reuses whatever is already cached and only reaches the network for partitions/columns it hasn't fetched yet. A fully warmed catalog — everything your queries touch already on disk — behaves exactly like a normal local HATS catalog, and works fully offline: if every byte a query needs is cached, no network call is made.

The cache is keyed to your ACID_PATH, not the URL string

A second machine, or a different ACID_PATH, starts with an empty cache and re-fetches metadata on first open. The cache is local state, not a shared artifact — to move a warmed catalog between machines, copy the cache directory like any other HATS catalog (it is one).

Registering from the shell: `acid download --incremental`¶

acid.open(<url>) registers the catalog the first time you open it in Python. To set one up from the shell — so it's ready to query by name, in a notebook or another process — use acid download with --incremental. It does the same metadata-mirror-and-marker setup, fetches no partition data, and prints where it landed:

acid download s3://stpubdata/panstarrs/ps1/public/hats/otmo ps1_objects \
    --incremental --storage-option anon=true

Registered catalog 'ps1_objects' for incremental access (metadata only — no data downloaded)
  cache dir: /data/hats/ps1_objects
  source:    s3://stpubdata/panstarrs/ps1/public/hats/otmo
  10,560,724,292 rows · 132 columns · 9,577 partitions · ~1.9 TB (not downloaded)
  Query it:  acid query "SELECT … FROM ps1_objects …"   (data is fetched on demand and cached)

Then query it by name; data is pulled column-sparse on demand:

acid query "SELECT raMean, decMean FROM ps1_objects WHERE ..."

--incremental takes the same source [dest] arguments as a full download (a bare dest lands under your first writable ACID_PATH; a path is used as-is), and accepts any fsspec URL — including s3://, which a full download does not. --storage-option KEY=VALUE (repeatable) supplies fsspec options; the non-secret ones (e.g. anon=true) are recorded in the marker so later queries authenticate without re-supplying them. Re-running --incremental with new --storage-options refreshes them in place (the way to fix a catalog registered without the access options it needs).

Backends and credentials¶

url is any scheme:// URL that fsspec can read:

Scheme	Backend	Notes
`https://` / `http://`	built-in	Public mirrors need no credentials.
`s3://`	`s3fs`	AWS S3 and S3-compatible stores.
`gs://`	`gcsfs`	Google Cloud Storage.
`abfs://`	`adlfs`	Azure Blob / Data Lake.
`ssh://` / `sftp://`	`paramiko`	SSH/SFTP hosts.
`file://`	built-in	A local path via a URL (mostly for testing).

The object-store backends (s3fs, gcsfs, adlfs, paramiko) are not ACID dependencies — install the one your scheme needs.

Pass credentials and endpoints through storage_options=, a dict forwarded verbatim to the fsspec backend:

# S3 with explicit keys
cat = acid.open(
    "s3://my-bucket/hats/my_catalog",
    storage_options={"key": "AKIA...", "secret": "..."},
)

# Anonymous public S3 bucket
cat = acid.open(
    "s3://nasa-irsa-something/hats/allwise",
    storage_options={"anon": True},
)

Public HTTPS (like data.lsdb.io) needs none — it's read anonymously.

Credentials are never written to disk

The .acid.remote marker stores the source URL only. When you re-open a credentialed remote (by URL or by its cached name), ACID re-reads it through your ambient fsspec credentials (environment, ~/.aws/credentials, instance role, …) or an explicit storage_options= you pass again. Secrets passed once are not persisted and not recoverable from the cache.

Choosing a cache mode¶

cache= controls what gets persisted as queries read partition data. The default ("blocks") is right for the common case — repeated queries against a wide catalog.

`cache=`	What it does	Use when
`"blocks"` (default)	Lazy, column-sparse, persistent block cache: fetch only the columns/partitions a query touches, cache them, reuse across queries and sessions.	Repeated queries against a wide catalog; you read a small slice of columns.
`"none"`	Read byte ranges straight from the remote each time; nothing persisted locally.	A fast object store next to compute; disk-constrained nodes; one-off queries.
`"whole"`	Fetch each touched partition file whole (all columns) into the cache, then read locally.	You expect to read most columns; you want a reusable local copy of the partitions you touch.

# Default: lazy, column-sparse, cached.
cat = acid.open("https://data.lsdb.io/hats/gaia_dr3")

# No local cache — read ranges live each time (e.g. on a compute node
# sitting next to a fast S3 bucket).
cat = acid.open("s3://my-bucket/hats/gaia_dr3", cache="none")

# Fetch whole partition files as they're touched.
cat = acid.open("https://data.lsdb.io/hats/gaia_dr3", cache="whole")

blocks vs whole: column width is the deciding factor

On a 1300-column Rubin object catalog where a query reads three columns, cache="blocks" fetches a few percent of each touched partition's bytes. cache="whole" fetches the entire partition file — wasteful if you only ever read three columns, but exactly right if you'll come back to read many columns and want the partition local once.

Seeing what came from the cache¶

Set ACID_CACHE_STATS=1 to print a one-line breakdown, on stderr, of how much of a query's partition data was served from the local cache versus pulled over the network:

$ ACID_CACHE_STATS=1 acid query "select objID, raMean, decMean FROM ps1_objects" --cone 10,20,1
acid: cache: 1.2 GB from local cache, 0 bytes over network (100% cache hit)
done. output: ...

A fully-warmed catalog reads everything locally (100% cache hit, 0 bytes over network); a query that touches a not-yet-cached column or partition shows the bytes it had to download. The diagnostic is off by default and adds nothing to the read path when unset; it covers cache="blocks" traffic (the default — cache="none" does no local caching, so there is nothing to report).

Remote single files¶

acid.open(<url>, ra=..., dec=...) also opens a single raw file over a URL — the same on-ramp as a local raw file (see Raw files and in-memory frames), but the file lives remotely. As with any raw file, ra= / dec= are required and never guessed (degrees, ICRS); the file is spilled to a virtual catalog at open():

import acid

targets = acid.open(
    "https://example.org/data/my_targets.parquet",
    ra="ra", dec="dec",
)
# behaves like any virtual catalog from here on — crossmatch it, query it

Unlike a HATS catalog, a single file has no partitions, so the column-sparse block cache doesn't apply — the right caching unit is the whole file. So unless you pass cache="none", a remote raw file is persisted whole under ACID_PATH (in an .acid_files/ subdirectory) the first time you open it, and re-opens read the local copy instead of re-downloading.

Raw-file caching is best-effort: a single file can always be read straight from its URL, so if no ACID_PATH root is writable, ACID skips the cache and reads the URL directly (no error — unlike a HATS catalog, which requires a local metadata mirror and fails loudly without one).

When to use this vs `acid download`¶

Both bring a remote catalog within reach; they differ in when the bytes move.

	`acid.open(<url>)`	`acid download`
When data moves	Lazily — only the partitions/columns a query touches, on demand.	Up front — the whole catalog (or an explicit `--cone` / `--columns` slice) before you query.
Result on disk	A growing cache, warmed by your queries.	A complete, self-contained HATS catalog.
Best for	Exploration; queries that touch a fraction of the data; one-off cross-survey lookups.	Repeated full-catalog work; offline/airgapped use; sharing a fixed local copy.
Spatial / column slice	Per-query (cone, `.in_region`, `.select`).	Per-download (`--cone`, `--columns`), then a normal local catalog.

A practical pattern: acid.open(<url>) to explore and confirm a catalog has what you need (schema, footprint, a quick cone query), then acid download it if you find you'll be reading most of it repeatedly. A warmed cache="blocks" directory and an acid download output are both just local HATS catalogs — the difference is how completely they're filled.