HCS NGFF Conversion Workflow#

The ngff-zarr HCS exporter converts a CZI plate into an OME-NGFF v0.5 or v0.6, Zarr v3 hierarchy. It writes v0.5 by default; v0.6 can be selected explicitly with version="0.6" and requires ngff-zarr>=0.47.0. The optimized path is designed around three constraints:

• parse expensive CZI metadata once and reuse it;
• read stored CZI pyramid levels directly, using native libCZI zoom only when the source has no suitable coarse level;
• overlap a bounded number of field reads and writes without materializing the complete plate in memory.

Conversion pipeline#

flowchart TD
    input[Input CZI plate] --> metadata[Parse CziMetadata once]
    metadata --> layout[Resolve plate, well, field, and scene layout]
    metadata --> pyramid[Detect stored CZI pyramid levels]
    layout --> queue[Create bounded field job queue]
    pyramid --> queue
    queue --> scene[Select one scene for one field]
    scene --> levels[Build lazy multiscale arrays]
    levels --> decision{Stored coarse level available?}
    decision -- Yes --> stored[Read stored CZI subblocks]
    decision -- No --> native[Read lower level with native libCZI zoom]
    stored --> chunks[Rechunk spatial axes toward 512 x 512]
    native --> chunks
    chunks --> ngff[Create NGFF multiscales and OMERO metadata]
    ngff --> writer[ngff-zarr HCSPlateWriter]
    writer --> shard[Configure Zarr v3 indexed sharding]
    shard --> compress[Compress inner chunks with Blosc]
    compress --> field[Write plate / well / field / scale arrays]
    field --> remaining{More fields?}
    remaining -- Yes --> queue
    remaining -- No --> validate[Validate OME-NGFF v0.5 output]

The exporter resolves the complete HCS layout before scheduling writes. Each job selects one scene and calls read_stacks_multiscale() for that field. The returned Dask arrays remain lazy until ngff-zarr writes them.

For every requested level:

1. A stored CZI pyramid level is used when available.
2. Otherwise, libCZI creates the level during the CZI read with its native zoom implementation. The exporter does not read level 0 and downsample it later in Python.
3. Y and X are rechunked toward 512 x 512 pixels. ngff-zarr clamps chunk and shard geometry to the actual array shape.
4. ngff-zarr writes OME-NGFF metadata and Zarr v3 arrays through zarrista.
5. chunks_per_shard={"y": 4, "x": 4} requests up to four inner chunks per spatial shard axis. Blosc compresses the inner chunks.

TensorStore is not part of this path. ngff-zarr uses zarrista and receives filesystem or .ozx paths directly. HCS validation selects the version-aware HCS reader for v0.6 stores and retains the v0.5 model validation path.

Metadata reuse#

Constructing CziMetadata repeatedly is expensive for large HCS files. The exporter creates it once, then passes it through read_stacks_multiscale() to each read_stacks() call.

read_stacks() uses a shallow metadata clone and copies only the scale object that it updates for the current zoom. It also copies image dimensions when adapt_metadata=True. The large parsed metadata graph and bounding-box model remain shared and read-only. This avoids both repeated CZI metadata parsing and a full deepcopy() for every field and pyramid level.

Bounded concurrency and memory#

flowchart LR
    czi[(One shared CZI file)] --> queue[Field job queue]
    queue --> worker1[Worker 1]
    queue --> worker2[Worker 2]
    queue --> worker3[Worker 3]
    queue --> worker4[Worker 4]

    subgraph active[At most max_workers active fields]
        worker1 --> graph1[Lazy field graph and active chunks]
        worker2 --> graph2[Lazy field graph and active chunks]
        worker3 --> graph3[Lazy field graph and active chunks]
        worker4 --> graph4[Lazy field graph and active chunks]
    end

    graph1 --> store[(Sharded Zarr v3 plate)]
    graph2 --> store
    graph3 --> store
    graph4 --> store

    metadata[Shared parsed metadata] -. shallow per-read copy .-> graph1
    metadata -. shallow per-read copy .-> graph2
    metadata -. shallow per-read copy .-> graph3
    metadata -. shallow per-read copy .-> graph4

    note[The complete plate is never materialized in RAM]
    active --- note

max_workers=4 is the default. Four field jobs may therefore build and write their lazy graphs concurrently. This overlaps CZI reads, compression, and filesystem writes while bounding the number of active fields.

This differs from loading the complete plate into one in-memory array. Peak memory depends on field shape, pixel type, T/C/Z depth, chunk geometry, codec buffers, and the number of concurrent workers. As a practical approximation, increasing max_workers can increase the active field-processing memory by a similar factor, although Dask and the writer operate chunk by chunk.

Use the default for typical HCS fields. Set max_workers=1 when fields are very large, T or Z is deep, memory is constrained, or the source storage performs poorly under concurrent random reads. Intermediate values such as 2 provide a middle ground.

Storage layout#

The output hierarchy is:

plate/
  <row>/<column>/
    <field>/
      scale0/<image-name>
      scale1/<image-name>
      ...

Each image array uses dimension order (t, c, z, y, x). For Zarr v3 output, indexed sharding groups compressed inner chunks into fewer outer objects. This reduces filesystem metadata overhead and file count; it does not itself reduce CZI decoding work.

For a 2000 x 2000 field, a nominal spatial chunk size of 512 and four chunks per shard can produce one spatial shard with approximately 500 x 500 inner chunks. For smaller fields, ngff-zarr clamps the geometry. A 708 x 980 field, for example, can use a 708 x 980 outer shard with 354 x 490 inner chunks.

API example#

from pathlib import Path

from czitools.export_tools import convert_czi2hcs_ngff, validate_ome_zarr

output = convert_czi2hcs_ngff(
    "plate.czi",
    output_dir=Path("exports"),
    spatial_chunk_size=512,
    chunks_per_shard={"y": 4, "x": 4},
    planes_per_chunk=64,
    max_workers=4,
)

if not validate_ome_zarr(output):
    raise RuntimeError(f"OME-NGFF validation failed: {output}")

Performance interpretation#

Conversion time is workload- and hardware-specific. The largest improvements come from eliminating repeated metadata parsing and full metadata deep copies. Stored or native CZI pyramid reads avoid Python-side resampling, while bounded field concurrency and sharding improve throughput and storage behavior.

Always compare outputs as well as timings. A useful verification includes:

• the expected plate, well, field, and scale counts;
• OME-NGFF schema validation;
• array shapes, data types, dimension names, and coordinate transforms;
• representative pixel comparisons against direct CZI reads;
• output size, file count, compression, and sharding codecs.