Experimental chunked compression

Experimental chunked compression#

Status#

Chunked compression is an experimental CZI subblock compression mode. Its binary format and libCZI API surface may still change, and support is only available in builds where the experimental feature is enabled.

The CZI compression mode is CompressionMode::ChunkedExtensible with the raw compression-mode value 7. The informal compression string used by libCZI is chunked.

Builds that enable the feature define LIBCZI_EXPERIMENTAL_CHUNKED_COMPRESSION_AVAILABLE to 1 in libCZI_Config.h. Builds that disable the feature define it to 0; callers should use this macro before referencing chunked-compression-only APIs.

Motivation#

The existing zstd compression modes store a subblock as one compressed payload. Chunked compression instead divides the logical pixel stream into bounded, independently compressed chunks and stores the chunk sizes in a small header.

This is intended to support implementations that work concurrently at tile or chunk level. It also makes it easier to use hardware-accelerated codec backends, for example GPU-accelerated zstd or LZ4 implementations such as nvCOMP, where batched compression and decompression of many independent chunks is a natural execution model.

The format is a container around an actual per-chunk codec. The currently defined per-chunk codecs are zstd and LZ4.

Build configuration#

Chunked compression is controlled by two CMake options:

  • LIBCZI_BUILD_ENABLE_EXPERIMENTAL_FUNCTIONALITY: Global policy for experimental features. The supported values are AUTO, ON, and OFF. AUTO is the conservative default and keeps chunked compression disabled unless the feature-specific option is set.

  • LIBCZI_BUILD_EXPERIMENTAL_CHUNKED_COMPRESSION: Feature-specific switch for chunked compression. When this option is set explicitly, it takes precedence over the global experimental policy.

For example:

cmake .. -DLIBCZI_BUILD_EXPERIMENTAL_CHUNKED_COMPRESSION=ON

or, to enable all experimental features by default:

cmake .. -DLIBCZI_BUILD_ENABLE_EXPERIMENTAL_FUNCTIONALITY=ON

When chunked compression is enabled, libCZI also needs LZ4. The option LIBCZI_BUILD_PREFER_EXTERNALPACKAGE_LZ4 controls whether CMake should use an externally provided LZ4 package or fetch and build a private copy.

Using the mode#

The compression options parser accepts the informal mode name chunked. The chunked-specific options are:

  • ChunkedMaxChunkSize: Maximum uncompressed chunk size in bytes.

  • ChunkedCodec: Per-chunk codec. Supported values are zstd and lz4.

  • ExplicitLevel: zstd compression level when ChunkedCodec=zstd.

  • PreProcess=HiLoByteUnpack: Request hi/lo byte packing preprocessing where applicable.

Examples:

chunked:ChunkedCodec=zstd;ChunkedMaxChunkSize=65536
chunked:ChunkedCodec=lz4;ChunkedMaxChunkSize=65536
chunked:ChunkedCodec=zstd;ChunkedMaxChunkSize=65536;ExplicitLevel=3

Format overview#

A chunked-compressed CZI subblock payload consists of a chunked-compression header followed immediately by the compressed chunk data:

+-----------------------------+
| chunked-compression header  |
+-----------------------------+
| compressed data chunk 0     |
+-----------------------------+
| compressed data chunk 1     |
+-----------------------------+
| ...                         |
+-----------------------------+
| compressed data chunk n     |
+-----------------------------+

The compressed chunks have no per-chunk frame in the payload. Chunk boundaries are derived from the compressed sizes stored in the header. The uncompressed sizes stored in the header describe the destination layout after decompression and before any size-mismatch resolution protocol is applied.

The logical input stream is the image data described by width, height, and pixel type, using the minimal row stride width * bytes-per-pixel. Any padding in a caller-provided source stride is not part of the logical stream. The compressor splits this logical stream into chunks of at most ChunkedMaxChunkSize; chunk boundaries do not need to align to image rows.

Integer encoding#

Header fields use a little-endian seven-bit-group varint encoding. Each byte contributes seven payload bits. The high bit indicates that another byte follows except on the final byte. The least significant seven bits are stored first.

The implementation uses bounded variants of this encoding:

Field Encoded size Value range
Header chunk ID 1-2 bytes 0 to 16,383
Header chunk payload length 1-3 bytes 0 to 4,194,303
Compressed chunk size 1-4 bytes 0 to 536,870,911
Decompressed chunk size 1-4 bytes 0 to 536,870,911

Header structure#

The header is a sequence of header chunks terminated by an end marker.

For every header chunk except the end marker, the layout is:

header-chunk-id | payload-length | payload

header-chunk-id is a 1-2 byte varint. payload-length is a 1-3 byte varint giving the byte size of the payload. The end marker is the header chunk ID 0 encoded by itself; it has no length and no payload.

The currently defined header chunk IDs are:

ID Name Required Payload
0 EndOfHeader yes none
1 ChunkSizes yes compressed chunk sizes
2 CompressionMethod no per-chunk codec
3 DecompressedSizes yes uncompressed chunk sizes
4 Preprocessing no preprocessing method

libCZI’s writer currently emits header chunks in this order:

ChunkSizes
DecompressedSizes
CompressionMethod, only when the codec is not zstd
Preprocessing, when explicitly specified
EndOfHeader

The parser interprets chunks by ID, not by position. Unknown header chunk IDs are rejected by the current implementation.

ChunkSizes header chunk#

ChunkSizes has ID 1 and is required. Its payload is a sequence of 1-4 byte varints, one per compressed data chunk, in chunk order:

compressed-size[0], compressed-size[1], ..., compressed-size[n]

The number of values in this header chunk is the number of compressed data chunks following the header.

DecompressedSizes header chunk#

DecompressedSizes has ID 3 and is required. Its payload is a compact sequence of 1-4 byte varints describing uncompressed chunk sizes.

The common forms are:

[C]

All chunks decompress to C bytes.

[C, L]

All chunks except the last decompress to C bytes. The last chunk decompresses to L bytes.

The parser also supports the more general suffix form used by the implementation: values are listed in chunk order, and if fewer values are present than chunks, the second-to-last value repeats for the earlier chunks that are not explicitly listed. The writer uses the one-value or two-value forms.

CompressionMethod header chunk#

CompressionMethod has ID 2 and is optional. Its payload is exactly one byte.

Value Codec
0 zstd
1 LZ4

If the header chunk is absent, the codec defaults to zstd. A chunked-compressed subblock uses one codec for all chunks in that subblock.

Preprocessing header chunk#

Preprocessing has ID 4 and is optional. Its payload is exactly one byte.

Value Meaning
0 no preprocessing
1 hi/lo byte packing was applied before compression

If the header chunk is absent, no preprocessing is assumed.

Hi/lo byte packing is used for pixel formats where byte-plane reordering can improve compression, such as Gray16 and Bgr48. A decoder first decompresses the chunks and then reverses this preprocessing step before returning bitmap data.

Decoder validation#

A decoder should:

  1. Walk the header until EndOfHeader.

  2. Parse ChunkSizes and DecompressedSizes.

  3. Apply defaults for omitted optional chunks: zstd for CompressionMethod and no preprocessing for Preprocessing.

  4. Validate that the codec and preprocessing values are supported.

  5. Verify that the compressed chunk sizes fit within the subblock payload.

  6. Decompress each chunk independently in chunk order.

  7. Reverse hi/lo byte packing when requested by the header.

For normal data written by libCZI, the sum of the decompressed chunk sizes should match the expected bitmap size computed from the subblock pixel type, width, and height. libCZI also contains a compatibility path for selected size-mismatch cases, controlled by the existing chunked-compression data-size mismatch option.