Class ChunkedCompressionHeaderHelper

Class ChunkedCompressionHeaderHelper#

Nested Relationships#

Nested Types#

Class Documentation#

class ChunkedCompressionHeaderHelper#

Here we gather utilities for working with chunked-compression headers. The concept of the chunked-compression scheme is to have a header which describes the structure of the compressed data in terms of “chunks”, and then the compressed data is organized in a sequence of chunks, where each chunk contains compressed data for a part of the uncompressed data. The header contains information about the size of each chunk, the compression method used for each chunk, and other parameters. This allows to have more flexibility in how the data is compressed and stored. The binary layout is as follows: +———————+ | HEADER | +———————+ | CHUNK #0 | +———————+ | CHUNK #1 | +———————+ | … | +———————+ | CHUNK #n | +———————+ The individual chunks contain the compressed data, and each chunk can be decompressed independently. The header contains the necessary information to interpret the chunks correctly. The header itself is organized in “header chunks”, where each header chunk has a chunk identifier, a size, and a payload.

Public Types

enum class Codec : std::uint8_t#

Values that represent codecs.

Values:

enumerator Invalid#

Invalid codec, used to indicate an error condition.

enumerator ZStd#

Zstd compression.

enumerator Lz4#

Lz4 compression.

enum class HeaderChunkId : std::uint16_t#

Values that represent header chunk Identifiers.

Values:

enumerator EndOfHeader#

This header chunk indicates the end of the header. This must be the last chunk in the header, and it has no payload.

enumerator ChunkSizes#

This header chunk contains the sizes of the compressed chunks.

enumerator CompressionMethod#

This header chunk contains the compression method (codec) used for the chunks.

enumerator DecompressedSizes#

This header chunk contains the sizes of the uncompressed data for the chunks.

enumerator Preprocessing#

This header chunk contains information about preprocessing applied to the data before compression (like hi-lo byte packing).

Public Static Functions

static bool WalkCompressionHeader(const void *data, size_t sizeData, const std::function<bool(const CompressionHeaderChunk&)> &callback, size_t *bytes_consumed)#

Walk the chunked-compression header and invoke the specified callback for each header chunk found. The callback receives a ‘CompressionHeaderChunk’ describing the current chunk; returning false from the callback stops the walk early. If the data does not represent a valid chunked-compression header, an exception is thrown.

Parameters:

  • data – Pointer to the data containing the chunked-compression header.

  • sizeData – The size of the data in bytes.

  • callback – The callback invoked for each header chunk. Return true to continue walking, false to stop.

  • bytes_consumed – [out] If non-null, receives the number of bytes consumed up to the end-of-header marker or up to the end of the chunk that caused the callback to stop early.

Returns:

True if the end-of-header marker was reached; false if the callback stopped the walk early. Invalid or truncated headers are reported by throwing an exception.

static size_t GetCompressionHeaderSize(const void *data, size_t sizeData)#

Parse the chunked-compression header, and return the size of the header (in bytes). If the given data does not contain a valid chunked-compression header, then an exception is thrown. Note that only the structure of the header is parsed here, not the semantic of the header content.

Parameters:

  • data – Pointer to the data to be parsed.

  • sizeData – The size of the data.

Returns:

The size of the header in units of bytes.

static std::tuple<size_t, HeaderInfo> ParseCompressionHeader(const void *data, size_t sizeData)#

Parse the chunked-compression header and return both its byte size and the extracted header information. If the given data does not contain a valid chunked-compression header, an exception is thrown.

Parameters:

  • data – Pointer to the data to be parsed.

  • sizeData – The size of the data in bytes.

Returns:

A tuple where the first element is the size of the header in bytes, and the second element is a ‘HeaderInfo’ structure containing the parsed header information.

static size_t CreateCompressionHeader(void *destination, size_t sizeDestination, const HeaderInfoForCreation &headerInfo)#

Creates a chunked-compression-header for the given header information. The created header is written to the memory pointed to by ‘destination’, and the size of the created header is returned. The required size of the destination buffer is in general not known (and not knowable) beforehand, but it is guaranteed that the required size is less or equal to the value returned by ‘DetermineMaxSizeForCompressionHeader’ for the same header information. If the specified destination buffer size is insufficient, then this method throws an exception (of type runtime_error). All other error conditions (like e.g. invalid arguments) also result in an exception being thrown.

Parameters:

  • destination – [inout] The pointer to the memory where the created header is written to. This argument must not be null.

  • sizeDestination – The size of the memory block pointed to by ‘destination’ in bytes.

  • headerInfo – Information describing the header.

Returns:

The size of the compression header written to the memory pointed to by ‘destination’ in bytes.

static size_t DetermineMaxSizeForCompressionHeader(const HeaderInfoForMaxSizeDetermination &headerInfo)#

This function determines the maximum size of the compression header for the given header information. The actual size of the compression header for a given set of header information may be smaller than this maximum size, but it will never be larger.

Parameters:

headerInfo – Information describing the header.

Returns:

The max number of bytes required for constructing a CompressionHeader with ‘CreateCompressionHeader’.

static size_t DetermineMaxSizeForCompressionHeader(const HeaderInfoForCreation &headerInfo)#

Convenience overload of the above function, which takes ‘HeaderInfoForCreation’ as argument. The required information for determining the max size is determined from the given ‘HeaderInfoForCreation’ argument, and in turn the function ‘DetermineMaxSizeForCompressionHeader(const HeaderInfoForMaxSizeDetermination& headerInfo)’ is called to determine the max size.

Parameters:

headerInfo – Information describing the header.

Returns:

The max number of bytes required for constructing a CompressionHeader with ‘CreateCompressionHeader’.

static size_t CalculateMaxCompressedSizeChunked(std::uint32_t sourceWidth, std::uint32_t sourceHeight, libCZI::PixelType sourcePixeltype, std::uint32_t maxChunkSize, Codec codec, bool hiLoBytePacking)#

Calculates the maximum buffer size required to hold the complete compressed blob (header + all chunks) produced by ChunkedCompress::Compress for the given source bitmap and parameters. If ChunkedCompress::Compress is called with an output buffer of at least this size, it will never fail due to insufficient buffer space.

Parameters:

  • sourceWidth – Width of the source bitmap in pixels.

  • sourceHeight – Height of the source bitmap in pixels.

  • sourcePixeltype – The pixel type of the source bitmap.

  • maxChunkSize – Maximum uncompressed size of a single chunk in bytes.

  • codec – The compression codec to be applied to each chunk.

  • hiLoBytePacking – Whether hi-lo byte packing preprocessing is applied before compression.

Returns:

An upper bound on the total size (header + compressed chunks) in bytes.

struct CompressionHeaderChunk#

This struct represents a single header chunk as encountered when walking the chunked-compression header. Each header chunk consists of an identifier, the size of its payload, and a pointer to the payload data.

Public Members

std::uint16_t chunkId#

The identifier of the header chunk. The value is to be interpreted as a HeaderChunkId enum value.

std::uint32_t chunkSize#

The size of the header chunk payload in bytes.

const void *chunkPayload#

Pointer to the payload data of the header chunk. The memory is valid only for the duration of the callback invocation.

size_t chunkPayloadSize#

The size of the data pointed to by ‘chunkPayload’ in bytes. This is equal to ‘chunkSize’.

struct HeaderInfo#

This struct contains the parsed information extracted from a chunked-compression header.

Public Members

Codec codec#

The codec used for compressing the chunks.

bool hiLoBytePackingApplied#

Whether the “hi-lo byte packing” preprocessing was applied to the data before compression.

std::vector<ChunkInfo> chunks#

The per-chunk information entries, one for each compressed chunk in order.

struct ChunkInfo#

This struct holds the compressed and uncompressed sizes of a single chunk.

Public Members

std::uint32_t compressedSize#

The size of the compressed chunk data in bytes.

std::uint32_t uncompressedSize#

The size of the uncompressed chunk data in bytes.

struct HeaderInfoForCreation#

This struct defines the parameters to be stored in the chunked-compression headers for use when creating a compression header. Note that only the case of “all uncompressed chunk sizes but the last

one” is currently supported.

Public Members

Codec codec#

The codec to be used for compressing the chunks.

std::uint8_t hiLoBytePackingApplied#

This flag indicates whether the “hi-lo byte packing” preprocessing is applied to the data before compression. 0 means “no hi-lo byte packing”, 1 means “hi-lo byte packing applied”, everything else means: unspecified. If the value is unspecified, then the corresponding header chunk is not included in the header.

std::vector<std::uint32_t> chunkSizes#

The sizes of the compressed chunks in bytes, one entry per chunk.

std::tuple<std::uint32_t, std::uint32_t> uncompressedSizes#

This describes the uncompressed sizes The first value is the uncompressed size of all data chunks but the last, and the second value is the uncompressed size of the last chunk. (Note that we here are assuming that all the sizes but the last are the same!). If the last chunk has the same uncompressed size as the other chunks or if there is only one chunk, then the second value is zero (and the first value gives the uncompressed size of all chunks).

struct HeaderInfoForMaxSizeDetermination#

Information used to determine the maximum size of a compression header. All fields must be set to valid values. The number_of_chunks field must be greater than 0.

Public Members

Codec codec#

The codec to be used for compressing the chunks.

std::uint8_t hiLoBytePackingApplied#

This flag indicates whether the “hi-lo byte packing” preprocessing is applied. 0 means “no hi-lo byte packing”, 1 means “applied”, everything else means: unspecified.

std::uint32_t number_of_chunks#

The number of chunks. Must be greater than 0.