Skip to content

Scalar integer compression algorithm based on the MILC #4

@burmanm

Description

@burmanm

Taking some inspiration from MILC: Inverted List Compression in Memory, we should implement their algorithm but add some changes for Java and some improvements for compression ratio:

  • No SIMD tricks (for now), but we should still ensure other features (such as direct access). With scalar processing and Java memory handling we might need to also use different memory layout for the "skip pointers"

  • Since dynamic programming is used to determine optimal block size, we should also choose the optimal offset which would use a minimal amount of bits in the block. Slows down compression but improves compression ratio. Original paper uses the first value always.

  • The original paper does not mention what happens when offset is negative, thus we will probably need to use ZigZag to encode the values to avoid issues/branches in decompression

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions