compress/compress_to_vec fails with 'memory allocation of 131072 bytes failed' in no_std microcontroller environment

[lucasw]

The messages range from 5 to 500 bytes but the same failed allocation results:

panic_handler: panicked at library/alloc/src/alloc.rs:438:13:
memory allocation of 131072 bytes failed

I've tried making my heap larger than 131072 but that still fails.

use alloc::vec:::Vec;

...
        use ruzstd::encoding::{compress_to_vec, CompressionLevel};
        let msg_bytes = self.encode()?; // get alloc vec Vec<u8>

        let compressed = compress_to_vec(&*msg_bytes, CompressionLevel::Fastest);

        // let mut compressed = Vec::new();
        // compress(&*msg_bytes, &mut compressed, CompressionLevel::Fastest);  

CompressionLevel::Uncompressed also fails the same.

Elsewhere decompresing works fine:

let decoder = ruzstd::decoding::StreamingDecoder::new(buf).unwrap();

// buf is &[u8]
let mut decode_buf = Vec::new();
let rv = decoder.read_to_end(&mut decode_buf);
...

The same code tests fine in the non-embedded standard library environment. I'm new to embedded rust so perhaps I have something configured wrong with my heap.

I notice 131072 comes from here: https://github.com/KillingSpark/zstd-rs/blob/master/src/encoding/blocks/compressed.rs#L268

I can do let output: &mut Vec<u8> = &mut Vec::with_capacity(1024 * 130); as in compress() once in my code and it works fine with my 200k heap.

I can

I'm using 0.8.1

[lucasw]

I just needed to make my heap bigger, apparently there are two allocations of around 2^17 in order to do any compression at all?

[KillingSpark]

Hi!

Yeah the compression code is pretty new and not yet very well optimized for embedded usecases. We do pre-allocate a few buffers per compressor.

That should become configurable at some point.

[paddor]

zrip supports no_std + alloc. Encoder hash table scales with level and input size: at L-7 it's 32 KiB for large inputs, but for your 5-500 byte messages it clamps down to ~2 KiB. CompressContext reuses that across calls. Decoder (DecompressContext) starts at zero and grows on demand.