Q: support for merge operators

[mindreframer]

Hi @Elzor ,

i have been playing with ex_rock and it's really great! I have one usecase, that is currently not covered by rocker/ex_rock: support for merge operators.

I want to maintain a list of items as value and right now i would be required to read the whole list, append an item and serialize and write it back... which breaks down pretty quickly with larger lists.

I saw this example:

• https://github.com/rust-rocksdb/rust-rocksdb/blob/master/src/merge_operator.rs

Here is a talk with some deeper explanation: https://youtu.be/LPAm2dP69pE?t=2235 (this exact moment).
Now, I'm not sure it is possible to expose this feature to Elixir... Defining and registering a custom merge operator.

Do you have any idea if it would be possible? I'm building my own indexes in Rocksdb and this feature would really allow proper index management.

Best,
Roman

[mindreframer]

I just found, that the the rocksdb hex package does provide support for it:

• https://hexdocs.pm/rocksdb/erlang_merge_operator.html. But it's not rust and does not have pre-compiled nifs.

Here is also an idea from Claude code:

⏺ Based on my analysis of the codebase, yes, it is possible to expose custom merge
  operators to Elixir, though it will require significant architectural changes.
  Here's my detailed analysis:

  Current Architecture

  The wrapper uses Rustler NIFs to bridge Erlang/Elixir with the rust-rocksdb crate.
   Currently:
  - The rust-rocksdb crate fully supports merge operators (see
  /Users/roman/Desktop/work/sandbox/0.KV/rust-rocksdb/src/merge_operator.rs:68-75)
  - The rocker wrapper does NOT expose merge operations at all - no merge function
  in the NIF interface
  - Options handling exists but lacks merge operator configuration

  Implementation Feasibility

  Challenges:

  1. Callback Complexity: Merge operators in RocksDB are callbacks that execute
  during compaction and reads. The challenge is that these callbacks need to be
  defined in Rust but invoked with logic from Elixir.
  2. Performance: Calling back into Elixir from Rust during merge operations could
  be expensive, especially during compaction when many merges occur.
  3. Memory Safety: Passing data between Erlang/Elixir and Rust merge callbacks
  requires careful memory management.

  Recommended Approach:

  For your use case (appending to lists), I recommend a hybrid approach:

  1. Predefined Merge Operators: Implement common merge patterns in Rust (like list
  append, counter increment, etc.) and expose them as configuration options:

  // In options.rs
  pub enum MergeOperatorType {
      ListAppend,      // Your use case
      Counter,         // Increment/decrement
      StringConcat,    // String concatenation
      Custom(String),  // Future extension point
  }

  2. Required Changes:

  2. a. Add merge operator to options
  (/Users/roman/Desktop/work/sandbox/0.KV/rocker/crates/rocker/src/options.rs):
  pub struct RockerOptions {
      // ... existing fields ...
      pub set_merge_operator: Option<String>, // "list_append", "counter", etc.
  }

  2. b. Implement predefined merge functions (new file):
  // crates/rocker/src/merge_ops.rs
  pub fn list_append_merge(
      _key: &[u8],
      existing: Option<&[u8]>,
      operands: &MergeOperands
  ) -> Option<Vec<u8>> {
      // Decode existing as Erlang term list
      // Append new items from operands
      // Encode back to Erlang term format
  }

  2. c. Add merge NIF functions
  (/Users/roman/Desktop/work/sandbox/0.KV/rocker/crates/rocker/src/nif.rs):
  #[rustler::nif]
  fn merge(env: Env, resource: ResourceArc<DbResource>,
          key: Binary, value: Binary) -> NifResult<Term> {
      // Call db.merge()
  }

  2. d. Wire up in lib.rs
  (/Users/roman/Desktop/work/sandbox/0.KV/rocker/crates/rocker/src/lib.rs:12-59):
  rustler::init!(
      "rocker",
      [
          // ... existing ...
          nif::merge,
          nif::merge_cf,
      ]
  );
  3. For List Append Specifically:
    - Store lists in Erlang External Term Format (ETF)
    - The merge operator would decode ETF, append items, re-encode
    - This keeps the data format compatible with Erlang/Elixir

  Implementation Complexity: Medium-High

  The implementation is definitely feasible but requires:
  - Understanding of Erlang term encoding/decoding
  - Careful handling of the Rustler NIF interface
  - Thorough testing of merge operations
  - Documentation of the predefined merge patterns

  Your use case (list append) is actually one of the most common merge operator
  patterns and would benefit many users of the library.