NO to_arrow_batch_reader support in python binding

[charlesdong1991]

Search before asking

• I searched in the issues and found nothing similar.

Description

Ad per title, to have to_arrow_batch_reader for streaming use case.

Willingness to contribute

• I'm willing to submit a PR!

[charlesdong1991]

this can be good first issue

maybe not a goof fit for "good first issue" given feedback below, but it depends

[fresh-borzoni]

@charlesdong1991 Shall we start first from Rust SDK?
Otherwise all the clients would reimplement the same

I see it like this:

Rust core:

• Move query_latest_offsets() from Python into Rust core so all bindings share it.
• RecordBatchLogReader: bounded reader that encapsulates the poll loop, offset filtering, and stopping logic. Exposes async fn next_batch() for async Rust users (returns ScanBatch with Arrow RecordBatch + bucket/offset metadata). Supports
  "read until latest offsets" (convenience) and custom stopping offsets (flexible).
• .to_record_batch_reader(runtime): sync adapter implementing arrow::record_batch::RecordBatchReader for Arrow ecosystem interop and FFI consumers.
• futures::Stream trait impl can follow later for full async Rust ecosystem integration.

Note: this is different from Java's BatchScanner which does server-side limit scans and KV snapshot reads. RecordBatchLogReader is client-side bounded log consumption, poll until latest offsets, then stop.

Python: thin wrapper, constructs RecordBatchLogReader, calls .to_record_batch_reader(runtime), returns pa.RecordBatchReader.from_batches(schema, generator). Existing to_arrow() can be refactored to use this internally.

C++: same thin wrapper when needed.

This gives bounded snapshot reads (poll until latest offsets, then stop) with lazy batch-by-batch iteration instead of loading everything into memory. Unbounded streaming is already covered by the existing async poll() loop.

[charlesdong1991]

Hi @fresh-borzoni i looked at your pointers and implementations in current core, and i totally agree with the proposal, since I overlooked the benefit of moving bounded-read to rust core while i was playing with python interface. 👍

In this way, then we can have both bindings become thin wrapper and avoid different implementations across languages!

Thanks for suggest!