otap: merge into one crate, real OtapPdata direct from SketchEnvelope, no staging required

[zzylol]

Why

Started from otap_bridge.rs (the real OtapPdata <-> OtapMetricRecords binding) being written entirely from reading upstream source, never compiled. Along the way, review surfaced three deeper architectural questions worth answering for real rather than by assumption:

1. Does sketch traffic riding a generic OTLP metric need its own dictionary/schema-reuse economics, or does OTAP's real Arrow encoding already provide that?
2. Why does otap-patch/ need to be a separate overlay staged into a checked-out OTAP Dataflow workspace at all — why not just depend on the real crates directly?
3. Given (2), why keep OtapMetricRecords/flat-RecordBatch as intermediate hops between SketchEnvelope and OtapPdata instead of encoding straight to the real type?

This PR answers all three with real, verified code, not reasoning.

What

One crate, not two. otap-patch/ is gone. asap-precompute-rs now depends directly on the real otel-arrow-dfe-* crates via a plain git dependency (same mechanism already used for asap_sketchlib), behind a new otap-engine Cargo feature. No more staging files into a separately-checked-out OTAP Dataflow workspace to compile — cargo build --features otap-engine just works, fetching from git like any other dependency.

One transport. AsapSketchesProcessor sends only via effect_handler.send_message_with_source_node. An earlier revision on this branch added a direct-TCP "wire lane" as a second transport (otap_wire.rs + a standalone AsapSketchesReceiver receiver node) — that's been removed.

One encoding, no intermediate hops. The wire lane existed to give sketch traffic dictionary/schema-reuse economics a generic OTLP metric didn't have (the legacy SeriesDictionary SCHEMA/DICTIONARY/RECORD tiering). Turns out unnecessary: OTAP's real Arrow encoder (encode_metrics_otap_batch) already dictionary-encodes the metric name and every string-valued attribute key/value by construction — confirmed against the real workspace and guarded by a permanent test:

=== payload_type UnivariateMetrics ===
  name : Dictionary(UInt8, Utf8)
=== payload_type NumberDpAttrs ===
  parent_id : Dictionary(UInt8, UInt32)
  key       : Dictionary(UInt8, Utf8)
  str       : Dictionary(UInt16, Utf8)

That's the same "send it once, reference it after that" shape SeriesDictionary was reinventing at the application layer, done at the columnar/Arrow-IPC level instead — no dictionary state for this adapter to track, no second wire protocol, none of the "must be one ordered, single-consumer stream" correctness constraint a hand-rolled series_id scheme carries.

Once that was settled, the old encode_batch (→ flat RecordBatch) → lift (→ OtapMetricRecords) → otap_bridge (→ OtapPdata) chain was also revisited: those three representations for one job only existed to serve a second adapter (Telegraf/Vector) that was never actually built in this repo. The new otap::codec module implements OTAP's own MetricsView trait family directly over &[SketchEnvelope] — one hop, SketchEnvelope straight to real OtapPdata, and back.

encode_batch/decode_batch/records::{flatten,lift} still exist, still tested — they now exclusively back the legacy SeriesDictionary/otap::wire transport and its standalone example binaries, not the otap-engine path.

How

A real bug found and fixed while rewriting the encoder: encode_batch unconditionally sets _asap_envelope for every row, including estimate-mode envelopes (empty payload, the gauge value rides in value instead). arrow_array::BinaryArray treats Some(&[]) as a present, empty value, not null — confirmed empirically (BinaryArray::from_opt_vec(vec![Some(&[])]).is_null(0) == false). So an estimate-mode row would silently misroute through the envelope decode path with an empty payload instead of the scalar path. otap::codec's direct encoder only attaches _asap_envelope (and its sibling _asap_* attributes) when the payload is actually non-empty, with a defensive .filter(|b| !b.is_empty()) on decode too. New regression test: estimate_mode_envelope_round_trips_as_a_scalar_not_an_empty_envelope. The legacy encode_batch/decode_batch pair keeps the original behavior untouched — fixing it there was out of scope here.

Real trait-signature drift the compiler caught during the rewrite (uninhabited-placeholder MetricsView impls, copied from the old file and re-typed by hand): ResourceView::AttributesIter (not AttributeIter), ResourceMetricsView::schema_url() -> Option<Str> (not bare Str), HistogramDataPointView/BucketsView needing explicit BucketCountIter/ExplicitBoundsIter associated types, ExponentialHistogramDataPointView::zero_threshold(). All caught by cargo build, none guessable from memory.

Why a git dependency works without workspace staging: open-telemetry/otel-arrow's own crates (otel-arrow-dfe-engine etc.) declare their own sibling dependencies as { workspace = true }. A plain { git = "...", package = "otel-arrow-dfe-engine" } dependency still resolves correctly because Cargo clones the whole repo and treats that clone's workspace root as the resolution context for the named package's own deps — this crate doesn't need to be a member of that workspace itself. Verified directly with a scratch crate before committing to this design.

Output

$ cargo build --features otap-engine && cargo test --features otap-engine
...
test result: ok. 113 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out   (unit tests, incl. otap::codec + otap::processor)
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out    (tests/api_surface.rs)
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out     (tests/otap_codec.rs)
test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out     (tests/otap_lifecycle.rs)
test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out    (tests/runtime.rs)

$ cargo clippy --features otap-engine --all-targets -- -D warnings   # clean
$ cargo fmt --check                                                   # clean

169 tests total at the otap-engine level. Also re-verified otap (100+11+4+9+32 = clean, minus the 13 otap-engine-only tests) and default (no-feature) builds are unaffected — all via this repo's own cargo build/cargo test, nothing staged externally.

Not done here

• No test exercises AsapSketchesProcessor inside a real running pipeline end to end (no Message::PData sent through process() against a live df_engine).
• The receiver role — ingesting another asap_sketches node's legacy SketchStreamBatch output — isn't addressed by this adapter; that's a different, ASAP-native hop with its own standalone example binaries.
• Cross-host byte-parity (Phase E) is unstarted.

🤖 Generated with Claude Code