feat: merge multiple recording files into one session (multi-mcap, mixed formats) (#21)

* chore: bump dependencies to latest within existing semver ranges

Refresh vite/vitest/@playwright/test and other devDependencies ahead of
the multi-source merge feature work.

* feat(sources): add CombinedSourceProxy to merge multiple recording sources

Introduce an ISourceHandle interface that both WorkerSerializedSource
(single file) and the new CombinedSourceProxy (N files/formats fanned
out to independent Workers) implement, so IterablePlayer can drive a
merged multi-source session without any change to its own logic.

- ISourceHandle: structural surface IterablePlayer depends on.
- mergeInitialization: pure merge of N per-source Initialization
  results (topics concatenated in file order, same-name topics folded
  together with accumulated sourceLabels, start/end unioned, topicStats
  merged).
- CombinedMessageCursor: parallel k-way merge of child message cursors
  by receiveTime, tagging each message with its origin for later
  routing.
- CombinedSourceProxy: fans out initialize/getMessageCursor/
  getBackfillMessages/getAdjacentMessage/preparePlaybackBuffer/
  getLoadProgress/getDataQualityReport to only the relevant member
  sources in parallel, and routes resolveMessageBatch/
  resolveMessageForHighFrequencyLane back to the originating member
  (required for per-Worker SharedArrayBuffer payload rings).

TopicInfo gains an optional sourceLabels field (unset for single-file
sessions, so existing UI is unaffected).

* feat(viewer): merge multiple recording files into one session by default

Add session grouping on top of the existing dataset list:
- DatasetItem gains an optional groupId (datasetGroupKey/groupDatasets
  helpers); items without one behave exactly as a standalone group of
  one, so single-file loading is byte-for-byte unchanged.
- RosViewerImpl builds one IterablePlayer per active group instead of
  per dataset: a group of one still wires WorkerSerializedSource
  directly; 2+ members are fanned out to independent Workers (in
  parallel, any mix of mcap/bag/db3/hdf5/bvh) and combined through
  CombinedSourceProxy.
- Default interactive behavior: dropping/opening more recording files
  while a session is active merges them into that session (its topic
  list and time range grow); dropping files with nothing loaded starts
  one fresh merged session. Tar extraction and history replay keep
  today's "open an independent, switchable session" behavior.
- Embed API (`file`/`files`/`url`/`urls`/`fileManifest`) is unchanged
  by default; a new opt-in `mergeSources` prop merges them into one
  group for hosts that want it.

AppShell's active-dataset lookup now resolves by group key so
extension context and Navbar/title labeling keep working for merged
sessions.

* feat(sidebar): surface merged-session provenance in topic "more" menu and Data tab

- TopicRow's overflow menu shows a "Source: a.mcap, b.mcap" line when a
  topic's sourceLabels are populated (multi-source sessions only; the
  menu is unchanged for single-file sessions).
- Sidebar's Data tab now renders one row per merged session (grouped
  by datasetGroupKey) instead of one row per file: a merged session
  shows "first.name +N", the full member list, and a "{count} files
  merged" badge, while a standalone file renders exactly as before.

Adds sidebar.topicRow.sourceFiles and sidebar.datasetGroup.fileCount
i18n strings (en/zh/ja).

* test: add multi-mcap and mcap+bag fixture generators for merge testing

- gen-test-mcap-multi-base.mjs / gen-test-mcap-multi-incremental.mjs:
  a "base recording" (camera + joint_states, 0-5s) and a separately-
  authored "incremental analysis" file (one new topic, 3-7s) that
  mirrors the described external hand-pose-overlay workflow. Loading
  both exercises topic concatenation and the union time range.
- gen-test-mcap-filtered.mjs: derives a third fixture from the base
  file using the real `mcap filter` CLI (not @mcap/core), so at least
  one multi-source test loads a CLI-derived file alongside a
  hand-authored one.
- gen-test-bag.py: writes a minimal ROS1 .bag (Python `rosbags`
  package, since there is no maintained JS/TS ROS1 bag writer) for
  mcap+bag mixed-format tests; verified against the app's own
  @foxglove/rosbag reader.

Both the CLI-derived and bag fixtures self-skip (exit 0, no output
file) when their external tool isn't installed, so `npm run
gen:e2e:fixtures` keeps working for contributors without them; CI
installs both (follow-up commit).

* ci: install mcap CLI for fixture generation; commit bag source fixture

- .github/workflows/ci.yml (e2e job): download the pinned mcap-cli
  Linux binary from GitHub releases so gen-test-mcap-filtered.mjs's
  `mcap filter` step actually runs in CI instead of self-skipping.
- gen-test-bag.py now follows gen-test-hdf5.py's pattern: the ROS1 bag
  is generated once into the committed
  test-fixtures/media/minimal-multi.bag, and gen:e2e:fixtures just
  copies it into public/examples/test_multi.bag. This means CI and
  most contributors never need the `rosbags` Python package at all;
  it's only required to regenerate the source fixture.
- docs/DEVELOPMENT.md: document the new fixtures, the two optional
  external tools and their auto-skip behavior, and a multi-source
  merge acceptance checklist.

* test(e2e): cover multi-mcap merge, mcap+bag mixing, and incremental add

Extend tests/multi-sources.spec.ts with:
- Two mcap files opened together merge into one session: topics from
  both files appear, and the playback range becomes the union of each
  file's time range (0-5s + 3-7s -> 0-7s), not either file's own range.
- Dropping a second file onto an already-loaded session extends it
  in place (topics/range grow) instead of creating a separate,
  switchable dataset; the Data tab shows exactly one merged-session
  row.
- The topic "more" menu surfaces which file a topic came from once
  sessions are merged.
- mcap + bag mixed-format merge, and a file derived via the real
  `mcap filter` CLI merging with a hand-authored one; both self-skip
  when their optional fixture is unavailable.

Adds a `playback-time-line` test id to PlaybackBar so tests can assert
the merged/union time range shown next to the scrubber.

Author: joaner

.github/workflows/ci.yml

@@ -22,6 +22,8 @@ jobs:
 
   e2e:
     runs-on: ubuntu-latest
+    env:
+      MCAP_CLI_VERSION: 'v0.2.0'
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-node@v4
@@ -30,6 +32,17 @@ jobs:
           cache: 'npm'
       - run: npm ci
       - run: npx playwright install chromium --with-deps
+      - name: Install mcap CLI
+        # Used by scripts/gen-test-mcap-filtered.mjs to derive a fixture via
+        # the real `mcap filter` command (not @mcap/core). That script
+        # self-skips if this step is ever removed, so it's not load-bearing
+        # for the rest of CI, but keeping it here exercises the multi-source
+        # merge feature's "mcap filter"-derived fixture on every run.
+        run: |
+          curl -sSL -o /usr/local/bin/mcap \
+            "https://github.com/foxglove/mcap/releases/download/releases/mcap-cli/${MCAP_CLI_VERSION}/mcap-linux-amd64"
+          chmod +x /usr/local/bin/mcap
+          mcap --version
       - name: Generate E2E fixtures
         run: npm run gen:e2e:fixtures
       - run: npm run test:e2e

.gitignore

@@ -38,4 +38,8 @@ yarn.lock
 /public/examples/
 /test-results/
 /playwright-report/
-.tmp-urdf-debug-*/
+.tmp-urdf-debug-*/
+
+# Python (fixture generator scripts, e.g. gen-test-hdf5.py / gen-test-bag.py)
+__pycache__/
+*.pyc

docs/DEVELOPMENT.md

@@ -29,9 +29,20 @@ This runs automatically as `pretest:e2e` before `npm run test:e2e`.
 | `test_h264.mcap` | H.264 CompressedImage decode |
 | `test_minimal.hdf5` | ALOHA-schema HDF5 (~7 KB) |
 | `test_minimal.bvh` | Minimal BVH skeleton |
+| `test_multi_base.mcap` | Multi-source merge: "base recording" (camera + `/joint_states`, 0-5s) |
+| `test_multi_incremental.mcap` | Multi-source merge: separately-authored file adding one topic, 3-7s |
+| `test_multi_filtered.mcap` | Multi-source merge: derived from `test_multi_base.mcap` via the real `mcap filter` CLI — **optional**, see below |
+| `test_multi.bag` | Multi-source merge: ROS1 bag for mcap+bag mixed-format tests (copied from the committed `test-fixtures/media/minimal-multi.bag`) |
 
 Vitest layout round-trip tests import JSON directly from `test-fixtures/layouts/`.
 
+### Optional external tools
+
+Two fixtures depend on a tool that isn't an npm package. Their generator scripts detect absence and skip (exit 0, no output file) rather than failing the whole `gen:e2e:fixtures` run; Playwright specs that need one of these files check for it and `test.skip()` themselves when it's missing.
+
+- **`mcap` CLI** (Rust rewrite, `mcap --version` ≥ 0.1) — needed only to (re)generate `test_multi_filtered.mcap` via `mcap filter`. Install with `brew install mcap` or download a binary from <https://github.com/foxglove/mcap/releases?q=mcap-cli>. CI installs it explicitly (see `.github/workflows/ci.yml`), so this fixture and its tests always run there.
+- **`rosbags`** Python package — needed only to *regenerate* the committed `test-fixtures/media/minimal-multi.bag` source (e.g. after changing its schema in `scripts/gen-test-bag.py`); `npm run gen:e2e:fixtures` just copies that committed file into `public/examples/test_multi.bag`, so CI and most contributors never need to install it. If you do: `pip install rosbags`.
+
 For sample deep links (`?url=sample://…`), set `VITE_SAMPLE_DATASETS_MANIFEST_URL` in `.env` to a reachable JSON manifest (see `src/services/sampleDatasets.ts`).
 
 For remote lists in the browser during dev, prefer **same-origin** URLs, e.g.  
@@ -81,3 +92,15 @@ npm run test:e2e
 4. Open `/`, upload or drag a local `.mcap`; confirm load succeeds.
 
 Full E2E coverage requires `npm run gen:e2e:fixtures` so `public/examples/` is populated; no files outside the repo are needed.
+
+## Acceptance (multi-source merge)
+
+Loading multiple recording files together (any mix of `.mcap`/`.bag`/`.db3`/`.hdf5`/`.bvh`) merges them into one session: topics from every file are concatenated in file order, and the playback range becomes the union of each file's time range. See `tests/multi-sources.spec.ts` for the automated coverage (multi-mcap merge, mcap+bag mixed formats, merging into an already-active session, Data tab grouping).
+
+**Manual checks**
+
+1. `npm run gen:e2e:fixtures`, then `npm run dev`.
+2. Drag `public/examples/test_multi_base.mcap` onto the welcome screen; confirm it loads with 2 topics (`/camera/front/image_raw/compressed`, `/joint_states`) and a 0-5s range.
+3. Drag `public/examples/test_multi_incremental.mcap` in on top of the already-loaded session; confirm the topic list grows to 3 topics and the playback range extends to 0-7s (the union), **without** switching away to a separate "Data" entry.
+4. Open the new topic's "more" menu (`⋯` in the sidebar Topics tab); confirm it shows a "Source: …" line naming both files once a topic name happens to be shared, or the single owning file otherwise.
+5. Optional, with the real `mcap` CLI installed and a large real-world recording on hand: split it into two files by topic/time range with `mcap filter in.mcap -o a.mcap -y /some/topic` (and a complementary `-n` regex for the rest), then load both together and confirm the merged topic count/time range match `mcap info` on the original file.