Skip to content

Replace expanded sequence arrays with a compact deterministic representation #1321

Description

@JackWilb

Summary

Large studies currently persist and transfer 1,000 fully expanded participant sequences. In library-calvi, the sequence artifact is approximately 3.35 MiB. Even after duplicate downloads are removed by #1176/PR #1311 and publication is moved out of the interaction barrier by #1319, this representation remains expensive to generate, upload, download, parse, and retain.

Replace the expanded artifact with a compact deterministic descriptor while preserving every existing assignment and randomization behavior.

Design direction

Persist a versioned descriptor containing the minimum information needed to reproduce an assigned sequence, such as:

  • Config hash.
  • Deterministic seed.
  • Sequence-generation algorithm version.
  • Compact permutation/order data only where it cannot be derived from the seed.

A descriptor must produce identical results across supported browsers and storage engines. Algorithm changes must create a new version rather than silently changing sequences for an existing descriptor.

This work should build on the config-hash build coordination in #1319. Existing full sequence arrays remain a supported legacy format.

Requirements

  • Define a versioned persisted schema for compact sequence descriptors.
  • Make sequence generation deterministic for a given config hash, seed, and algorithm version.
  • Derive only the assigned row when possible rather than expanding every participant sequence during startup.
  • Preserve Latin-square ordering, random ordering, URL conditions, stages, nested/dynamic blocks, rejected-slot reuse, and participant-resume behavior.
  • Continue reading existing expanded sequence arrays without manual migration.
  • Preserve access to the exact config/sequence representation used by an existing participant.
  • Support Firebase and Supabase; preserve LocalStorage behavior.
  • Detect unsupported or corrupt descriptor versions and fail with a recoverable, actionable error.
  • Do not add a dependency unless separately approved.

Acceptance criteria

  • A newly published study stores a compact versioned descriptor instead of 1,000 expanded sequences.
  • The descriptor is materially smaller than the current expanded artifact for library-calvi, with before/after byte sizes recorded.
  • The same descriptor produces byte-equivalent assigned sequences across repeated runs and supported browsers.
  • Concurrent clients resolving the same descriptor cannot produce different sequences.
  • Existing expanded arrays and participant sessions continue to load without migration.
  • Tests compare legacy and compact output for Latin-square, random, conditional, staged, and dynamic study configurations.
  • Tests lock algorithm-version behavior so future generator changes cannot alter existing participant sequences.
  • Browser profiling records download, parse/generation, and time-to-interactive changes for library-calvi.
  • yarn unittest --run, yarn typecheck, yarn lint, and yarn build pass.

Out of scope

  • Changing study balancing or randomization semantics.
  • Removing compatibility with existing expanded arrays.
  • General application bundle optimization.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

SpecifiedIssue has enough detail to implementbugSomething isn't working

Type

No type

Projects

  • Status
    Todo

Relationships

None yet

Development

No branches or pull requests

Issue actions