A/B Experiment Runner

A modular browser experiment runner that compiles to one self-contained, minified JavaScript line for manual copy-paste deployment.

The source is designed for maintenance. The artifact is designed for first-visit performance and operational simplicity. There is no runtime package install, network request, or CDN dependency on the storefront.

Goals

• Keep assignment and treatment behavior consistent across experiments.
• Make the deployed artifact a single copy-paste line.
• Keep the critical assignment path small and synchronous where possible.
• Support CSS-first treatments for asynchronously rendered Shopify elements.
• Retain richer DOM and custom behavior without putting it in the bootstrap.
• Prevent namespace collisions when several experiments share a page.
• Make every source change reproducible through one documented build command.

Quick start

Requirements: Node.js 20 or newer.

npm install
npm run check

1. Edit src/config/experiment.js.
2. Run npm run check.
3. Copy the one line from dist/experiment.min.js.
4. Paste it inside a <script> element in the intended theme location.
5. Preview every variant using ?<experiment-key>=<variant>.

The committed configuration is a working reference for 028-product-inventory-price.

Architecture

The source has two execution layers:

1. Critical bootstrap: validates configuration, checks a URL override, subscribes to PostHog, resolves one variant, and then hands off execution.
2. Experiment runtime: applies ordered CSS, element, and custom actions only after assignment resolves.

The build uses esbuild to bundle and tree-shake the module graph into a minified IIFE. No source modules or variables are exposed globally.

src/config/experiment.js
          |
          v
src/critical/bootstrap.js ---> src/providers/posthog.js
          |
          v
src/runtime/apply-variant.js
          |
          +-- css action
          +-- element action --> DOM utilities
          +-- custom action
          |
          v
dist/experiment.min.js

See Architecture for execution details and design boundaries.

Current features

• Forced variant previews through the experiment-key URL parameter
• PostHog feature-flag assignment through onFeatureFlags
• Safe fallback for missing, unavailable, or unknown assignments
• One-time variant resolution
• Ordered actions with per-action error isolation
• Idempotent experiment-scoped CSS injection
• Exact and dynamic CSS selectors
• Styles, DOM properties, attributes, and class updates
• Optional bounded MutationObserver waiting for delayed elements
• Per-element and experiment-level custom functions
• One-line and raw/gzip size-budget verification
• No eval, global variables, polling, or runtime dependencies

See Current features and configuration for the complete action schema.

Commands

Command	Purpose
npm run build	Generate dist/experiment.min.js.
npm test	Run assignment and action tests.
npm run check:artifact	Verify one-line output and size budgets.
npm run check:docs	Verify required documentation and version references.
npm run check	Run the complete pre-deployment check.

The generated artifact is committed intentionally. Continuous integration rebuilds it and fails if the committed output differs from the source.

Project structure

src/config/       Experiment-specific configuration
src/critical/     Assignment and one-time bootstrap
src/providers/    Assignment-platform adapters
src/runtime/      Ordered variant execution
src/actions/      CSS, element, and custom action dispatch
src/dom/          DOM querying and update utilities
src/utils/        Validation and diagnostics
scripts/          Build and consistency checks
test/             Browser-behavior tests
dist/             Committed copy-paste artifact
docs/             Architecture and feature reference

Keeping source, artifact, and docs in sync

Every behavior change must include:

1. The source change.
2. A corresponding test.
3. Documentation updates when the public configuration contract changes.
4. A CHANGELOG.md entry.
5. A rebuilt dist/experiment.min.js.

Run npm run check before committing. CI independently repeats the tests, build, artifact checks, documentation checks, and generated-file diff.

Possible CDN distribution later

The source architecture does not depend on copy-paste deployment. A future CDN release can build a stable runtime bundle that exposes one namespaced API, while small inline experiment configurations call that runtime.

That migration should happen only when centralized caching and rollout control outweigh the extra first-visit request, cache invalidation requirements, failure mode, and version coordination. The existing modules, tests, action schema, and PostHog provider can remain unchanged; only the entry point and release pipeline need to differ.

Status

The project is at 0.1.0: usable for controlled Shopify experiments, with the API still open to refinement before a stable 1.0.0 contract.

See Roadmap, Changelog, and Contributing.

License

MIT