Skip to content

Conv-AI/Classic-Chess

BranchesTags

Repository files navigation

Classic Chess

Classic Chess is a React, TypeScript, Vite chess coaching app. It combines chess.js for board rules, Stockfish for coach moves, Convai for spoken coach dialogue, and Three.js / React Three Fiber for animated Reallusion coach avatars.

Features

  • Optional Google sign-in for Convai long-term memory — published for external Google accounts (guest play unchanged; see docs/google-convai-memory-tutorial.md)
  • Quick Play against Magnus, Sofia, Arjun, Leila, or custom coaches you create — Leila is the default coach
  • Menu coach headshots — circular portrait avatars per coach (public/coach-portraits/), rendered from the in-game models with unified framing; sharp 384×384 thumbnails for the picker
  • 3D coach portrait — studio lighting with frontal fill, adaptive canvas DPR, optional bloom/vignette post-processing (PortraitScene); phones load lighter .mobile.glb builds with post-FX off
  • Full-face lipsync — the complete 62-channel Convai neurosync performance: mouth visemes, brows, squints, gaze, performance blinks, and head sway (channels 52-54); expression afterglow lets a smile linger after speech while articulation releases fast
  • AI coach speech through Convai with two coaching strategies:
    • Coach mode (default) — Convai's LLM sees full board context every turn and decides whether to speak (run_llm: 'auto')
    • Game mode — local heuristics in analyzeCoachMoveContext() pick teaching moments; the coach only speaks on captures, tactics, king safety, and similar events
  • Coaching Control toggle on the menu (persisted in localStorage, defaults to Coach)
  • Corner chat drawer — chat button on the coach card; drawer opens bottom-right during play and post-game analysis
  • Mic on coach card — microphone toggle lives in the caption bar under the portrait (removed from the game topbar)
  • Turn feedback — calm chime + subtle board pulse when it becomes your turn
  • UI sounds — gentle chimes for navigation, confirm, toggle, send, and tap actions (src/uiSounds.ts)
  • API key management — Convai key in .env.convai.local (local dev), GitHub Secret on deploy, in-app modal, or localStorage; masked key badge on the menu (09***)
  • Custom coach creator — Convai Core API with language-filtered voices and verified model list (default: gemini-2.5-flash-lite)
  • Board vision — chess board canvas is published via Convai Vision Dynamic Context (@convai/web-sdk@1.6.0-beta.1) so the coach sees the position alongside text context
  • Microphone toggle for voice chat, off by default
  • Puzzle mode with hints, five-puzzle groups, and mistake review
  • Saved games and replay
  • Post-game analysis with Menu button and coach chat
  • Responsive desktop, ultrawide, and mobile layouts (turn-card panel scrolls when clipped on narrow viewports)
  • Loading cover through Convai connect, game setup, and a short “taking your seat” pause — welcome speech starts ~1s after the board appears
  • Portal-based tooltips (not clipped by card overflow)
  • Optional Dialogue Dataset tooling (npm run dev:dataset)

Character assets (Git LFS)

Coach avatars are Reallusion CC4 characters. The raw exports (~540 MB total) live in assets-src/; the app ships the optimized builds in public/ (~48 MB desktop + ~27 MB mobile), produced by scripts/optimize-coach-models.mjs:

  • Morph targets pruned to the set the lipsync/blink runtime actually drives (349–437 → 68–133 per mesh). The raw exports exceed the 256-layer DataArrayTexture limit of many mobile GPUs and need >2 GB of VRAM — phones cannot render them.
  • Leila's export gets a nose repair pass: her mouth visemes dragged nostril vertices (the "pink peel" artifact), so per-vertex deltas are attenuated by how strongly the nostril morphs own each vertex — funnels/O-shapes and visible teeth became safe.
  • Textures resized per region and re-encoded to WebP; geometry quantized + meshopt-compressed.
  • Each coach gets <name>.glb (desktop, 2K textures) and <name>.mobile.glb (1K textures, clearcoat/specular stripped). CoachCard picks the variant via isMobilePortrait().

All .glb files are tracked with Git LFS. After cloning, install LFS and pull the models once:

git lfs install
git lfs pull --include="public/**"   # site assets only; drop --include to also get assets-src/ raw exports

After changing a raw export in assets-src/ (or the morph keep-list in scripts/portrait-morph-keep.json), regenerate and spot-check the optimized builds:

npm run models:optimize   # writes public/<name>.glb + public/<name>.mobile.glb
npm run models:verify     # renders every coach x variant through Three.js (real GPU) into .verify-out/

src/coachModelKeepList.test.ts fails if the runtime starts driving a morph that is missing from the keep-list. The *-animations.glb idle clips are used as-is (they contain no morph tracks). The app loads everything from public/ by default (no external CDN required).

Menu headshots live in public/coach-portraits/ (magnus.png, sofia.png, etc.). The coach picker uses pre-generated *-thumb.png files (192×192, Lanczos downscale). After replacing a portrait source image, regenerate thumbs:

npm run portraits:thumbs

Run

npm install
npm run dev

Dataset tooling (extra menu tile, coach-card ➕ button, /api/dataset, Dataset screen):

npm run dev:dataset

Build and test:

npm run build
npm run test
npm run portraits:thumbs   # regenerate menu headshot thumbnails after changing coach-portraits/*.png

Uses @convai/web-sdk@1.6.0-beta.1.

Environment

Local Convai API key — copy .env.convai.local.example to .env.convai.local and paste your key. This file is gitignored (via .env.*) and is what npm run dev uses (loaded in vite.config.ts). Do not put the Convai key in .env.

Other local settings — copy .env.example to .env for Google sign-in (VITE_GOOGLE_CLIENT_ID, GOOGLE_CLIENT_ID). Never commit .env or .env.convai.local.

cp .env.convai.local.example .env.convai.local   # then edit — Convai API key
cp .env.example .env                             # Google client ID, optional guest clones

.env.convai.local takes priority over a key in .env or localStorage when present.

GitHub Actions secrets (Pages deploy)

The deploy workflow (.github/workflows/deploy-pages.yml) injects secrets at build time — they are not stored in the repo.

  1. Regenerate your Convai API key in the Convai dashboard (revoke the old one if it was ever committed).
  2. Open your GitHub repo → Settings → Secrets and variables → Actions.
  3. Click New repository secret and add:
    • VITE_CONVAI_API_KEY — your new Convai API key (required for deploy).
    • VITE_GOOGLE_CLIENT_ID — Google OAuth Web client ID (optional; enables sign-in on the deployed site).
  4. Push to master or run the Deploy GitHub Pages workflow manually.

Important: VITE_* variables are embedded in the built JavaScript bundle. GitHub Secrets keep the key out of git history, but anyone who loads your public Pages site can still extract it from the client bundle. For a fully private key, serve short-lived Convai auth tokens from a backend instead of baking VITE_CONVAI_API_KEY into the frontend.

Optional local overrides:

# Optional: LTM-disabled Convai character clones for anonymous guests (one per builtin coach)
# VITE_CONVAI_GUEST_CHARACTER_LEILA=...

# Optional: override character GLB asset base URL (defaults to bundled public/ assets)
# VITE_CHARACTER_ASSET_BASE_URL=https://...

Builtin coach character IDs live in src/coachConfig.ts. Custom coaches are saved locally and merged into the menu via getAllCoaches(). Each builtin coach has portraitFile and optional portraitFocusY for menu crop alignment.

Recommended LLM for live coaching: Gemini 2.5 Flash Lite or Claude 4 Sonnet (configure in the Convai dashboard for each character).

Code Map

File Role
src/App.tsx Screens, game loop, chat drawer, API key modal, custom coach creator, turn/UI sound wiring
src/MenuScreen.tsx Menu, coach/difficulty pickers with headshot avatars, coaching control toggle, API key badge
src/CoachCard.tsx Avatar card, PortraitScene canvas, adaptive DPR, chat toggle, mic slot, optional dataset button
src/PortraitScene.tsx Portrait lighting, environment map, optional bloom/vignette post-FX
src/ReallusionCharacter.tsx GLB avatar, ARKit→CC4 lipsync, idle sanitization, procedural blink, jaw-bone motion
src/cc4Lipsync.ts Per-asset lipsync profiles, morph mapping, Leila thin-lip / nostril-safe tuning
src/cc4Correctives.ts CC4 C_* corrective morph products (optional per profile)
src/cc4TeethMotion.ts Jaw / teeth / tongue bone motion (jawOnly mode for Leila)
src/convaiLipsyncPlayer.ts Single-clock 60fps blendshape consume loop (Convai reference pattern)
src/isMobilePortrait.ts Mobile GLES portrait rendering guards
src/portraitBlink.ts Procedural eye-blink morph driver for idle + speech
src/sanitizeIdleClip.ts Portrait-safe idle clip cleanup (bone root, head/eye lock)
src/MicButton.tsx Shared Convai microphone toggle
src/useOverflowScroll.tsx ScrollWhenClipped — enables scroll only when panel content overflows
src/ChatDrawer.tsx Fixed bottom-right chat panel
src/Tooltip.tsx Viewport-aware portal tooltips
src/uiSounds.ts Web Audio UI chimes (tap, nav, back, confirm, toggle, send, yourTurn)
src/auth.ts Google identity mapping, stable guest endUserId, LTM gating
src/convaiEndUsers.ts Convai end-user list/delete API helpers and MAU-limit detection
src/convaiApiKey.ts API key from localStorage + env fallback
src/customCoaches.ts localStorage for user-created coaches
src/convaiManager.ts Convai lifecycle, static policy, dynamic context, speech queue, lipsync
src/convaiCoreApi.ts Convai REST helpers (voices, languages, character create/update)
src/chessAi.ts Board context, coach instructions, Game-mode speech gating, welcome variety
src/boardVision.ts Optional board canvas → LiveKit video track
src/coachConfig.ts Personas, difficulties, getAllCoaches()
src/storage.ts Sessions, puzzle progress, coachingControlMode (default: coach)
src/debugLog.ts Browser → debug.log via /api/log
scripts/generate-coach-portrait-thumbs.mjs Build 384×384 menu headshots from public/coach-portraits/*.png
scripts/optimize-coach-models.mjs Raw CC4 export → optimized web/mobile GLBs (morph prune, nose repair, WebP, meshopt)
scripts/verify-coach-models.mjs Real-GPU render check of every coach × variant

Portrait & Lipsync

  • Idle animation — sanitizePortraitIdleClip() locks CC_Base_BoneRoot, hip, spine, head, jaw, and eye bones; strips drifting eye translation so the coach faces the camera without hip sway or nostril flicker.
  • Blinking — performance blinks from Convai channels 0/7 during speech (never attenuated, mirrored onto lash/lid/occlusion follower meshes); procedural Eye_Blink_L/R (portraitBlink.ts) fills the idle gaps.
  • Lipsync playback — convaiLipsyncPlayer.ts drives ARKit frames through one 60fps consumeFrames clock. The held frame releases ~450ms after the queue runs dry (even when the SDK's isBotSpeaking flag wedges), and ReallusionCharacter has a 600ms stale-frame failsafe so the face always decays to neutral between utterances.
  • CC4 mapping — cc4Lipsync.ts maps the full ARKit set to CC4 morphs with per-asset LIPSYNC_PROFILES: mouth visemes plus brows, squints, eye-wide, gaze, cheek raise and sneer (attenuated); head rotation channels 52-54 sway CC_Base_Head additively.
  • Expression afterglow — after speech, articulation (jaw, visemes, lip rolls/presses) decays fast while expressions (smile, frown, cheeks, brows) linger toward a 0.45 cap and fade over a few seconds. Mouth_Corner_Pull/Mouth_Dimple are deliberately NOT held — pulled corners across closed lips read as lip-biting.
  • Leila (leila.glb) — jaw-bone rotation (not face Jaw_Open/V_Open morphs — broken in the export), boosted lower-lip / V_Lip_Open drive, funnels and visible teeth enabled by the build-time nose repair; nose/cheek/tongue morphs stay blocked on the face mesh.
  • Rendering — PortraitScene uses soft directional lights + frontal fill (lifts the eye sockets), a procedural RoomEnvironment map (no CDN fetch), ACES tone mapping, and bloom/vignette on desktop. Mobile skips the composer (CSS vignette instead), caps DPR at 1.5×, and loads the .mobile.glb variant.

Coaching Control

Coach mode (default) Game mode
Who decides Convai LLM every turn Local analyzeCoachMoveContext() heuristics
Context Full dynamic board info each turn Same info, but LLM only invoked on teaching moments
Best for Conversational, adaptive coaching Quieter games with commentary on notable moments

Puzzles and chat always use explicit scripted prompts.

Coach move timing

The coach applies her Stockfish move only after TTS finishes. runCoachTurn uses waitForFullSpeech: true and waitUntilSpeechFinished() (SDK speaking state, blendshape queue, stuck-audio detection, and queue getTimeLeftMs / isAllFramesConsumed) so moves follow speech without a multi-second gap or premature cut-off. The status line avoids a post-speech “thinking” flash by holding recentCoachSpeech until the Convai turn completes.

Welcome lines

Opening greetings use varied casual lines from buildWelcomeDynamicInfo(). During Quick Play, setup runs behind the loading overlay; the board reveals with a brief pause, then the welcome line plays so players are not hit with speech during the screen transition.

Convai Integration

  • Static coaching policy seeded once with keepInContext: true; per-turn updates send dynamic board state only (reduces prompt leak and latency).
  • updateContext({ mode: 'replace', run_llm }) for coach-decides turns; updateDynamicInfo for silent context refresh in Game mode.
  • Prompt-leak responses starting with Human: / System: / User: are suppressed.
  • End-user identity: signed-in Google users connect as google:{sub} with LTM memory writes; guests use a stable per-browser guest:{uuid} in localStorage (no app-side LTM writes). Game session ids remain separate for saved games.
  • On Convai MAU limit errors, the app deletes known end users via the Convai API and retries connect once.
  • New game: resetSession + context reset; welcome is delivered after loading peels (Quick Play) or inline on rematch.
  • Game-over modal waits until the coach finishes speaking.

Custom Coach Creator

  1. Open Custom Coach from the menu (API key required).
  2. Voices and languages load automatically; the voice list filters to the selected language.
  3. Default model: gemini-2.5-flash-lite (Convai-supported model codes only).
  4. On success, the coach is saved to localStorage and appears in the menu coach picker (Leila portrait placeholder by default).

Google Sign-In (production)

The OAuth consent screen is configured as External and published, so any Google account can sign in (not limited to a Workspace org or manual test-user list). Users may still see Google's unverified-app warning until Google verification completes; they can proceed via Advanced → Go to Classic Chess (unsafe). See docs/google-convai-memory-tutorial.md for Cloud Console setup and troubleshooting.

Menu Coach Portraits

Builtin coaches reference PNG headshots under public/coach-portraits/. Config fields:

  • portraitFile — full-resolution source (used as fallback for custom coaches)
  • portraitFocusY — vertical crop focus (percent from top) when generating thumbs

The picker loads getCoachPortraitThumbUrl() (*-thumb.png). Regenerate after changing sources with npm run portraits:thumbs (uses sharp dev dependency).

Debug Logging

debugLog(scope, message) → console + batched POST to /api/log → debug.log in the project root.

Useful lines: [Convai] BOT READY, FINAL, Coach move applied, Published live board video track, coachingControlMode -> coach|game.

Copy-log button in the game topbar copies the in-browser log buffer.

URL debug flags (?portraitDebug=...): decay logs face morphs + jaw/teeth/head pose every tick (idle vs speech vs decay); basic swaps portrait materials to unlit MeshBasicMaterial. The portrait capture page (scripts/portrait-capture.html, used by the render/verify scripts) accepts ?coach=, &variant=mobile, &model=/file.glb, &pose=Morph:value,..., &hide=meshPattern, &showOral=1 for A/B checks of model builds.

References

About

Play chess with AI coaches who talk you through every game

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages