SignLingo — Learn ASL with AI

SignLingo is a web-based, AI-powered ASL (American Sign Language) learning platform. Open your browser, turn on your webcam, and the app watches your hands in real-time — giving instant feedback on whether you signed a letter, word, or phrase correctly.

No downloads. No special hardware. Just a browser and your hands.

Features

• Real-time Hand Detection — MediaPipe Hands tracks 21 landmarks per hand at 30fps
• AI Classification — TensorFlow.js model classifies ASL signs entirely in-browser (no video leaves your device)
• Structured Curriculum — Units, lessons, and exercises from fingerspelling to full words
• Progress Tracking — XP, levels, streaks, and accuracy metrics
• Practice Mode — Free-form signing with live AI feedback
• Privacy First — All ML runs locally. No camera data is ever sent to a server.

Tech Stack

Layer	Technology
Framework	Next.js 14 (App Router, TypeScript)
Hosting	Vercel
Database	Supabase (PostgreSQL + Auth + Storage)
ML (browser)	TensorFlow.js + MediaPipe Hands
Auth	Supabase Auth (with modular Clerk adapter)
Styling	Tailwind CSS
UI Components	shadcn/ui
Charts	Recharts
State	Zustand

Getting Started

Prerequisites

• Node.js 18+
• npm
• A Supabase project (free tier works)
• A webcam

1. Clone and Install

git clone <your-repo>
cd signlingo
npm install

2. Set Up Supabase

1. Create a project at supabase.com
2. In the SQL Editor, run supabase/migrations/001_initial_schema.sql to create the schema
3. Run supabase/seed.sql to populate the curriculum
4. Enable email/password auth and get your project URL and anon key

3. Configure Environment

cp .env.local.example .env.local

Edit .env.local with your Supabase credentials:

NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_ROLE_KEY=your-service-role-key

4. Run Development Server

npm run dev

Open http://localhost:3000. Sign up and start learning!

Project Structure

src/
├── app/                    # Next.js App Router pages
│   ├── (auth)/             # Sign in, sign up, forgot password
│   ├── (app)/              # Protected routes (dashboard, learn, practice, profile)
│   └── api/                # API routes (progress, profile, webhooks)
├── components/
│   ├── ui/                 # Reusable UI components (button, card, input, etc.)
│   ├── auth/               # Authentication forms and guards
│   ├── camera/             # Webcam, hand landmarks, sign feedback
│   ├── lesson/             # Exercise card, progress bar, XP toast
│   ├── dashboard/          # Stats cards, streak calendar, accuracy radar
│   └── layout/             # Navbar
├── hooks/                  # useWebcam, useASLDetector, useAuth, useProgress
├── lib/
│   ├── auth/               # Auth adapter pattern (Supabase active, Clerk stubbed)
│   ├── email/              # Email adapter pattern (stub active, Resend stubbed)
│   ├── supabase/           # Client, server, schema types
│   ├── ml/                 # MediaPipe, TF.js classifier, normalization, labels
│   ├── curriculum/         # Static curriculum data
│   └── utils/              # cn (Tailwind merge), XP calculator
├── store/                  # Zustand store for session state
└── types/                  # Shared TypeScript types

Switching Auth Providers

SignLingo uses a modular adapter pattern:

1. Supabase Auth (active by default) — configured in src/lib/auth/supabase.ts
2. Clerk (stubbed) — swap by:

   npm install @clerk/nextjs

   Set NEXT_PUBLIC_AUTH_PROVIDER=clerk in your env vars Implement src/lib/auth/clerk.ts

Training the ASL Model

The ML model is trained offline using Python:

cd ml
pip install -r requirements.txt

# 1. Extract landmarks from images
python extract_landmarks.py --data-dir data --output landmarks.csv

# 2. Train the classifier
python train.py --data landmarks.csv --model-dir models/asl_classifier

# 3. Export to TF.js format
python export_tfjs.py --model models/asl_classifier/model.h5 --output ../public/models/asl_classifier/

Deployment

Deploy to Vercel

1. Push to GitHub
2. Import into Vercel
3. Add environment variables (from .env.local.example)
4. Configure Supabase Database and Auth in Vercel env
5. Deploy!

License

MIT