Skip to content
AdemGhallebPublic

About

SignLens — An end-to-end real-time ASL recognition system integrating computer vision, sequence modeling, FastAPI, and a Chrome extension to translate recognized signs into YouTube subtitles.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Latest commit

 

History

20 Commits

Folders and files

Repository files navigation

SignLens

An end-to-end real-time AI application that integrates browser extensions, computer vision, MediaPipe landmark extraction, temporal sequence modeling, FastAPI services, and real-time UI overlays.


Features

  • Real-Time Video Injection: Overlays subtitles directly inside the YouTube .html5-video-player DOM, fully supporting fullscreen mode.
  • Single Page Application (SPA) Resilience: Gracefully handles YouTube's dynamic page navigation without crashing or requiring page reloads.
  • Zero-Queue Backpressure: Hardcoded concurrency locks prevent the backend from being overloaded and guarantee sub-150ms real-time processing.
  • Temporal ML Smoothing: Combines 30-frame sequence buffering, confidence-threshold gating (0.65), and rolling majority-vote logic to completely eliminate UI flickering.

System Architecture

graph TD
    A[YouTube Video Element] -->|Canvas Extract| B[Chrome Extension Content Script]
    B -->|HTTP POST Base64 JPEG| C[FastAPI Backend]
    
    subgraph Backend Pipeline
    C --> D[MediaPipe Holistic Extractor]
    D -->|543x3 Landmarks| E[30-Frame Rolling Buffer]
    E -->|When Full| F[1D CNN + Transformer Model]
    F -->|Raw Prediction & Confidence| G[Confidence Filter]
    G -->|> 0.65 Threshold| H[Prediction Smoother]
    end
    
    H -->|Stable Label| I[JSON Response]
    I -->|Update UI| J[YouTube Subtitle Overlay]
Loading

Tech Stack

  • Frontend/Extension: JavaScript (Manifest V3), HTML, CSS
  • Backend API: Python, FastAPI, Uvicorn
  • Computer Vision: Google MediaPipe Holistic (Pose, Face, Hands)
  • Machine Learning: TensorFlow / Keras (1D CNN + Transformer architecture)

Installation & Setup

1. Clone the Repository

git clone https://github.com/AdemGhalleb/SignLens.git
cd SignLens

2. Backend Setup

The backend requires Python 3.11+.

# Set up virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .\.venv\Scripts\activate

# Install dependencies
pip install -r backend/requirements.txt

# (Optional) Create environment variables
cp .env.example .env

# Run the FastAPI server
cd backend
uvicorn app:app --host 0.0.0.0 --port 8000 --reload

3. Chrome Extension Setup

  1. Open Google Chrome and navigate to chrome://extensions/.
  2. Enable Developer mode in the top right corner.
  3. Click Load unpacked in the top left corner.
  4. Select the SignLens/extension directory from this repository.
  5. Open any YouTube video and the extension will automatically initialize.

Limitations & Honesty

This project is engineered to demonstrate real-time AI integration and browser architecture. It is not a state-of-the-art ASL translation model.

  • Vocabulary Limit: The model can only recognize the specific classes present in the training dataset.
  • Word-Level Only: Current implementation performs isolated, word-level recognition, not fluent continuous conversational translation.
  • Out of Distribution: Unknown signs or arbitrary hand waving may be misclassified if they bypass the 0.65 confidence filter.
  • Accuracy Constraints: Recognition accuracy depends heavily on video quality, framerate, and clear visibility of the signer's hands and face.

Contributing

Contributions are welcome, especially regarding backend performance optimizations, model improvements, or expanding support to platforms beyond YouTube (e.g., Zoom Web, Twitch).

License

This project is licensed under the MIT License - see the LICENSE file for details.

About

SignLens — An end-to-end real-time ASL recognition system integrating computer vision, sequence modeling, FastAPI, and a Chrome extension to translate recognized signs into YouTube subtitles.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages