An end-to-end real-time AI application that integrates browser extensions, computer vision, MediaPipe landmark extraction, temporal sequence modeling, FastAPI services, and real-time UI overlays.
- Real-Time Video Injection: Overlays subtitles directly inside the YouTube
.html5-video-playerDOM, fully supporting fullscreen mode. - Single Page Application (SPA) Resilience: Gracefully handles YouTube's dynamic page navigation without crashing or requiring page reloads.
- Zero-Queue Backpressure: Hardcoded concurrency locks prevent the backend from being overloaded and guarantee sub-150ms real-time processing.
- Temporal ML Smoothing: Combines 30-frame sequence buffering, confidence-threshold gating (0.65), and rolling majority-vote logic to completely eliminate UI flickering.
graph TD
A[YouTube Video Element] -->|Canvas Extract| B[Chrome Extension Content Script]
B -->|HTTP POST Base64 JPEG| C[FastAPI Backend]
subgraph Backend Pipeline
C --> D[MediaPipe Holistic Extractor]
D -->|543x3 Landmarks| E[30-Frame Rolling Buffer]
E -->|When Full| F[1D CNN + Transformer Model]
F -->|Raw Prediction & Confidence| G[Confidence Filter]
G -->|> 0.65 Threshold| H[Prediction Smoother]
end
H -->|Stable Label| I[JSON Response]
I -->|Update UI| J[YouTube Subtitle Overlay]
- Frontend/Extension: JavaScript (Manifest V3), HTML, CSS
- Backend API: Python, FastAPI, Uvicorn
- Computer Vision: Google MediaPipe Holistic (Pose, Face, Hands)
- Machine Learning: TensorFlow / Keras (1D CNN + Transformer architecture)
git clone https://github.com/AdemGhalleb/SignLens.git
cd SignLensThe backend requires Python 3.11+.
# Set up virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .\.venv\Scripts\activate
# Install dependencies
pip install -r backend/requirements.txt
# (Optional) Create environment variables
cp .env.example .env
# Run the FastAPI server
cd backend
uvicorn app:app --host 0.0.0.0 --port 8000 --reload- Open Google Chrome and navigate to
chrome://extensions/. - Enable Developer mode in the top right corner.
- Click Load unpacked in the top left corner.
- Select the
SignLens/extensiondirectory from this repository. - Open any YouTube video and the extension will automatically initialize.
This project is engineered to demonstrate real-time AI integration and browser architecture. It is not a state-of-the-art ASL translation model.
- Vocabulary Limit: The model can only recognize the specific classes present in the training dataset.
- Word-Level Only: Current implementation performs isolated, word-level recognition, not fluent continuous conversational translation.
- Out of Distribution: Unknown signs or arbitrary hand waving may be misclassified if they bypass the 0.65 confidence filter.
- Accuracy Constraints: Recognition accuracy depends heavily on video quality, framerate, and clear visibility of the signer's hands and face.
Contributions are welcome, especially regarding backend performance optimizations, model improvements, or expanding support to platforms beyond YouTube (e.g., Zoom Web, Twitch).
This project is licensed under the MIT License - see the LICENSE file for details.