Finger Spelling + Digit Recogniser

This program allows users to capture finger spelling (A to Z excluding J and Z) and digits (0 to 9) and recognise them by using AI and machine learning in Python. It uses the MNIST dataset and PyTorch, an open-source machine learning library, to train and recognise digits.

📖 Project Overview

This application provides a solution for sign language recognition with:

• 36-class classification (26 letters + 10 digits) - excluding J and Z
• PyQt GUI with interactive controls
• Dataset management with download, search, and progress tracking features
• Multiple CNN models for training (MobileNetV2, EfficientNetB0, and a custom CNN)
• Real-time predictions via webcam, image capture or dataset image upload
• Training configuration and visualisation with live metrics
• Interactive system and navigation controls

---

🛠️ Installation and Setup

This project uses Anaconda to manage the environment and dependencies. Make sure Anaconda is installed on your system before proceeding.

1. Create and activate the environment:

   Open Anaconda Prompt and run

   conda create -n cs302 python=3.9
   conda activate cs302

2. Install required packages:

   pip install torch==2.2.1 torchvision opencv-python==4.7.0.72 PyQt5==5.15.10 numpy==1.24.4 matplotlib mediapipe tqdm

3. Set Python interpreter in VS Code:

   • Open the project folder in Visual Studio Code.
   • Select interpreter: Python 3.9.21 ('cs302': conda)

---

▶️ Using the Application

Navigation Guide

• Use the menu bar for primary functions
• Access quick help instructions via i icon (bottom-right)

Step-by-Step Application Flow

1. Open and run app.py.
2. A window titled "SIGNET AI" will appear.
3. Click Load Data → Import Dataset.
4. In the "Import Data" window:
   • Click Select File.
   • Select your dataset of images.
   • After the dataset is selected, click Download (this will take a few seconds).
   • Monitor progress with time estimates and a progress bar.
   • Can cancel anytime using Cancel button
5. Explore Dataset
   • Scroll through dataset images
   • Filter by class/character using search bar
   • View class distribution statistics
6. Click Train → Train Model.
7. In the "Train Model" window:
   • Select your preferred model in the Select Model dropdown.
   • Configure your Test/Train Ratio, Batch Size, Epoch Number with the respective sliders.
   • Click Start Training (this will take a minutes to hours).
   • Stop training anytime with Stop Training button.
   • View live training loss/validation accuracy graphs, estimated time to completion, and progress with the progress bar.
8. Once training reaches 100%, to test the model:
   • Click Test → Select Model
   • Select the saved model from the saved_models folder
   • Click on an image and a probability plot with the prediction will pop up.
9. For Webcam Prediction:
   • Click Test → Open Webcam for live detection
   • Select a pre-trained saved model from the saved_models folder
     • Shows live translation
     • Hand tracking visualisation
   • Click Capture Photo for image capture prediction
     • Shows probability plot with prediction.
10. To quit:
    • Click the close window button

---

🏆 Performance

• Achieves 95%+ validation accuracy in 30 epochs

---

👥 Contributors

• Jayti Pattni
• Rudra Patel
• Jerry Kim