ECE 228 — Machine Learning for Physical Applications
UC San Diego · Team 18 · Spring 2026
Shubhan Mital · Shlok Rupesh Bagmar · Shambhavi Singh · Mitali Agrawal
StayTuned estimates whether a student or driver is attentive or distracted in real time by combining visual cues (face geometry, gaze, head pose) and audio cues (microphone activity during a lecture). The system is complete and deployable: trained audio and video classifiers are fused in Combined/, which supports offline video batch processing, live webcam + microphone inference, and validation against ground-truth spreadsheets. A companion ESP32 OLED + buzzer hardware alert is provided under Hardware_oled_code/.
- System Overview
- Repository Structure
- Visual Pipeline (Active Path)
- Audio Pipeline
- Alternate Visual Approach (TBD)
- Multimodal Fusion
- Hardware Alert Display
- Current Results
- Requirements & Installation
- How to Run
- Design Decisions
- Known Limitations & Future Work
The project is split into independently trainable subsystems that feed a shared fusion layer:
┌─────────────────────────────────────┐
microphone ───► │ AUDIO COMPONENT │
│ Audio_Classifier/ ├──► s_audio ∈ [0, 1]
│ (Whisper → chunks → ML classifier) │
└─────────────────────────────────────┘
│
▼
┌──────────────────────┐
│ Combined/ │──► Excel report / live alert
│ weighted fusion │ + optional OLED hardware
└──────────────────────┘
▲
┌─────────────────────────────────────┐
webcam ───► │ VIDEO COMPONENT │
│ Video_Classifier/ (Steps 1–5) ├──► s_video ∈ [0, 1]
│ OR TBD/ CNN ensemble (PyTorch) │
└─────────────────────────────────────┘
| Component | Technology | Training data | Output artifact |
|---|---|---|---|
| Video (primary) | MediaPipe landmarks → 10 geometric features → BiLSTM | DAiSEE video dataset | Video_Classifier/results_attention/bilstm_attention.keras, inference_config.json |
| Video (alternate) | Dlib HOG crop → ResNet / AlexNet / VGG / FaceNet ensemble | Human Faces images + DAiSEE frames | TBD/models/*.pth (when trained) |
| Audio | Whisper transcripts → 5–10 s chunks → ~75 audio features → LR / RF | Team-recorded lecture audio | Audio_Classifier/model_output/trained_model.joblib |
| Fusion | Weighted combination in 5-second intervals | Team evaluation videos | Combined/multimodal_attention_detector.py → s_fused = 0.85·s_video + 0.15·s_audio |
| Hardware | ESP32 + SSD1306 OLED + piezo buzzer | — | Hardware_oled_code/oled_pay_attention.ino |
What is in this repo: the full geometric visual pipeline (Video_Classifier/, Steps 1–5) with trained BiLSTM weights, the complete audio preprocessing + training stack (Audio_Classifier/), the multimodal integrator (Combined/) for offline video analysis and live webcam inference, validation tooling, hyperparameter tuning for fusion weights, and the physical alert hardware firmware (Hardware_oled_code/).
Codes/
│
├── README.md # This file — full project overview
│
├── Video_Classifier/ # Visual subsystem (primary path)
│ ├── Step 1 - Point detection for images.py
│ ├── Step 2 - EAR MAR Gaze ratios for images.py
│ ├── Step 3 - Feature extraction for videos.py
│ ├── Step 4 - Complete Visual data.py
│ ├── Step 5 - ML Model.py
│ ├── results_attention/ # Trained BiLSTM + metrics
│ └── README_Visual.md # Visual pipeline deep dive
│
├── Audio_Classifier/ # Audio subsystem
│ ├── audioconverter.py # 6-stage preprocessing pipeline
│ ├── feature_extractor.py
│ ├── train_model.py
│ ├── temporal_smoothing.py
│ ├── model_output/ # trained_model.joblib + reports
│ └── README.md
│
├── Combined/ # Multimodal fusion (final integrator)
│ ├── multimodal_attention_detector.py # Offline video + live webcam
│ ├── validate_predictions.py # Compare predictions vs ground truth
│ ├── Hyperparameter Tuning.py # Sweep fusion weights & thresholds
│ └── README_combined.md
│
├── Hardware_oled_code/ # Physical alert display + buzzer
│ ├── oled_pay_attention.ino
│ └── README.md
│
└── TBD/ # Alternate CNN-based visual path
├── Facial feature extractor.py
└── Facial feature extractor README.md
External data (not in repo):
- DAiSEE — 10-second
.aviclips with Boredom, Engagement, Confusion, Frustration labels (0–3). Used for the geometric + BiLSTM path. - Human Faces — six labelled image folders (
Closed,Drowsy,yawn, etc.) for Steps 1–2 and the CNN ensemble path. - Team audio — per-person folders under
audio/<Name>/with filename-encoded labels (focused,talking, etc.). - MediaPipe model —
face_landmarker.task(download from MediaPipe; path configured in Step scripts).
The primary visual approach does not train end-to-end on pixels. It extracts interpretable geometric features per frame, then trains a Bidirectional LSTM on short temporal windows. This matches what a live deployment can compute from a webcam without DAiSEE labels.
| Step | Script | Purpose | Key outputs |
|---|---|---|---|
| 1 | Step 1 - Point detection for images.py |
MediaPipe Face Landmarker on static images; 21 keypoints (eyes, mouth, irises) | train_landmarks.csv, test_landmarks.csv |
| 2 | Step 2 - EAR MAR Gaze ratios for images.py |
EAR (blink), MAR (yawn), normalized gaze offsets from landmarks | ear_mar_dataset.csv |
| 3 | Step 3 - Feature extraction for videos.py |
DAiSEE videos: parallel CPU extraction, blink/yawn state machines, optional frame skip | Raw feature parquets per split |
| 4 | Step 4 - Complete Visual data.py |
Everything in Step 3 plus 3D PnP head pose (yaw/pitch/roll, yaw velocity, geometric AttentionScore) | daisee_features_train.parquet, _validation.parquet, _test.parquet |
| 5 | Step 5 - ML Model.py |
Binary attention classifier: sliding windows → BiLSTM v2 → metrics & live inference config | results_attention/ |
Steps 1–2 validate feature math on the Human Faces image set. Steps 3–4 scale to DAiSEE. Step 5 is the production-oriented model.
After Step 4, each frame has geometric features only — no raw labels in the model:
| Feature | Description |
|---|---|
| EAR | Eye Aspect Ratio — eye closure / drowsiness |
| MAR | Mouth Aspect Ratio — yawning |
| Gaze_H / Gaze_V | Iris position normalized within eye box (0–1) |
| BlinkRate / YawnRate | Cumulative events per second (state-machine counters) |
| Yaw / Pitch / Roll | Head pose (degrees) via cv2.solvePnP on 6-point 3D face model |
| YawVelocity | Rolling mean absolute change in yaw |
AttentionScore (geometric proxy for head toward camera) is stored in parquet for analysis but excluded from training to avoid label leakage.
The four DAiSEE columns are combined once into a composite score, then binarised. Labels are never passed to the model:
score = 0.40 × Engagement
+ 0.30 × (3 − Boredom)
+ 0.15 × (3 − Confusion)
+ 0.15 × (3 − Frustration)
Attention = 1 if score ≥ T_label else 0
Current T_label = 2.5 (in Step 5 - ML Model.py). At the older threshold 1.8, ~90% of windows were labelled attentive, so the model could cheat by always predicting 1. At 2.5, the split is roughly balanced (~55–65% positive), giving a usable learning signal.
After labelling, these columns are dropped: Boredom, Engagement, Confusion, Frustration, AttentionScore.
- Window: 5 seconds of frames (e.g. 150 frames at 30 fps) → 5 timesteps × 1 second each; each timestep = mean of its 1-second block across the 10 features.
- Training stride: 5 frames (dense overlap for data augmentation).
- Val / test stride: 5 frames.
- Window label: majority vote of frame-level
Attentioninside the window.
Input (batch, 5 timesteps, 10 features)
└─ BatchNormalization
└─ Dense(64, relu) + LayerNorm
└─ BiLSTM(128, return_sequences=True) + Dropout(0.3)
└─ BiLSTM(64) + Dropout(0.3)
└─ Dense(64, relu) + BatchNorm + Dropout(0.2)
└─ Dense(32, relu)
└─ Dense(1, sigmoid) → P(Attentive)
Training (v2 vs v1): Focal Loss (γ=2, α=0.25), StandardScaler on features, class weights, early stopping on val_macro_f1, Adam 3e-4 with gradient clipping.
| Threshold | Role | Current value |
|---|---|---|
| T_label | Binarises DAiSEE composite score for training targets | 2.5 |
| T_decision | Converts sigmoid output to prediction; tuned on val ROC (Youden's J) | ~0.421 (in inference_config.json) |
Changing T_label relabels the dataset. Changing T_decision only shifts precision/recall at inference.
- Load
bilstm_attention.kerasandinference_config.json. - Run the Step 4 MediaPipe + head-pose pipeline on each frame → 10 values (with Kalman-smoothed pose).
- Keep a rolling buffer of 5 seconds of frames (~150 at 30 fps).
- Every second, build the 5×10 sequence (mean of each 1-second block), apply saved scaler, forward pass.
s_video = sigmoid(output); used directly or passed to the fusion layer inCombined/.
No DAiSEE columns exist at runtime — by design.
For step-by-step formulas, plots, and file-level detail, see Video_Classifier/README_Visual.md.
The Audio_Classifier/ folder is a self-contained subsystem that turns raw lecture/student microphone recordings into a binary attentive / distracted classifier and a deployable s_audio score.
- Convert — any format → 16 kHz mono WAV (ffmpeg, parallel).
- Transcribe — OpenAI Whisper with word-level timestamps.
- Chunk — split at natural pauses (30–60 s segments; cut at silence midpoint).
- Filter — drop too-short, too-silent, or music-like chunks.
- Split — train/test by person (never by chunk) to avoid voice leakage.
- Verify — compare Whisper output to ground-truth PDFs (
content_overlap, not WER alone). - Features — ~75 features per chunk (energy, spectral/MFCC, voice activity, pitch via Parselmouth).
- Train — Logistic Regression vs Random Forest with GroupKFold by person; save
trained_model.joblib. - Smooth — optional centred majority-vote over consecutive chunks (mirrors proposal hysteresis).
| Filename contains | Label |
|---|---|
focused, brief |
attentive |
talking, environment, passive, distract |
distracted |
Five categories: focused (~12 min), talking (8 min), environmental distraction (8 min), passive disengagement (8 min), brief interruptions (6 min). Use the laptop built-in mic to match deployment hardware.
from feature_extractor import extract_features
feats = extract_features("path/to/chunk.wav")
# Load joblib model → probability → s_audioFull documentation: Audio_Classifier/README.md (scripts, folder layout, troubleshooting, RAVDESS validation).
Under TBD/, an earlier / parallel deep learning path trains four CNNs on face crops (Dlib HOG detector → 224×224) with an unweighted softmax ensemble:
- ResNet-50, AlexNet, VGG-16 (ImageNet), FaceNet / InceptionResNetV1 (VGGFace2)
- Training data: Human Faces images + DAiSEE frames (every 10th frame cached as JPEG)
- DAiSEE label: Engagement ≥ 2 → attentive (simpler than the composite formula in Step 5)
This path optimises for clip/image classification accuracy on pixels, not the geometric live feature stream. It is kept for ablation and ensemble comparison; Steps 1–5 are the deployment-aligned visual stack.
See TBD/Facial feature extractor README.md for CONFIG reference, runtimes, and CNN design notes.
The Combined/ folder is the final deployable integrator. It loads the trained video and audio models from their sibling directories and fuses their per-interval scores.
For each 5-second interval:
s_fused = 0.85 · s_video + 0.15 · s_audio
| Parameter | Value | Notes |
|---|---|---|
| w_video | 0.85 | Raw BiLSTM sigmoid probability |
| w_audio | 0.15 | Attentive-class probability from trained_model.joblib |
| Decision threshold | 0.37 | Yes (attentive) if s_fused ≥ 0.37, else No |
If the audio model is missing, the system falls back to video-only mode (s_fused = s_video) without crashing.
Weights and threshold were tuned with Hyperparameter Tuning.py against team ground-truth spreadsheets. Audio temporal_smoothing.py prototypes chunk-level hysteresis during training; the live and offline fusion paths operate on 5-second intervals.
| Script | Purpose |
|---|---|
multimodal_attention_detector.py |
Offline single-video or batch-folder processing → Excel report; --live for webcam + mic with on-screen red overlay and PLEASE PAY ATTENTION console alert |
validate_predictions.py |
Compare generated Excel vs ground truth (accuracy, precision, recall, F1, confusion matrix) |
Hyperparameter Tuning.py |
One-pass score collection per video, then grid search over (w_video, w_audio, threshold) without re-running feature extraction |
By default, multimodal_attention_detector.py loads:
- Video:
Video_Classifier/results_attention/bilstm_attention.keras+inference_config.json - Audio:
Audio_Classifier/model_output/trained_model.joblib - MediaPipe:
Video_Classifier/face_landmarker.task
Override with --video-model-dir and --audio-model-dir if needed.
Full documentation: Combined/README_combined.md.
The Hardware_oled_code/ folder contains firmware for a LilyGO T-SIM7000G ESP32 driving a 0.96" SSD1306 OLED (I2C) and a piezo buzzer. When attention drops, the display shows a warning and the buzzer beeps in a timed on/off pattern.
This is a standalone physical alert channel — complementary to the software overlay in Combined/ live mode. Wire SDA/SCL to GPIO 21/22, buzzer signal to GPIO 25.
Full documentation: Hardware_oled_code/README.md.
| Split | Samples | Accuracy | Precision | Recall | F1 | Macro F1 | ROC-AUC |
|---|---|---|---|---|---|---|---|
| Train | 5667 | 0.674 | 0.701 | 0.630 | 0.664 | 0.673 | 0.739 |
| Val | 1720 | 0.612 | 0.450 | 0.446 | 0.448 | 0.574 | 0.585 |
| Test | 1723 | 0.616 | 0.649 | 0.491 | 0.559 | 0.609 | 0.650 |
Test confusion matrix (rows = actual, cols = predicted): [[641, 227], [435, 420]] — not attentive vs attentive.
Val metrics are weaker than train (expected with person-level DAiSEE splits and overlapping windows). Test macro-F1 ~0.61 shows the model learns beyond majority-class guessing at T_label = 2.5.
Metrics depend on collected team data and are written to Audio_Classifier/model_output/model_results.txt after train_model.py. Recall on the distracted class is the headline metric for this subsystem. Run python ravdness_verify.py to validate the feature extractor on RAVDESS (pipeline code check, not attention labels).
Run Combined/validate_predictions.py against team ground-truth Excel sheets to obtain fused accuracy, precision, recall, and F1. Use Hyperparameter Tuning.py to reproduce the weight/threshold search that produced w_v = 0.85, w_a = 0.15, and threshold 0.37.
pip install mediapipe opencv-python pandas numpy tqdm pyarrow \
tensorflow scikit-learn matplotlib- Python 3.10+
- Download
face_landmarker.taskand set paths in Step 1–4 scripts - DAiSEE and Human Faces on disk (paths in scripts are machine-specific — update before running)
pip install numpy pandas scipy scikit-learn matplotlib joblib \
librosa soundfile pydub praat-parselmouth openai-whisper \
pdfplumber pypdf reportlab- ffmpeg on PATH
- See
Audio_Classifier/README.mdfor RAVDESS layout and folder structure
pip install tensorflow opencv-python mediapipe librosa soundfile openpyxl \
scikit-learn pandas numpy joblib- ffmpeg on PATH (for audio track extraction from video)
- Conda environment
facemesh_envis recommended for MediaPipe compatibility (seeCombined/README_combined.md)
pip install torch torchvision facenet-pytorch dlib opencv-python \
numpy pandas scikit-learn tqdm Pillowcd Video_Classifier- Update
DATA_DIRS,LABELS_CSV, andMODEL_PATHinStep 4 - Complete Visual data.py. - Run Step 4 → produces
daisee_features_*.parquet. - Run
Step 5 - ML Model.py→ trains, evaluates, writesresults_attention/(log:console_output.log).
Optional: run Steps 1–2 on Human Faces to validate EAR/MAR/gaze; Step 3 if you need video features without head pose.
cd Audio_Classifier
python test_setup_helper.py setup # create person folders
# add recordings under audio/<Person>/
python test_setup_helper.py check # validate naming
python audioconverter.py
python feature_extractor.py
python train_model.py
python temporal_smoothing.py # optional
python generate_report.pyOr: python run_all_test.py for an integrated smoke test.
cd CombinedSingle video → Excel report:
python multimodal_attention_detector.py --video-path "path/to/video.mp4" --output-excel predictions.xlsxBatch folder → one Excel sheet:
python multimodal_attention_detector.py --videos-folder "videos/" --output-excel all_predictions.xlsxLive webcam + microphone:
python multimodal_attention_detector.py --liveValidate against ground truth:
python validate_predictions.py --generated predictions.xlsx --ground-truth ground_truth.xlsxOn Windows with Conda, use the facemesh_env interpreter as documented in Combined/README_combined.md.
Flash Hardware_oled_code/oled_pay_attention.ino to the LilyGO ESP32 via Arduino IDE (see Hardware_oled_code/README.md).
cd Audio_Classifier
python ravdness_verify.py # feature pipeline vs RAVDESSWhy geometric features + LSTM instead of CNNs for deployment?
Live inference only needs MediaPipe + OpenCV — no GPU-heavy four-model ensemble. Features are interpretable (blink, yawn, head turn) and align with driver-monitoring literature (EAR/MAR).
Why drop DAiSEE columns from model inputs?
At runtime there are no boredom/engagement annotations. Training on them would be label leakage; the composite formula is used only to construct Attention.
Why speaker-aware splits in audio?
Without grouping by person, the model memorises voices instead of attention behaviours.
Why filename-as-label for audio?
Recording protocol encodes category in the name; no separate annotation pass.
Why two visual stacks?
CNN ensemble explores transfer learning on face crops; BiLSTM stack is the path tied to real-time geometric features and DAiSEE temporal labels.
Why focal loss and macro-F1?
Class imbalance and overlapping windows make accuracy and AUC misleading; focal loss focuses on hard examples; macro-F1 penalises ignoring the minority class.
Why 0.85 / 0.15 fusion weights?
Video (head pose, gaze, blink/yawn) is the stronger and more reliable modality for attention; audio adds complementary distraction cues (talking, environmental noise) but has more blind spots. Weights were selected via grid search in Hyperparameter Tuning.py.
| Area | Limitation | Direction |
|---|---|---|
| Video val gap | Val F1 below train | More data, person-aware CV, calibration |
| Audio | Small participant pool (4 people) | Collect more; GroupKFold mitigates but does not fix generalisation |
| Audio blind spots | Brief talking while still facing screen; passive silence vs focused micro-vocalisations | Video head pose + gaze (Video_Classifier/ Steps 4–5) |
| DAiSEE labels | Clip-level scores applied to every frame in a window | Finer temporal labels or weak supervision |
| Paths | Hard-coded Windows paths in some scripts | Central config or environment variables |
| Deployment | BiLSTM + MediaPipe latency on CPU | Quantisation, frame skip, edge device profiling |
| Hardware ↔ software | OLED firmware is standalone; not yet serial-linked to Combined/ live mode |
UART/Bluetooth bridge from Python to ESP32 |
| CNN path | Heavy; not wired to fusion stream | Distillation or abandon in favour of geometric path |
| Document | Contents |
|---|---|
Video_Classifier/README_Visual.md |
Deep dive: Steps 1–5, BiLSTM v2, live deployment |
Audio_Classifier/README.md |
Full audio pipeline, scripts, troubleshooting |
Combined/README_combined.md |
Multimodal fusion, live webcam, validation, hyperparameter tuning |
Hardware_oled_code/README.md |
ESP32 OLED + buzzer wiring and flashing |
TBD/Facial feature extractor README.md |
CNN ensemble visual path (alternate) |
Last updated: complete StayTuned system — visual pipeline (Video_Classifier/), audio pipeline (Audio_Classifier/), multimodal fusion (Combined/), and hardware alerts (Hardware_oled_code/).