EatSmart Glasses

An iOS companion app for Meta Ray-Ban Gen 2 glasses that delivers real-time food intelligence through voice. Point your glasses at a meal or pantry, and EatSmart speaks personalized nutritional and bioactivity insights directly through the glasses — no screen required.

Built on the Meta Wearables Device Access Toolkit (DAT SDK) and powered by the SIS (Smart Ingredient System) backend.

Features

• Pantry scan — capture a photo through the glasses, receive spoken nutritional and bioactivity analysis
• Voice Q&A — ask follow-up questions about what you're looking at; answers are spoken back through the glasses
• Ingredient matching — identifies individual ingredients and retrieves molecular bioactivity data (anti-inflammatory, antioxidant, NRF score, etc.)
• AI recipe generation — generates a recipe from detected ingredients and walks through cooking steps via audio
• Cooking flow — step-by-step audio guidance with pacing controls
• Health goal personalization — tailor recommendations to Cardiovascular, Anti-inflammatory, Blood sugar, Antioxidant support, or General wellness goals

Requirements

• iOS 17.0+
• Xcode 15.0+
• Swift 5.9+
• Meta Wearables Device Access Toolkit (included)
• Meta Ray-Ban Gen 2 glasses (required for full audio pipeline; mock device available for development)

Setup

See SETUP.md for full build and signing instructions.

Quick start:

1. Clone this repository
2. Open CameraAccess.xcodeproj in Xcode
3. Select your iPhone as the target device
4. Build and run (Cmd+R)
5. Enable Developer Mode on your glasses: Meta AI app → Settings → your glasses → Developer Mode ON

Note: Developer Mode resets after firmware updates — re-enable it after any glasses update.

Architecture

Meta Ray-Ban Gen 2 (camera + speakers)
        │  DAT SDK (MWDATCamera)
        ▼
  iOS App (Swift/SwiftUI)
  ├── StreamSessionViewModel   — glasses connection, photo capture, voice recording
  ├── SISService               — backend API client (multipart/form-data)
  └── Views                   — CameraStreamView, IngredientResultsView, CookingFlowView
        │  HTTPS
        ▼
  SIS Backend (Google Cloud Run)
  └── /api/sis/*              — dish scan, ingredient match, recipe generation, bioactivity data

Backend

Base URL: https://sis-621577110242.us-west1.run.app

Key endpoints used by this app:

Endpoint	Purpose
POST /api/sis/dish-scan/details	Nutritional analysis of a food photo
POST /api/sis/dish-scan/question	Voice Q&A (photo + audio → spoken answer)
POST /api/sis/ai/ingredient-photo-match	Ingredient identification + bioactivity data
POST /api/sis/ai/recipe-generate	AI recipe generation from ingredients
GET /api/sis/dishes/{id}	Full dish record including bioactivity scores

Changelog

See CHANGELOG.md.

Credits

Developed by

• Pranav Gupta
• Haoyu Yan

Funded by

• AI Institute for Next Generation Food Systems (AIFS)
• Meta

License

This source code is licensed under the license found in the LICENSE file in the root directory of this source tree.