Skip to content

New models and refactor around proc_data.py#15

Merged
QuarticCat merged 43 commits into
mainfrom
cloud-dev
Jun 2, 2026
Merged

New models and refactor around proc_data.py#15
QuarticCat merged 43 commits into
mainfrom
cloud-dev

Conversation

@Cloud0310
Copy link
Copy Markdown
Collaborator

@Cloud0310 Cloud0310 commented May 5, 2026

Summary

This PR refactors the Python data pipeline and adds configurable MobileNet model training.

Data Pipeline

  • Split the previous proc_data.py implementation into packaged modules under python/detypify.
  • Store raw LaTeX-labelled datasets separately, then map labels to Typst symbols locally with datasets caching.
  • Generate frontend metadata under build/generated.
  • Update the weekly Typst metadata workflow to compare effective mapping digests before regenerating metadata.

Training And Models

  • Replace the fixed model enum with mobilenet_{v4|v5}_{size} model names.
  • Add MobileNetV5 support as experimental.
  • Add --compile/--no-compile for torch.compile.
  • Add CPU/CUDA/RoCM dependency extras for training environments.
  • Remove TensorRT dependencies and options.

Frontend And CI

  • Update frontend service metadata imports to use generated metadata.
  • Add frontend dependency security updates.
  • Move Ruff configuration into ruff.toml.

Notes

  • MobileNetV5 support is experimental and needs training validation.
  • Training should use explicit accelerator extras, for example uv run --extra cuda python/train.py.

Due to the size of the tensorrt lib and the optimization of current
model is already speedy enough, we remove the dep for better
installation experience.
Replace hardcoded ModelName enum with regex-based parser supporting
mobilenet_{v4|v5}_{size} naming convention. Refactor TimmModel to
MobileNetModel using create_project_model factory with V4 support.
Add V5 model path to create_project_model using timm's mobilenetv5_base
with configurable channel multiplier. Update default training models to
include mobilenet_v5_010 and mobilenet_v5_005.
@Cloud0310 Cloud0310 requested a review from QuarticCat May 5, 2026 09:18
@Cloud0310
Copy link
Copy Markdown
Collaborator Author

Cloud0310 commented May 5, 2026

@QuarticCat Please try new model training, offer me the test set result image and training log. All should be under dir build.

Here's the some things needs your notification when training:
When training, add --no-ema cli option, as I want just the raw exp data.
Models to be trained: v4_035, v5_010, v_005. (the number means percentage of full size model. here, is 35% of small_v4 variant, 10% of full size v5).

@QuarticCat QuarticCat merged commit 08c2249 into main Jun 2, 2026
6 checks passed
@QuarticCat QuarticCat deleted the cloud-dev branch June 2, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants