title	Persuasive MAGIC
emoji	📈
colorFrom	purple
colorTo	pink
sdk	gradio
sdk_version	6.4.0
app_file	app.py
pinned	false
short_description	Demo for automatic grading and feedback

Persuasive MAGIC

Our repository expands the original MAGIC framework for automatic essay scoring by specializing the agents into persuasive writting through modified prompts and evaluation workflows.

Additionally, our work aims to simplify the framework so that non-technical people can use it, while also adding a local distillation path for smaller student models and direct comparisons against the original MAGIC setup.

If you wish to simply test the hosted UI, use this link: https://huggingface.co/spaces/samodelkini/MAGIC-persuade

The current published finetuned adapter is available here: https://huggingface.co/samodelkini/smollm2-360m-unsloth-essay-scoring-finetune

Folder structure

corpus: Contains the different datasets used for our experiments. This includes the GRE essay datasets, subsets of essays from the PERSUADE 2.0 corpus, helper conversion scripts, and the committed persuade_sample_1200.json subset used by the current distillation recipes.

feedback_analysis: Contains the analysis of the written feedback, such as Human Feedback Evaluation Results, Cohen k interannotation agreement, Jaccard similarity, and lexical diversity.

MAGIC_USER: Contains the main UI elements and base agentic architecture and prompts.

• app.py contains the main Gradio interface for the project. It supports the regular OpenRouter-backed grading flow, and also local professor-mode scoring with a base model plus optional adapter.
• process_essays.py allows you to grade multiple essays from a JSON file using the Persuasive MAGIC agent + orchestrator pipeline. It outputs structured JSON files and an Excel summary.
• process_essays_original_magic.py runs the same batch process, but patches the app to use the prompt set from prompts_original_magic/ so the original MAGIC framework can be used as a baseline.
• run_batch_original_magic.py is a convenience launcher for running selected models across the GRE and Persuade corpora with the Original MAGIC prompts.
• prompts: Folder with the main Persuasive MAGIC prompts.
  • agents.py: specialized agent prompts and rubrics.
  • orchestrator.py: orchestrator base prompt.
  • base.py: base one-shot prompt without multiple agents.
  • baseline.py: base one-shot approach.
  • judge.py: feedback analysis prompt for LLM-as-a-judge runs.
• prompts_original_magic: Folder with the original MAGIC prompt versions preserved for comparison runs.

distillation: Contains the teacher-student workflow for creating and evaluating smaller local grading models.

• prepare_sft_data.py builds supervised fine-tuning JSONL data from teacher result files.
• prepare_preference_data.py builds preference-pair datasets for DPO-style experiments.
• train_student.py fine-tunes a Hugging Face causal LM on the SFT data.
• train_student_unsloth.py fine-tunes the same task through the Unsloth path.
• evaluate_student.py evaluates a local student model with the Persuasive MAGIC prompts.
• evaluate_student_original_magic.py evaluates a local student model with the Original MAGIC prompts.
• evaluate_ollama.py evaluates a local Ollama model through the multi-agent pipeline.
• evaluate_ollama_original_magic.py does the same with the Original MAGIC prompts.
• run_batch_original_magic_local.py and run_batch_original_magic_ollama.py are convenience scripts for running Original MAGIC baseline evaluations locally and then calling numeric_eval.py.
• run_recipe.py contains predefined training recipes for the currently supported student-model experiments.
• common.py contains shared helpers used by the evaluation scripts.
• notebooks/ contains the kept Colab notebooks for the main SmolLM2 experiments.
• base_models/README.md explains how to recreate local-only base-model snapshots.
• SMOLLM2_360M_UNSLOTH_ESSAY_SCORING_FINETUNE.md contains notes for the SmolLM2 finetune experiment.

graphs: Contains scripts and generated outputs for visual comparisons across pipelines and models.

numeric_eval_details: Contains a more in-depth analysis of the scoring aspect of the architecture. This includes confusion matrices and per-score breakdowns for the top-level Persuasive MAGIC result files.

The folder called final refers to the results of the orchestrator when compared to the ground truth. The folder called median refers to the results of the median value of the agents when compared to the ground truth.

numeric_eval_details_original_magic: Contains the same type of evaluation details for runs made with the Original MAGIC prompt set, including the committed untuned and finetuned SmolLM2 baseline runs.

results: Contains the full feedback and final score outputs of the essays as .json and .xlsx files. This includes both the orchestrator feedback and the agent feedback for the Persuasive MAGIC pipeline, plus the committed untuned and finetuned SmolLM2 runs. Also includes a folder with the baseline outputs from running the same datasets against the Original MAGIC prompts. This includes result JSON/XLSX files and summary CSVs for the current comparison experiments.

deploy_to_spaces.py: Script used to deploy a minimal version of the application to a Hugging Face Space for testing purposes.

DEPLOYMENT.md: Step-by-step guide for deploying only the app-facing files to a Hugging Face Space.

DISTILLATION.md: Main documentation for the local teacher-student workflow, including data preparation, training, evaluation, and adapter usage.

numeric_eval.py: Script for evaluating result files and producing summary tables with QWK, RMSE, accuracy, adjacent agreement, bias, confidence intervals, and optional detailed breakdown folders.

numeric_eval_summary.csv: Contains the current summary table for the Persuasive MAGIC runs.

requirements.txt: Main dependency list for the app, evaluation, and graph generation.

requirements-distillation.txt: Extra dependencies for the local distillation workflow.

requirements-unsloth.txt: Extra dependencies for the Unsloth training path.

Use of scripts

deploy_to_spaces.py

deploy_to_spaces.py allows you to run the UI version of Persuasive MAGIC in a Hugging Face Space in your own account while only uploading the files needed for the app.

If you want deployment instructions in detail, see DEPLOYMENT.md.

MAGIC_USER/process_essays.py

This script processes a corpus through the Persuasive MAGIC pipeline.

Example:

python MAGIC_USER/process_essays.py corpus/GRE/GRE_essays.json \
  --output results/GRE_olmo_results.json \
  --workers 12 \
  --model allenai/olmo-3.1-32b-instruct

MAGIC_USER/process_essays_original_magic.py

This script does the same thing as process_essays.py, but swaps in the Original MAGIC prompt set.

Example:

python MAGIC_USER/process_essays_original_magic.py corpus/GRE/GRE_essays.json \
  --output results/original_magic/GRE_original_magic.json \
  --workers 12 \
  --model allenai/olmo-3.1-32b-instruct

numeric_eval.py

numeric_eval.py evaluates Automated Essay Scoring outputs stored in JSON files and compares predicted scores against human ratings.

The script computes:

• Quadratic Weighted Kappa (QWK)
• Root Mean Squared Error (RMSE)
• Exact agreement
• Adjacent agreement (±1)
• Mean prediction bias
• Confidence intervals

Run on a single file:

python numeric_eval.py results/gemma_GRE_results.json

Run on multiple files:

python numeric_eval.py results/*.json

distillation/run_recipe.py

distillation/run_recipe.py provides a shorter way to launch the predefined student-model training recipes without manually retyping the long command lines.

Example:

python distillation/run_recipe.py smollm2-360m

For the full training and evaluation workflow, see DISTILLATION.md.

graphs/create_comparison_figures.py

This script rebuilds the current model-comparison figures and the accompanying summary outputs stored in graphs/comparison_figures/.

Example:

python graphs/create_comparison_figures.py

Distillation notebooks

The following notebooks are made for running the distillation scripts on Google Colab:

• distillation/notebooks/smollm2_360m_distillation_colab.ipynb
• distillation/notebooks/smollm2_360m_unsloth_finetune_colab.ipynb
• distillation/notebooks/smollm2_360m_full_eval_colab.ipynb

These notebooks clone the repo, install dependencies, run the recipe, and perform a small evaluation pass.

Notes

• Large local model snapshots under distillation/base_models/ and training outputs under distillation/checkpoints/ are intended for local use and should not be committed unless there is a specific reason.
• The tracked PNG, CSV, Markdown, TeX, and JSON files under graphs/comparison_figures/, results/, and results/original_magic/ are experiment artifacts, not source code.
• The Gradio certificate file was intentionally removed from version control and should stay untracked.