TopoKD — Robust & Dynamic Symbolic Knowledge Distillation

Undergraduate Research Project (UGP), Jun 2025 – May 2026 Adiba Khan (230061) · Dept. of Chemistry, IIT Kanpur Mentor: Prof. Rajesh M. Hegde · MIPS Lab, Dept. of Electrical Engineering, IIT Kanpur

---

What this is

Symbolic Knowledge Distillation (West et al., NAACL 2022) trains a compact student on a corpus generated once by a large teacher. The student studies a fixed textbook and has no way to ask for help when it is confused.

This project asks whether letting the student re-consult the teacher only when it is uncertain is a better use of the same teacher budget — and then actually tests that claim instead of asserting it.

Three components:

1. ADCFT (Adaptive Dynamic Confusion-Triggered Fine-Tuning) — an online loop where the student scores its own predictive uncertainty and queries the teacher only above an adaptive threshold.
2. Geometry-aware distillation — aligning teacher/student representational structure (Gram matrix over the batch) rather than matching raw activations or logits alone.
3. Edge deployment — INT8 quantization and CPU inference benchmarking of the distilled student.

Results

All numbers are measured. Full tables and caveats in results/.

ADCFT ablation — matched teacher budget and matched gradient steps

GPT-2 (124M) student ← FLAN-T5-Large (780M) teacher · 100 ATOMIC seed events × 7 relations · 3 seeds

arm	held-out NLL ↓	ROUGE-L ↑
untrained student	5.013	0.068
Static (SKD baseline)	4.054 ± 0.634	0.093
Random-online (rate-matched control)	3.614 ± 0.197	0.082
ADCFT (confusion-gated)	3.783 ± 0.142	0.102

Confusion gating won on ROUGE-L but not on held-out likelihood, where a rate-matched random control was equal or better. Seed variance (σ up to 0.63) exceeds the between-arm gaps, so this is reported as inconclusive, not a win. Consultation rate 1.2% → 20.6% moves NLL 4.52 → 3.18, confirming the gate works as a budget control.

Geometry-aware distillation — CIFAR-100

DeiT-Small teacher (81.57%) → DeiT-Tiny student · 20k-image subset · 3 epochs · L = CE + α·KD + β·Gram

arm	α	β	top-1
CE only	–	–	74.67%
KD	1.0	0	71.91%
KD	0.5	0	73.99%
KD + Gram	1.0	1	74.89%
KD + Gram	0.5	1	75.29%

Gram alignment at fixed α: +2.98 pts (α=1.0) and +1.30 pts (α=0.5). Logit KD alone never beat plain fine-tuning — the gain comes from the geometric term. Best student: 3.95× fewer parameters (21.7M → 5.5M) at 92.3% of teacher top-1. Swin-Tiny baseline: 81.71%.

INT8 quantization — CPU inference

variant	size	tok/s	NLL	ROUGE-L
FP32	497.8 MB	33.1	3.347	0.1026
INT8, all Linear	281.7 MB	54.5	8.193 (+145%)	0.077
INT8, lm_head excluded	243.1 MB (2.05×)	39.8 (1.20×)	3.436 (+2.7%)	0.090 (87.8%)

Quantizing GPT-2's output projection both degrades quality catastrophically and makes the model larger — lm_head shares its weight tensor with the token embedding, so quantizing it breaks the tie and stores the 50257×768 matrix twice.

Two silent failures worth documenting

Neither raised an error, and in both cases the loss kept going down:

1. Teacher prompt format. The reference template was written for a completion-style GPT-3 teacher. Against an instruction-tuned FLAN-T5 it produced fluent 16-word rambles where 4-word answers belonged. Every metric looked plausible; the outputs were useless. Fixed by rewriting to Event/Question/Answer: form with beam-4 decoding — median reference length 16 → 4 words, reference quality 100%.
2. Weight-tied quantization. See above. The tell was the size increase, not the accuracy drop.

A third: the original fixed confusion threshold (τ=3.5) never gated anything — the entire uncertainty distribution spans 1.7 nats, so a 5% vs 10% query rate differs by τ=0.04. Replaced with an adaptive running-percentile gate.

Layout

decks/       Part 1 (symbolic KD) and Part 2 (experiments & results) presentations
report/      final report (submitted document: describes the proposed framework;
             measured outcomes live in results/)
notebooks/   P1 ADCFT ablation notebook (Kaggle-ready)
experiments/ teacher-prompt fix, percentile-gated run, diagnostics, analysis
results/     full result tables with caveats and non-claims
references/  West et al. (2022), Symbolic Knowledge Distillation, NAACL

Reproducing

notebooks/P1_ADCFT_ablation.ipynb runs on Kaggle (GPU + Internet ON), ~90 min, and checkpoints every stage to /kaggle/working so a disconnect never costs more than the current stage. Then apply experiments/p1_teacher_prompt_fix.py and experiments/p1_percentile_gated_run.py, and read results with experiments/p1_analysis.py (standalone — reads saved JSON, survives kernel restarts).

Scope — what is not implemented

Designed and specified in the report, but not evaluated: Procrustes alignment, 2nd-order/Hessian matching, the cross-modal VLM loop, ImageNet-scale training, and Jetson/Raspberry-Pi deployment. CIFAR-100 results use a 20k subset at a 3-epoch budget with a single seed per configuration.

References

• West, P. et al. (2022). Symbolic Knowledge Distillation: from General Language Models to Commonsense Models. NAACL.
• Williams, A. H. et al. (2021). Generalized Shape Metrics on Neural Representations. NeurIPS.
• Hinton, G., Vinyals, O., Dean, J. (2015). Distilling the Knowledge in a Neural Network. NIPS DL Workshop.
• Touvron, H. et al. (2021). Training data-efficient image transformers (DeiT). ICML.
• Liu, Z. et al. (2021). Swin Transformer. ICCV.