Algebraic Pruning: Experimental W²-Norm Pruning Heuristic

Research hypothesis: low normalized W² norm may identify projection matrices that respond differently to a specific SVD-pruning experiment.

Experimental setup

The repository contains GPT-2 pruning experiments using a project-defined structural diagnostic and SVD pruning procedure. For square weight matrices, the diagnostic normalizes W and records the norm of W²; this is a descriptive signal, not a proof of mathematical nilpotency.

The strongest committed reproducibility artifact is:

• benchmarks/publication-evidence/2026-08-19-v38/ — GPT-2 Medium / Tiny Shakespeare evaluation with fixed token blocks, block-level losses, condition summaries, per-layer pruning details, environment metadata, source/model/dataset provenance, and restoration checks.

Under that exact V38 setup, attention c_proj matrices tolerated the tested 25% and 50% singular-value pruning conditions substantially better than MLP c_fc matrices. The artifact does not establish that the W² diagnostic causes that difference, does not classify rectangular c_fc matrices as non-nilpotent, and does not publish the historical multiplicative resilience ratio.

Hypotheses under test

• whether low normalized W² norm is associated with greater pruning tolerance for the tested square projection matrices;
• whether the observed behavior persists across checkpoints, datasets, pruning methods, seeds, and evaluation corpora;
• whether spectral summaries are stable enough across layers to be useful as features.

These are empirical questions. The current experiments do not establish a universal algebraic law or a causal pruning criterion.

Interpretation

The repository is useful as a compact experiment harness for testing algebraic/spectral signals against pruning behavior. A stronger conclusion would require broader evaluation, baselines against established pruning methods, repeated trials across models/datasets, and preserved raw results.

Zeroing singular values also does not by itself reduce the stored parameter count; an actual compression/deployment claim requires a representation that realizes the sparsity or low-rank structure.

Usage

# Inspect square-matrix W² diagnostics
python weight_algebra.py

# Pruning experiment on GPT-2 Small
python algebraic_pruning.py

# Pruning experiment on GPT-2 Medium
python algebraic_pruning_medium.py

Requirements

• PyTorch 2.0+
• transformers
• NVIDIA GPU for the recorded GPU workflow

How It Works

1. W²-norm diagnostic: for each eligible square weight matrix W, compute a normalized ||W²|| summary. Treat it as an experimental feature, not a nilpotency proof.
2. SVD pruning: decompose W = U S V^T and zero the smallest configured fraction of singular values.
3. Perplexity/loss check: measure model behavior on a fixed evaluation corpus and preserve raw block-level results.
4. Comparison: compare the tested projection groups under the same evaluation blocks and pruning fractions.

Citation

@software{algebraic-pruning,
  title={Algebraic Pruning: Experimental W2-Norm Pruning Heuristic},
  author={Salih Can Kurnaz},
  year={2026},
  url={https://github.com/salihcankurnaz/algebraic-pruning}
}

License

MIT. See LICENSE.