Add single-token fingerprint complementarity analysis

paper/appendix.tex

@@ -441,6 +441,10 @@ \section{Knowledge Fingerprinting: Detailed Tables}
 
 \paragraph{When the three metrics disagree.} The three metrics are complementary, not redundant. Heavy post-training can preserve rare-fact knowledge (high $J$ and high lift) while rewriting the generation style enough that a model gives \emph{differently worded} wrong answers on hard probes---driving $\mathrm{HSS}$ down. Nemotron-70B vs.\ Llama-3.1-70B is the textbook example: $J = 0.81$ and lift = 6.8 (clear shared base) but $\mathrm{HSS} = 0.08$ (SFT has scrambled the surface form of wrong answers). Conversely, a high-$\mathrm{HSS}$ pair with low $J$ and low lift is a weaker signal than it sounds, because it may reflect a single over-represented hallucination (e.g., a common name collision) rather than broad knowledge overlap. We classify a pair as \emph{shared base} only when all three metrics point the same way; one-sided signals are reported as ``possible lineage'' and should be investigated case-by-case.
 
+\paragraph{Comparison with single-token output distributions.} \citet{bruckner2026onetoken} release a pairwise Jensen--Shannon distance matrix built from repeated one-token answers to trivial prompts. We join that matrix to IKP by exact served-model identifier and exclude IKP thinking variants, whose post-reasoning answer channel is not comparable to the direct one-token channel used in their protocol. The resulting 82-model intersection contains all 3,321 unordered pairs. Table~\ref{tab:single-token-complementarity} reports descriptive Spearman correlations between their behavioral similarity ($-\mathrm{JSD}$) and our knowledge-fingerprint metrics. Overall rank association is weak for correct-set Jaccard ($\rho=0.282$) and nearly absent for HSS ($\rho=0.074$); the relationship is stronger within vendors but remains far from redundant. Cross-vendor HSS has $\rho=0.039$, supporting the use of shared wrong facts and low-entropy answer priors as complementary provenance evidence, without showing that they arise from separate causal sources. Because pairwise observations share models, ordinary significance tests for unrelated pairs do not apply; the different collection windows and serving paths add further uncertainty. We therefore treat the comparison as hypothesis-generating rather than a significance test. A future fused-verification experiment should enroll trusted reference endpoints, evaluate held-out model families, and compare knowledge-only, output-prior-only, and combined decision rules under a fixed query budget.
+
+\input{../results/tables/single_token_complementarity.tex}
+
 \paragraph{Release-practice summary.} Applied to consecutive-generation pairs across all tracked families (see Table~\ref{tab:fp-all-families}), three release patterns recur:
 \begin{itemize}[leftmargin=*]
     \item \emph{Fine-tune-on-same-base releases} (lineage): Claude Opus 4 $\to$ 4.1, Sonnet 4 $\to$ 4.5, DeepSeek V3 $\to$ V3.1 $\to$ V3.2, GLM 4.6 $\to$ 4.7, Kimi K2 $\to$ K2.5 $\to$ K2.6. These preserve a large fraction of the base's rare-fact mistakes.
@@ -649,4 +653,3 @@ \section{Ethical Considerations}
 This work estimates proprietary model sizes from public API access, which raises considerations for competitive intelligence and intellectual property. We note that: (1)~parameter count is only one dimension of model capability, and our estimates are effective capacity (not literal architecture); (2)~the method provides ${\sim}3.2\times$ precision, insufficient for reverse-engineering proprietary designs; and (3)~similar estimates are already publicly available from inference-economics analyses~\citep{epochai2024}. Knowledge fingerprinting for cross-model lineage analysis could be used to enforce licensing terms (a legitimate use) or to make unfounded IP claims (a misuse). We encourage responsible use of these tools for transparency and governance rather than competitive surveillance.
 
 All information presented in this paper is drawn from publicly available sources. The author has no internal access to, confidential briefings about, or non-public collaboration with any frontier LLM vendor; in particular, the parameter and effective-capacity estimates for proprietary models are produced \emph{entirely} from external behavioral measurements and the open-weight calibration set.
-

paper/main.tex

@@ -97,7 +97,7 @@ \subsection{Long-Tail Knowledge and Scaling}
 
 \subsection{Existing Model Size Estimation}
 
-Epoch AI's inference economics approach~\citep{epochai2024} estimates parameter counts from token throughput and API pricing, with acknowledged $2\times$+ uncertainty. \citet{gao2025modelequality} formalize API model verification as a two-sample testing problem, finding that 11 of 31 Llama API endpoints serve distributions different from Meta's reference weights with just ${\sim}10$ samples per prompt. LLMmap~\citep{llmmap2025} fingerprints identify \emph{which} model is behind an API ($>95\%$ accuracy across 42 versions) but do not estimate size of unknown models. More recent fingerprinting work includes RoFL~\citep{tsai2025rofl}, which identifies models from statistical output patterns without model modification, and \citet{nasery2025fingerprinting}, who embed $24{,}576$ fingerprints into a single model that persist after fine-tuning. Both are complementary to IKP: they identify \emph{which} model is served but do not estimate size of unknown models. \citet{cai2025substitution} show that text-output statistical tests fail at ${\sim}50\%$ for detecting quantization, motivating knowledge-based approaches. The Densing Law~\citep{densing2025} demonstrates why standard benchmarks~\citep{hendrycks2021measuring, joshi2017triviaqa, kwiatkowski2019natural, liang2023holistic} fail as size proxies: capability density doubles every ${\sim}3.5$ months, making a 7B model from 2026 match a 70B model from 2023 on reasoning tasks.
+Epoch AI's inference economics approach~\citep{epochai2024} estimates parameter counts from token throughput and API pricing, with acknowledged $2\times$+ uncertainty. \citet{gao2025modelequality} formalize API model verification as a two-sample testing problem, finding that 11 of 31 Llama API endpoints serve distributions different from Meta's reference weights with just ${\sim}10$ samples per prompt. LLMmap~\citep{llmmap2025} fingerprints identify \emph{which} model is behind an API ($>95\%$ accuracy across 42 versions) but do not estimate size of unknown models. More recent fingerprinting work includes RoFL~\citep{tsai2025rofl}, which identifies models from statistical output patterns without model modification, and \citet{nasery2025fingerprinting}, who embed $24{,}576$ fingerprints into a single model that persist after fine-tuning. \citet{bruckner2026onetoken} fingerprint 165 served models from repeated one-token answers to trivial prompts (random numbers, colors, words, and related tasks in four languages); Jensen--Shannon divergence over these answer distributions yields a model-verification AUC of $0.971$ and equal-error rate of $7.3\%$ with the full 40-cell battery. These methods are complementary to IKP: they identify \emph{which} model is served but do not estimate size of unknown models. \citet{cai2025substitution} show that text-output statistical tests fail at ${\sim}50\%$ for detecting quantization, motivating knowledge-based approaches. The Densing Law~\citep{densing2025} demonstrates why standard benchmarks~\citep{hendrycks2021measuring, joshi2017triviaqa, kwiatkowski2019natural, liang2023holistic} fail as size proxies: capability density doubles every ${\sim}3.5$ months, making a 7B model from 2026 match a 70B model from 2023 on reasoning tasks.
 
 \subsection{Parameter Specialization}
 
@@ -413,7 +413,7 @@ \subsection{Knowledge Fingerprinting: Lineage vs.\ Retraining}
     \item \textbf{Zhipu GLM.} GLM 4.5 $\to$ 4.6 (retrained, $0.04$), 4.6 $\to$ 4.7 (lineage, $0.21$), 4.7 $\to$ 5 (retrained, $0.01$) alternate, consistent with a pattern of minor point releases interleaved with full retrains.
 \end{itemize}
 
-\paragraph{Cross-family outliers.} Applying the same test to the ${\sim}14{,}000$ cross-vendor pairs flags a small number as $\mathrm{HSS} \geq 0.20$ with $\geq 10$ joint-wrong probes---the regime occupied by shared weights within a vendor. The strongest signal (across multiple tests) is Baidu ERNIE 4.5, which achieves $\mathrm{HSS} = 0.36$--$0.50$ against GPT-4o, Llama-3-70B, Mistral and Qwen-Max simultaneously---a pattern consistent with heavy training on mixed distilled outputs rather than a single teacher. Llama 3.1 70B shows up as an apparent ``teacher'' in a striking number of pairings ($\mathrm{HSS} \geq 0.30$ against grok-3, gemini-2.0-flash, qwen3-max, GPT-4.1-nano, and several other models), which is most likely an artifact of Llama 3.1 being the most widely used open base for synthetic-data generation in 2024. GPT-5 vs.\ Grok-4 ($\mathrm{HSS} = 0.38$, 21 joint-wrong) and GPT-5-pro vs.\ Kimi-K2.6 ($\mathrm{HSS} = 0.32$, 22 joint-wrong) are individually above threshold and merit follow-up, though neither is large enough, at our probe count, to reject an innocent-explanation null with confidence. Full rankings and raw counts are in Appendix~\ref{app:fingerprint}.
+\paragraph{Cross-family outliers.} Applying the same test to the ${\sim}14{,}000$ cross-vendor pairs flags a small number as $\mathrm{HSS} \geq 0.20$ with $\geq 10$ joint-wrong probes---the regime occupied by shared weights within a vendor. The strongest signal (across multiple tests) is Baidu ERNIE 4.5, which achieves $\mathrm{HSS} = 0.36$--$0.50$ against GPT-4o, Llama-3-70B, Mistral and Qwen-Max simultaneously---a pattern consistent with heavy training on mixed distilled outputs rather than a single teacher. Llama 3.1 70B shows up as an apparent ``teacher'' in a striking number of pairings ($\mathrm{HSS} \geq 0.30$ against grok-3, gemini-2.0-flash, qwen3-max, GPT-4.1-nano, and several other models), which is most likely an artifact of Llama 3.1 being the most widely used open base for synthetic-data generation in 2024. GPT-5 vs.\ Grok-4 ($\mathrm{HSS} = 0.38$, 21 joint-wrong) and GPT-5-pro vs.\ Kimi-K2.6 ($\mathrm{HSS} = 0.32$, 22 joint-wrong) are individually above threshold and merit follow-up, though neither is large enough, at our probe count, to reject an innocent-explanation null with confidence. Full rankings and raw counts are in Appendix~\ref{app:fingerprint}. As a redundancy check, aligning the released single-token distance matrix of \citet{bruckner2026onetoken} to our non-thinking runs (82 models, 3,321 pairs) shows weak rank association with our knowledge metrics ($\rho=0.282$ for Jaccard, $\rho=0.074$ for HSS), supporting their use as complementary evidence; the full retrospective comparison is in Appendix~\ref{app:fingerprint}.
 
 \subsection{What Determines Whether an LLM Knows Something?}
 \label{sec:recognition}

paper/references.bib

@@ -508,6 +508,15 @@ @article{tsai2025rofl
   year={2025}
 }
 
+@article{bruckner2026onetoken,
+  title={One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions},
+  author={Bruckner, Tom{\'a}{\v{s}}},
+  journal={arXiv preprint arXiv:2607.10252},
+  year={2026},
+  doi={10.48550/arXiv.2607.10252},
+  url={https://arxiv.org/abs/2607.10252}
+}
+
 @article{nasery2025fingerprinting,
   title={Scalable Fingerprinting of Large Language Models},
   author={Nasery, Anshul and Hayase, Jonathan and Brooks, Creston and Sheng, Peiyao and Tyagi, Himanshu and Viswanath, Pramod and Oh, Sewoong},

results/single_token_complementarity.json

@@ -0,0 +1,45 @@
+{
+  "n_models": 82,
+  "n_pairs": 3321,
+  "min_joint_wrong_for_hss": 10,
+  "correlations": {
+    "all": {
+      "jaccard": {
+        "n": 3321,
+        "spearman_rho": 0.282
+      },
+      "hss": {
+        "n": 2598,
+        "spearman_rho": 0.074
+      }
+    },
+    "same_vendor": {
+      "jaccard": {
+        "n": 276,
+        "spearman_rho": 0.374
+      },
+      "hss": {
+        "n": 219,
+        "spearman_rho": 0.261
+      }
+    },
+    "cross_vendor": {
+      "jaccard": {
+        "n": 3045,
+        "spearman_rho": 0.256
+      },
+      "hss": {
+        "n": 2379,
+        "spearman_rho": 0.039
+      }
+    }
+  },
+  "source": {
+    "single_token_dataset_doi": "10.5281/zenodo.21278557",
+    "single_token_matrix_member": "results/divergence-matrix.csv",
+    "single_token_matrix_sha256": "0eb6821716d6420c814285db73d4edacb6c7104b576079be2500cbcda21d76a5",
+    "ikp_fingerprint_results": "results/comprehensive_fingerprint_results.json",
+    "ikp_fingerprint_results_sha256": "2837a6f9bdc35b79adfc805bbe5ba4d342a732eb351fd3bd54049d23d433372d",
+    "alignment": "exact served model identifier; non-thinking IKP runs only"
+  }
+}

results/tables/single_token_complementarity.tex

@@ -0,0 +1,19 @@
+% Generated by scripts/20_single_token_complementarity.py.
+\begin{table}[H]
+    \centering
+    \small
+    \begin{tabular}{llrr}
+        \toprule
+        Pair subset & IKP metric & $n$ pairs & Spearman $\rho$ \\
+        \midrule
+All pairs & Jaccard & 3321 & 0.282 \\
+All pairs & HSS & 2598 & 0.074 \\
+Same vendor & Jaccard & 276 & 0.374 \\
+Same vendor & HSS & 219 & 0.261 \\
+Cross vendor & Jaccard & 3045 & 0.256 \\
+Cross vendor & HSS & 2379 & 0.039 \\
+        \bottomrule
+    \end{tabular}
+    \caption{Descriptive rank association between single-token behavioral similarity ($-\mathrm{JSD}$) and IKP knowledge-fingerprint similarity. HSS rows require at least ten probes on which both models are wrong. Pairwise observations share models, so ordinary significance tests for unrelated pairs do not apply; the correlations quantify signal overlap only.}
+    \label{tab:single-token-complementarity}
+\end{table}

scripts/20_single_token_complementarity.py

@@ -0,0 +1,108 @@
+#!/usr/bin/env python3
+"""Align IKP knowledge fingerprints with Bruckner's single-token JSD matrix.
+
+Download ``pamela-publish-data.zip`` from DOI 10.5281/zenodo.21278557,
+then run:
+
+    python3 scripts/20_single_token_complementarity.py \
+        --single-token-artifact /path/to/pamela-publish-data.zip
+
+The external dataset is read in place and is not copied into this repository.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import sys
+import zipfile
+from pathlib import Path
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+from src.single_token_complementarity import (
+    PINNED_SINGLE_TOKEN_MATRIX_SHA256,
+    align_fingerprint_pairs,
+    read_verified_distance_matrix,
+    render_latex_table,
+    summarize_alignment,
+)
+
+
+DEFAULT_JSON = PROJECT_ROOT / "results" / "single_token_complementarity.json"
+DEFAULT_TABLE = PROJECT_ROOT / "results" / "tables" / "single_token_complementarity.tex"
+DISTANCE_MEMBER = "results/divergence-matrix.csv"
+SOURCE_DOI = "10.5281/zenodo.21278557"
+
+
+def load_model_ids(results_dir: Path) -> dict[str, str]:
+    """Map IKP result-file stems to exact served model identifiers."""
+    model_ids = {}
+    for path in sorted(results_dir.glob("*.json")):
+        if "think" in path.stem.lower():
+            continue
+        try:
+            payload = json.loads(path.read_text())
+        except (json.JSONDecodeError, OSError):
+            continue
+        if isinstance(payload, dict) and isinstance(payload.get("model_id"), str):
+            model_ids[path.stem] = payload["model_id"]
+    return model_ids
+
+
+def load_external_distances(path: Path):
+    """Return the exact, digest-verified source matrix and its SHA-256."""
+    if path.suffix.lower() == ".zip":
+        with zipfile.ZipFile(path) as archive:
+            raw_bytes = archive.read(DISTANCE_MEMBER)
+    else:
+        raw_bytes = path.read_bytes()
+    return read_verified_distance_matrix(
+        raw_bytes,
+        expected_sha256=PINNED_SINGLE_TOKEN_MATRIX_SHA256,
+    )
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--single-token-artifact",
+        type=Path,
+        required=True,
+        help="Zenodo data ZIP or extracted results/divergence-matrix.csv",
+    )
+    parser.add_argument("--output-json", type=Path, default=DEFAULT_JSON)
+    parser.add_argument("--output-table", type=Path, default=DEFAULT_TABLE)
+    args = parser.parse_args()
+
+    knowledge_path = PROJECT_ROOT / "results" / "comprehensive_fingerprint_results.json"
+    knowledge_bytes = knowledge_path.read_bytes()
+    knowledge_pairs = json.loads(knowledge_bytes)["all_pairs"]
+    knowledge_sha256 = hashlib.sha256(knowledge_bytes).hexdigest()
+    model_ids = load_model_ids(PROJECT_ROOT / "data" / "results")
+    distances, distance_sha256 = load_external_distances(args.single_token_artifact)
+    aligned = align_fingerprint_pairs(knowledge_pairs, model_ids, distances)
+    summary = summarize_alignment(aligned)
+    summary["source"] = {
+        "single_token_dataset_doi": SOURCE_DOI,
+        "single_token_matrix_member": DISTANCE_MEMBER,
+        "single_token_matrix_sha256": distance_sha256,
+        "ikp_fingerprint_results": str(knowledge_path.relative_to(PROJECT_ROOT)),
+        "ikp_fingerprint_results_sha256": knowledge_sha256,
+        "alignment": "exact served model identifier; non-thinking IKP runs only",
+    }
+
+    args.output_json.parent.mkdir(parents=True, exist_ok=True)
+    args.output_table.parent.mkdir(parents=True, exist_ok=True)
+    args.output_json.write_text(json.dumps(summary, indent=2) + "\n")
+    args.output_table.write_text(render_latex_table(summary))
+
+    print(f"Aligned {summary['n_models']} models across {summary['n_pairs']} pairs")
+    print(f"Wrote {args.output_json}")
+    print(f"Wrote {args.output_table}")
+
+
+if __name__ == "__main__":
+    main()

scripts/README.md

@@ -42,6 +42,7 @@ already ships in `data/`.
 | `loo_cv_analysis.py` | Leave-one-out CV + writes Fig 7 (`paper/figures/fig7_loo_validation.pdf`). |
 | `17_adversarial_robustness.py` | Quantifies how cheaply a black-box operator can game its IKP estimate (sandbagging vs. contamination) across the full roster; writes `data/results/adversarial_ikp.json` + `paper/figures/adversarial_ikp.png`. Findings in `../ADVERSARIAL_IKP.md`. |
 | `18_v2_validation.py` | Validates IKP v2: reproduces the repo's own accuracy (0 diff), refits the paper's calibration, checks the refusal-interval behavior. Writes `data/results/ikp_v2_validation.json` + `paper/figures/ikp_v2_intervals.png`. |
+| `20_single_token_complementarity.py` | Joins IKP knowledge fingerprints to Bruckner's released single-token JSD matrix by exact model ID; writes `results/single_token_complementarity.json` and the appendix table. Requires the Zenodo artifact from DOI `10.5281/zenodo.21278557` and verifies the source matrix against its pinned SHA-256. |
 | `show_progress.py` | Quick text progress dump across `data/results/`. |
 
 ## legacy/