[Evaluation, not a bug] Needle 2 as a tool-call router, 4-phase experiment

[etchevarneM]

Evaluation report: Needle 2 as a tool-call router (4-phase experiment: retrieval, fine-tuning, argument accuracy, confidence gating)

Sharing results from evaluating Needle 2 as a candidate to replace an existing tool-routing layer in a small local voice-assistant project. Not a bug report (we independently isolated the same bug down to a 2-example toy dataset before finding it had already been reported and fixed as #53, thanks for the quick turnaround). Just a data point that might be useful for others sizing up Needle for a similar use case, and a question for the maintainers on whether our results match expectations.

Setup

• Catalogue: 16 tools across 6 categories (1-5 tools per category), declared with realistic natural-language descriptions and typed parameters via @tool.
• Prompts and tool descriptions in a non-English language (Spanish). Flagging this since Needle 2's training data mix may skew English-heavy; it's a real confound we can't fully separate from raw model capability.
• Semantic overlap: several tools within the same category share similar verbs/shapes (e.g. multiple "create an X" actions with overlapping parameter names, multiple mail-lifecycle actions like search, read, draft, send that take similar-looking string arguments). Probably close to a worst case for a 45M-param retrieval+extraction model, and likely a big part of the gap below.
• Baseline comparison: an existing production 2-stage router (lightweight LLM classification into a category, then a ~30B local LLM constrained to that category's tools), measured at 3/3 and 3/3 on the two test scenarios below.
• Used only Needle.complete() throughout (decision only); tool execution is always gated by application code, never by the model itself, so we never called .run().

Phase 1: baseline accuracy (no fine-tuning)

Scenario	Tool selection	Argument accuracy (on correct-tool picks)
Full 16-tool catalogue, single target tool, 3 phrasings	2/3	0/3 (arguments consistently disconnected from the query, e.g. a "task description" field kept getting filled with an unrelated noun from the prompt rather than the actual task)
5-tool category subset, 5 distinct queries	1/5	1/1 (only correct pick had correct args)

Phase 2: fine-tuning

~300 hand-templated examples (not LLM-generated) across the 4 busiest categories, positive + negative + ambiguous examples, query/tools/answers/reasoning format. Hit the NaN-at-step-2 bug here, isolated it down to a 2-example toy dataset before finding the issue was already reported. Installed from main post-fix and training completed cleanly (final loss 1.40, LoRA rank 16, defaults otherwise, 3 epochs).

Phase 3: post-fine-tuning re-test (same two scenarios)

Scenario	Tool selection	Argument accuracy (hand-audited, not substring-matched)
Full 16-tool catalogue	2/3 (unchanged)	0/3 (unchanged)
5-tool category subset	3/5 (up from 1/5)	2/5 fully correct end-to-end (one pick had the right tool + one right field but a fabricated date the query never provided)

Headline finding: fine-tuning on a small clean dataset measurably improved retrieval (which tool) but did not proportionally improve argument grounding (correct field values). Invented/hallucinated arguments persisted even on some correct tool picks, post-fine-tune.

Phase 4: confidence-gated hybrid (fall back to the existing router below a threshold)

Using the 8 combined test cases with Needle's own reported confidence score:

Threshold	% handled by Needle alone	Precision on handled cases	Hybrid avg. latency*
0.001	87.5%	28.6%	0.95s
0.05	50.0%	25.0%	3.01s
0.1	37.5%	33.3%	3.70s
0.2	25.0%	50.0%	4.39s
0.35	12.5%	100.0%	5.08s
0.4	0%	N/A	5.76s (about equal to baseline)

* baseline (existing router alone) is about 5.5s avg per turn in our production logs; hybrid latency = Needle's own decision time on handled turns, plus Needle's probe time and baseline latency on fallback turns.

The only threshold with perfect precision only covers 1 of our 8 cases (n=8 is tiny, please don't read the 100% as a real number, it's a single data point), for an ~8% average latency improvement in the best case. Below that threshold, precision degrades quickly (down to 20-50%), which we judged too risky for an assistant that sometimes proposes irreversible actions.

Takeaway (for us)

Not adopting Needle for this router in its current form. The combination of "argument grounding doesn't fine-tune away as easily as retrieval does" and "confidence doesn't cleanly separate good/bad decisions at usable coverage" was the deciding factor, not raw speed (which is excellent: decisions consistently landed in 0.1-0.8s regardless of correctness).

Genuinely curious whether the argument-grounding gap is a known characteristic at this parameter count / with this training approach, or something we could address differently (larger fine-tuning set? different confidence calibration? separating "which tool" from "which arguments" into two passes?). Happy to share more methodology detail if useful.

Related: #53 (NaN during fine-tuning, independently reproduced during this evaluation).

[HenryNdubuaku]

Thanks @etchevarneM, this is the most useful external eval we’ve had, and the results match a few known gaps:

1. Spanish: tokenization is ~1.7× worse than English, and training/confidence calibration are English-heavy. We’ve seen correct Spanish calls score 0.0 confidence, so non-English confidence gating isn’t reliable yet. LoRA also doesn’t recalibrate the confidence head.

2. Grounding: your two-pass approach works today. First pick the tool, then rerun with only that tool declared. That’s essentially how extract() works.

3. Fine-tuning: 300 examples is likely too small. needle generate-data --augment your.jsonl can expand the set in Spanish; we’d expect a few thousand examples to move grounding more.

If you can share the methodology or a redacted eval slice, we’d love to use it as a regression benchmark.

[etchevarneM]

Thanks for the detailed pointers, we ran both.

Confidence calibration (Spanish): confirmed a genuine correct call scoring suspiciously low (0.0047, matches what you described), but found a structural ceiling beyond the calibration issue: a false-positive call already scores higher confidence (0.32) than that genuine correct one (0.0047) in our data. Since recalibration is a monotonic transform, it can't invert that ordering, the theoretical best-case coverage ceiling we calculated is ~25% (2/8), and that's unverified without an actually recalibrated model. Might be useful context if you're looking at confidence-head training for non-English data.

Two-pass grounding: implemented as described (pick tool, then re-run with only that tool declared). Real but modest improvement on our 5-tool-category scenario (2/5 → 3/5 correct), offset by two new regressions we hadn't seen before, a hallucinated URL argument on one call, and fabricated field values on another where the single-pass version had at least failed by picking the wrong tool (visible failure) rather than inventing plausible-looking wrong arguments (silent failure). No change on our full-catalogue scenario (0/3 → 0/3).

Net result for us: not adopting for this router, the confidence ceiling plus the argument-grounding gap don't clear the bar for an assistant that sometimes proposes irreversible actions. But this was a genuinely useful exchange, and happy to share the redacted eval slice if it's still useful as a regression benchmark, let us know what format works best for you.

[HenryNdubuaku]

Thanks, we just noticed that the confidence in the engine had a bug which has now been fixed, so the next patch releasing in about 20hrs, ships this improvements. You can immediately use this fix with pip install --force-reinstall git+https://github.com/cactus-compute/needle.git

[TeaShaman-cyber]

Thanks for sharing this evaluation — we have a downstream result that looks complementary to the routing/grounding split discussed here, with one important causal caveat discovered after our first post.

We ran a reproducibility-focused experiment on pinned cactus-needle==2.0.8 with 360 synthetic/public training rows (300 route-positive + 60 explicit route-negative answers: [] cases), 15 epochs, batch 16, LR 1e-4, LoRA rank 16 / alpha 32, and zero truncation. Two separately executed equal-configuration jobs produced the same deployed-artifact heldout behavior:

Metric	Base	Tuned .cact
Positive correctness	21/72	32/72
Positive route calls	49/72	72/72
Negative NO_CALL	24/24	0/24
Dominant semantic decision share	0.632653	0.888889

So the end-to-end 2.0.8 pipeline produced an applicability regression even while positive routing improved. Our preregistered disposition was REJECTED_APPLICABILITY_REGRESSION.

Correction / confounder: after posting, we found #91. The maintainer notes that cactus-needle==2.0.12 fixes both the fine-tuning/native-inference template mismatch and deployment-numerics mismatch via QAT. Our Stage B scientific evaluation used the built .cact artifact and did not include a separate float-LoRA control before needle build. Therefore this result is authoritative for the 2.0.8 finetune -> build -> deployed artifact, but it should not be attributed uniquely to the SFT objective and should not be generalized to current main without retraining.

The evaluation distinction still seems useful:

positive routing / semantic correctness != applicability specificity / NO_CALL preservation

Full corrected report with exact config, workflow, commit, hashes, and the #91 causal boundary:
https://github.com/TeaShaman-cyber/theseus-needle-lab/blob/bdfa9d76a5cc6b63c5bb1d1825e3e690a67f74d2/docs/downstream-findings/2026-09-04-needle-stage-b-applicability-regression.md

We are now preparing a current-main/2.0.12 canary that separates float adapter behavior from the built artifact before drawing any stronger upstream conclusion.

— Semyon Poklad & Шут (Jester), Theseus research project