Skip to content

fix: agent hooks regarding tool confirmation #969

Description

@mkoruszowic

Problem

The built-in create_confirmation_hook() from ragbits.agents.hooks.confirmation does not work reliably across conversation turns.

When the hook sets decision="ask", the HookManager.execute_pre_tool() method generates a confirmation_id by hashing hook_name:tool_name:json.dumps(arguments, sort_keys=True). Within a single agent turn this
works fine the hash is computed once and matched against itself. But in a human-in-the-loop scenario, where the agent pauses, yields a ConfirmationRequest, and resumes on the next turn after the user confirms, the
LLM regenerates the tool call from scratch. The new arguments are semantically identical but textually different (reordered nested keys, whitespace changes, float 1.0 vs 1, rephrased free-text fields), so the
freshly computed hash no longer matches the confirmation_id the user approved.

This causes either:

  1. Infinite confirmation loop - the agent keeps asking for confirmation on the same action
  2. Abrupt termination - the loop detection breaks the agent run, silently dropping the user's intent

Root Cause

confirmation_id generation in HookManager.execute_pre_tool() is too sensitive to argument text:

confirmation_id_str = (
    f"{hook_name}:{tool_name}:{json.dumps(current_tool_call.arguments, sort_keys=True)}"
)                                                                                                                                                                                                                        
confirmation_id = hashlib.sha256(confirmation_id_str.encode()).hexdigest()[:16]

sort_keys=True stabilizes top-level key order but does not account for:

  • Nested object key ordering differences
  • Whitespace or formatting differences in string argument values
  • The LLM rephrasing free-text fields while preserving semantic intent
  • Numeric representation drift (1.0 vs 1, 100 vs 1e2)

Context

The internal-rag project hit this issue with Slack message confirmations and worked around it by replacing the built-in confirmation hook entirely with a custom PRE_TOOL hook that checks whether the user has
already accepted any confirmation in the current turn (has_accepted flag), bypassing the confirmation_id matching in HookManager altogether. The confirmation_id generation and matching logic in ragbits was
left unchanged because the workaround just avoids it.

Expected Behavior

Once a user confirms a tool execution, the agent should execute that tool without re-requesting confirmation, even if the LLM regenerates the call with cosmetically different arguments.

Proposed Approach

  1. Persist the original arguments from the ConfirmationRequest into ChatContext.state["pending_confirmation"] so they survive the turn boundary.
  2. On the continuation turn, when tool_confirmations contains confirmed=True and a pending confirmation exists, normalize the user message into an explicit continuation prompt that tells the LLM to call the
    tool with the exact original arguments avoiding argument regeneration entirely.
  3. Scope confirmation acceptance to tool name + turn as a fallback, so that if the LLM still produces different arguments, the prior approval for the same tool is honored within the same turn.
  4. Keep the existing confirmation_id mechanism for cases where multiple calls to the same tool in one turn need independent confirmation.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status
In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions