Test MCP implementation with different LLMs and improve accordingly

[esgn]

Context

This is a task we will repeat at different stages

• Some models will not have access to MCP resources
• Some small models will struggle with complex behavior
• Some environments implement only part of the MCP standard or with specifics
• MCP tools description must be tested with different models and different environments

Therefore we must test and improve accordingly.

Actions

• Prepare work in a dedicated repository : ignfab/geocontext-test
• Allow the creation of a langgraph agent for test purpose
• Create the first tests relying on the agent
• Integrate ignfab/geocontext-test in geocontext (npm run test:e2e)
• Archive ignfab/geocontext-test
• Add missing dependency for google models ( npm install --save-dev @langchain/google-genai )
• Define a list of model to test (see bellow, in progress)
• Run npm run test:e2e for each model and produce a report

Models

• anthropic:claude-haiku-4-5
• anthropic:claude-sonnet-4-6
• google-genai:gemini-2.5-flash (WARNING : google_genai with an underscore in python)
• google-genai:gemini-3.1-flash-lite-preview
• ollama:qwen3.5:9b : too small to chain tool calls
• ... TO BE COMPLETED at least with some mistral model ...

Linked issues

• study different models to find a better/cheaper one demo-geocontext#49

[mborne]

With main (v0.9.8) :

• claude-haiku-4-5 : 7/7 OK

export MODEL_NAME="anthropic:claude-haiku-4-5"
npm run test:e2e

• claude-sonnet-4-6 : 6/7 OK (skip invocation of gpf_wfs_describe_type before gpf_wfs_get_features)

export MODEL_NAME="anthropic:claude-sonnet-4-6"
npm run test:e2e

AssertionError: Scenario "should find about 1 741 batiments in the commune of Saint Mandé": expected tool "gpf_wfs_describe_type" to have been called.: expected false to be true // Object.is equality

• gemini-3.1-flash-lite-preview : 1/7 OK - exclusiveMinimum not supported in tool schema? (@esgn by removing .positive() on dwithin_distance_m, 7/7 OK)

export MODEL_NAME="google-genai:gemini-3.1-flash-lite-preview"
npm run test:e2e

Error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent: [400 Bad Request] Invalid JSON payload received. Unknown name "exclusiveMinimum" at 'tools[0].function_declarations[7].parameters.properties[15].value': Cannot find field. [{"@type":"type.googleapis.com/google.rpc.BadRequest","fieldViolations":[{"field":"tools[0].function_declarations[7].parameters.properties[15].value","description":"Invalid JSON payload received. Unknown name "exclusiveMinimum" at 'tools[0].function_declarations[7].parameters.properties[15].value': Cannot find field."}]}]