Skip to content

Make the whole IT suite runnable in bulk #1925

Description

@azanux

Problem

Running the ITs in one go needs a hand-maintained exclusion:

mvn -Dtest='*IT,!LLMOllama*IT' -Dsurefire.failIfNoSpecifiedTests=false test

19 of the 38 *IT classes guard themselves with @EnabledIfEnvironmentVariable.
The other 19 fail on any machine missing what they assume. !LLMOllama*IT is a
manual workaround for two of them.

Needs a credential or a live service (16) - guard these:

Variable Classes
ANTHROPIC_API_KEY (no IT guards on it today) LLMAnthropicGuardRailsIntegrationIT, LLMAnthropicStreamingBuilderIT, LLMAnthropicThinkingIT, LlmAnthropicCachingIT, Opus48IntegrationIT
OLLAMA_BASE_URL LLMOllamaThinkingIT, LLMOllamaStreamingBuilderIT
OPENAI_API_KEY LLMOpenAiCostTrackingIT, LLMOpenAiGuardRailsIntegrationIT, LLMOpenAiStreamingBuilderIT, ParallelToolLoopGuardRailIT, AgentProcessCostAggregationRealLlmIT, LLMStreamingIT, ChatClientLlmOperationsIT, AgentPlatformAutoConfigurationIT
AWS credentials AgentBedrockAutoConfigurationIT

Needs network, no credential (1) - OnnxEmbeddingServiceIT downloads
all-MiniLM-L6-v2 from HuggingFace. Guard it, but say in disabledReason that
it is a slow download, not a missing key.

Needs nothing external (2) - MistralAiHttpClientTimeoutIT runs against a
local StubMistralServer; QuiteMcpClientAutoConfigurationIT is an
ApplicationContextRunner test over mocks. Both already pass anywhere. Leave
them unguarded: they must run in the sweep. MistralAiHttpClientTimeoutIT is
named ...IT only because of its deliberate ~12s wait - worth a javadoc line so
nobody adds a guard later.

Separately: surefire's default includes don't match *IT, and there is no
failsafe plugin and no integration profile in the tree. So ITs never run in CI's
mvn test verify, and failIfNoSpecifiedTests=false is needed because most
modules match no IT at all.

Proposal

  1. Guard the 17 above, following LangChain4jStreamingToolLoopIT. Missing
    credentials must mean skipped, never failed. (One existing guard reads
    named = "GEMINI_API_KEY" with disabledReason = "... requires GOOGLE_API_KEY" - fix that in passing.)
  2. The exclusion then disappears:
    mvn -Dtest='*IT' -Dsurefire.failIfNoSpecifiedTests=false test
  3. Wrap it in a profile (mvn -Pintegration-tests test) setting surefire
    <includes>, and document the expected variables. The ITs already live in
    src/test, so failsafe would only add the verify lifecycle for nothing.

Acceptance

  • The command exits 0 with no credentials and no local Ollama: the 17 skip, the
    2 self-contained ITs run and pass.
  • With OPENAI_API_KEY set, the OpenAI ITs run and the rest still skip.
  • No IT needs to be excluded by name to keep the sweep green.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions