Is your feature request related to a problem? Please describe.
Currently, the LlmInsightTypeRouter is active during the query phase, which relies on LLM to process and route insights. While this provides high accuracy, it introduces significant latency and increases the cost of API calls. For scenarios where absolute precision via LLM is not strictly required, this results in slower extraction speeds and unnecessary resource consumption.
Describe the solution you'd like
I would like to propose adding a configuration switch (e.g., an environment variable like ENABLE_LLM_ROUTER=false or a specific configuration property) that allows users to disable the LlmInsightTypeRouter.
When this switch is turned off:
The system should bypass the LLM routing logic during the query phase.
It should fall back to a faster, rule-based, or direct extraction method.
This will significantly accelerate the extraction speed for users who prioritize performance over LLM-driven routing.
Describe alternatives you've considered
Optimizing the LLM prompt: Tried to make the prompt shorter to reduce latency, but it still cannot match the speed of a non-LLM approach.
Using a smaller/faster LLM model: Considered switching to a lighter model for routing, but it still adds network latency and cost compared to completely bypassing the LLM.
Additional context
This feature would be highly beneficial for high-throughput scenarios or local development/testing environments where users want to quickly extract insights without waiting for LLM responses. It gives users more flexibility to balance between accuracy and performance based on their specific use cases.
Is your feature request related to a problem? Please describe.
Currently, the LlmInsightTypeRouter is active during the query phase, which relies on LLM to process and route insights. While this provides high accuracy, it introduces significant latency and increases the cost of API calls. For scenarios where absolute precision via LLM is not strictly required, this results in slower extraction speeds and unnecessary resource consumption.
Describe the solution you'd like
I would like to propose adding a configuration switch (e.g., an environment variable like ENABLE_LLM_ROUTER=false or a specific configuration property) that allows users to disable the LlmInsightTypeRouter.
When this switch is turned off:
The system should bypass the LLM routing logic during the query phase.
It should fall back to a faster, rule-based, or direct extraction method.
This will significantly accelerate the extraction speed for users who prioritize performance over LLM-driven routing.
Describe alternatives you've considered
Optimizing the LLM prompt: Tried to make the prompt shorter to reduce latency, but it still cannot match the speed of a non-LLM approach.
Using a smaller/faster LLM model: Considered switching to a lighter model for routing, but it still adds network latency and cost compared to completely bypassing the LLM.
Additional context
This feature would be highly beneficial for high-throughput scenarios or local development/testing environments where users want to quickly extract insights without waiting for LLM responses. It gives users more flexibility to balance between accuracy and performance based on their specific use cases.