Skip to content

fix: guard against empty/filtered LLM responses in legacy LLM models#457

Open
qizwiz wants to merge 3 commits into
GreyDGL:mainfrom
qizwiz:fix/llm-response-unguarded
Open

fix: guard against empty/filtered LLM responses in legacy LLM models#457
qizwiz wants to merge 3 commits into
GreyDGL:mainfrom
qizwiz:fix/llm-response-unguarded

Conversation

@qizwiz

@qizwiz qizwiz commented May 18, 2026

Copy link
Copy Markdown

What

Adds null checks before accessing choices[0].message in five files under legacy/pentestgpt/:

  • llm_generation/models/open_ai.py
  • llm_generation/models/deepseek.py
  • llm_generation/models/jina.py
  • utils/APIs/chatgpt_api.py
  • utils/APIs/deepseek_api.py

Why

Two silent failure modes crash at these lines:

  1. IndexErrorchoices is an empty list ([]) when the provider returns no completions
  2. AttributeErrorchoices[0].message is None when content is filtered (some providers return HTTP 200 with PROHIBITED_CONTENT finish reason)

Fix

if not response.choices or response.choices[0].message is None:
    raise ValueError("LLM returned empty or filtered response")
content = response.choices[0].message.content

Detected by pact static analysis.

Add null checks before accessing choices[0].message across 5 files
in legacy/pentestgpt/: open_ai.py, deepseek.py, jina.py, chatgpt_api.py,
deepseek_api.py.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds validation logic to ensure LLM responses are not empty or filtered before processing them across several files. However, the implementation in chatgpt_api.py and deepseek_api.py contains critical syntax errors because if and raise statements were placed directly inside class constructor arguments. Additionally, these checks are positioned after the response data is already accessed, which would cause a crash before the validation logic is executed. These checks should be moved to the start of the methods.

Comment on lines +147 to +148
if not response.choices or response.choices[0].message is None:
raise ValueError("LLM returned empty or filtered response")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

This code introduces a SyntaxError because an if statement and raise cannot be placed inside a class constructor call (InitialGeneration(...)).

Furthermore, this check is performed too late for the langfuse logging block. If response.choices is empty, the code will crash at line 149 when attempting to access response.choices[0].message.content before the check at line 156 is ever reached.

Recommendation: Move the validation check to the beginning of the _chat_completion method (or at least before line 140) to ensure the response is valid before any of its attributes are accessed.

Comment on lines +150 to +151
if not response.choices or response.choices[0].message is None:
raise ValueError("LLM returned empty or filtered response")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

This code introduces a SyntaxError because an if statement and raise cannot be placed inside a class constructor call (InitialGeneration(...)).

Furthermore, this check is performed too late for the langfuse logging block. If response.choices is empty, the code will crash at line 152 when attempting to access response.choices[0].message.content before the check at line 159 is ever reached.

Recommendation: Move the validation check to the beginning of the _chat_completion method (or at least before line 143) to ensure the response is valid before any of its attributes are accessed.

The guard was mistakenly placed inside the InitialGeneration constructor
argument list, which is invalid Python syntax. Move it before the
langfuse logging block so it runs before the constructor call.

This also removes the now-redundant second guard after the block since
a single guard at the top of the response-handling section is sufficient.
@qizwiz

qizwiz commented May 18, 2026

Copy link
Copy Markdown
Author

Thanks @gemini-code-assist — you're right, the guard was incorrectly placed inside the InitialGeneration(...) constructor argument list in both chatgpt_api.py and deepseek_api.py, which is invalid Python syntax.

Fixed in commit 88d021a: moved the guard before the langfuse block in both files, so it runs as a standalone statement before any constructor is called. The now-redundant second guard after the langfuse block was removed. Syntax verified with ast.parse().

@gemini-code-assist

Copy link
Copy Markdown

Thanks for the update, @qizwiz. Moving the guard before the langfuse block is the correct approach to ensure the response is validated before any downstream processing or logging occurs. This effectively prevents potential AttributeError or IndexError exceptions and keeps the logic clean. The fix looks solid.

Extend the empty-response guard to cover the case where message exists
but content is None (e.g., tool-call responses or filtered content).
Guards message.content before it is passed to langfuse InitialGeneration
and before the return value.
@qizwiz

qizwiz commented May 18, 2026

Copy link
Copy Markdown
Author

Thanks for the review! A quick clarification on the placement concern:

The guard at lines 138–142 is outside the InitialGeneration(...) constructor call — it's an if statement at function body scope that executes before the langfuse block begins. If choices is empty or message is None, ValueError is raised immediately and lines 146–158 are never reached. Both files pass ast.parse cleanly (no SyntaxError).

That said, your point about message.content being None is valid — I've just pushed an update that extends the guard to also cover message.content is None, which can occur for tool-call responses or filtered content. The langfuse completion= parameter will now never receive None.

@qizwiz

qizwiz commented May 18, 2026

Copy link
Copy Markdown
Author

The syntax issue was fixed in 88d021a (guard moved before the langfuse block). The current HEAD (d274d06) passes ast.parse cleanly on both files — the guard is at function-body scope, not inside any constructor argument. Happy to address any remaining concerns once the review reflects the current diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants