Add multimodal support for Gemini tool calls (Blob & Arrays)#57
Open
Add multimodal support for Gemini tool calls (Blob & Arrays)#57
Conversation
Contributor
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdded multimodal function-calling: functions returning blobs or blob arrays are converted into Gemini inline parts and emitted as Changes
Sequence DiagramsequenceDiagram
participant Client as Client
participant Handler as ToolCallHandler
participant Exec as FunctionExecutor
participant Conv as BlobConverter
participant API as APIClient
Client->>Handler: invoke function call (Gemini/OpenAI)
Handler->>Exec: execute target function
Exec-->>Handler: returns (string | object | blob | [blobs])
alt blob or [blobs]
Handler->>Conv: convert blob(s) to Gemini inline parts
Conv-->>Handler: return parts array
Handler->>API: send tool-role response with `parts` array
else JSON/object or string
Handler->>API: send jsonResponse payload
end
API-->>Client: deliver API response
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs). Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/code.gs (2)
490-513:⚠️ Potential issue | 🟡 MinorMixed arrays (blobs + non-blobs) silently lose binary data.
When
functionResponseis an array where only some elements are blob-like, theevery(isBlobLike)check on line 495 fails and the entire array falls into thejsonResponsepath (line 498). Blob objects don't JSON-serialize meaningfully (they'll become empty objects or lose their binary content).Consider either:
- Partitioning the array into blob vs. non-blob items and handling each group separately.
- Throwing or logging a warning for mixed arrays so callers know the blobs are dropped.
💡 Option 2: warn on mixed arrays
if (Array.isArray(functionResponse)) { - if (functionResponse.length > 0 && functionResponse.every(isBlobLike)) { - multimodalParts = functionResponse.map(blobToGeminiInlinePart); - } else { // non-blob arrays - jsonResponse = functionResponse; - } + const blobs = functionResponse.filter(isBlobLike); + const nonBlobs = functionResponse.filter(item => !isBlobLike(item)); + if (blobs.length > 0 && nonBlobs.length > 0) { + console.warn('[GenAIApp] - Mixed array (blobs + non-blobs) returned by function. Blobs will be sent as multimodal parts; non-blobs as JSON.'); + } + if (blobs.length > 0) { + multimodalParts = blobs.map(blobToGeminiInlinePart); + } + if (nonBlobs.length > 0) { + jsonResponse = nonBlobs; + }
483-487: 🧹 Nitpick | 🔵 TrivialLogging placement: log before the processing logic for easier debugging.
The verbose log at line 485 is placed after the function call (line 482) but before the response processing (lines 488+). This is fine for knowing the function was called, but consider also logging a summary of the response type (blob, array, string, etc.) after the classification logic to aid debugging multimodal flows.
🤖 Fix all issues with AI agents
In `@src/code.gs`:
- Around line 1779-1786: The helper createGeminiInlinePart currently accepts a
filename and sets inlineData.displayName, but displayName is not a supported
field so the parameter and mapping are unused; remove the filename parameter
from createGeminiInlinePart and delete the displayName property from the
returned inlineData object, and update any callers of createGeminiInlinePart to
stop passing a filename (or, if Gemini actually expects filename at a different
level, instead move that value out of inlineData to the correct parent field
where used); reference function name createGeminiInlinePart when making these
changes and ensure all call sites are adjusted accordingly.
- Around line 196-202: The addImage logic is constructing Gemini parts using
snake_case fields (inline_data, mime_type) which the REST API won't accept;
update addImage to call the existing createGeminiInlinePart helper instead of
building parts manually so the payload uses camelCase (inlineData, mimeType).
Locate addImage and replace its parts array construction with a call to
createGeminiInlinePart(fileInfo.mimeType, blobToBase64, fileInfo.fileName) (or
pass parameters in the helper's expected order), ensuring the parts property
uses the helper return value so all field names match the Gemini REST API.
…ion.gs Reuse simple PDF generation logic in blob-return test helpers
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/testFunctions.gs`:
- Around line 189-193: The receipt currently interpolates amount directly in
generateReceipt and can drop trailing zeros or show float artifacts; before
calling _createSimplePdf (and before embedding in the template array), format
the numeric amount to a deterministic two-decimal string (e.g., use
amount.toFixed(2) or Intl.NumberFormat with minimumFractionDigits:2) and then
interpolate that formatted string into the `Customer: ...`/`Amount: €...` lines
so receipts always show two decimal places; update generateReceipt to compute
formattedAmount and pass `Amount: €${formattedAmount}` to _createSimplePdf.
…ile-name-handling Document file-name limitation for function-returned blobs
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/code.gs (1)
1594-1625: 🧹 Nitpick | 🔵 TrivialOpenAI blob-output path is correct; minor log-placement inconsistency.
The OpenAI Responses API
function_call_output.outputaccepts either a string or an array ofResponseInputFile/ResponseInputImage/ResponseInputTextobjects, and the docs explicitly state that for functions returning files, an array of file objects can be passed. TheblobToResponseInputFileContent→ array approach is therefore valid.Minor nit: the Gemini handler logs
function ${functionName}() called by Geminibefore blob processing (line 1485), while the OpenAI handler logs the equivalent after blob processing (line 1624). Aligning the log position for consistency would make traces easier to correlate.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/code.gs` around lines 1594 - 1625, The verbose logging for OpenAI handler is placed after blob/object processing, causing inconsistent trace ordering with the Gemini handler; move the console.log(`[GenAIApp] - function ${functionName}() called by OpenAI.`) so it runs immediately when the function is invoked (i.e., before the block that inspects and converts functionResponse using isBlobLike and blobToResponseInputFileContent), keeping the same verbose conditional and message text to align log placement with the Gemini handler.
♻️ Duplicate comments (2)
src/code.gs (2)
1781-1799: New helpers are correctly implemented.
createGeminiInlinePartuses the correct camelCase field names (inlineData,mimeType) required by the Gemini REST API, anddisplayNameinsideinlineDatais a valid field — the Gemini docs confirm it must be unique when referenced via{"$ref": "..."}fromfunctionResponse.response. Both helpers look good.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/code.gs` around lines 1781 - 1799, Both helpers are correctly implemented: keep createGeminiInlinePart and blobToGeminiInlinePart as-is (they use inlineData, mimeType and displayName correctly and properly base64-encode blob bytes via Utilities.base64Encode(blob.getBytes())); no code changes required—leave the functions createGeminiInlinePart and blobToGeminiInlinePart unchanged and mark the change approved.
196-202:addFilenow correctly usescreateGeminiInlinePart(camelCase).The fix is correct. Note that
addImage(lines 119–124) still builds Gemini content withinline_data/mime_type(snake_case) rather than the same helper — that inconsistency was flagged in a prior review and remains unaddressed.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/code.gs` around lines 196 - 202, addImage currently constructs Gemini content using snake_case keys (inline_data, mime_type) instead of reusing the helper createGeminiInlinePart like addFile does; update the addImage implementation to call createGeminiInlinePart(fileInfo.mimeType, blobToBase64, fileInfo.fileName) (or equivalent arg order used by createGeminiInlinePart) and remove the manual inline_data/mime_type object so both addFile and addImage consistently use the same camelCase helper.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/code.gs`:
- Around line 1494-1499: The current Array branch treats mixed arrays as
"non-blob" and lets Blob objects be stringified to "{}", losing data; update the
Array handling around functionResponse so you detect mixed arrays (use
functionResponse.some(isBlobLike) && functionResponse.some(x => !isBlobLike(x)))
and then either throw an error or filter with a clear warning. If failing fast:
throw a descriptive Error indicating a mixed blob/non-blob array was received;
if filtering: call processLogger.warn (or equivalent) and build multimodalParts
= functionResponse.filter(isBlobLike).map(blobToGeminiInlinePart) and set
jsonResponse = functionResponse.filter(x => !isBlobLike(x)) so blobs are
preserved and non-blob elements are handled separately instead of being lost
during JSON.stringify.
- Around line 1530-1534: The code pushes function results into contents with
role: "tool", which is invalid for generateContent; change the pushed object to
use role: "user" and ensure the pushed parts (responseParts) are formatted as
function responses (e.g., parts with type "functionResponse") so the
generateContent API accepts them; update the block that constructs the contents
array where responseParts is used (the push that currently sets role: "tool") to
set role: "user" and include the appropriate functionResponse part metadata.
- Around line 1515-1526: The code unconditionally adds parts to the
functionResponse (responseParts.push -> functionResponse) when multimodalParts
exists, which breaks Gemini 2.5 compatibility; update the logic to check the
active model/version (e.g., your model identifier or config used when calling
the API) and only include the parts field for Gemini 3+ models (Gemini 3
series/Vertex AI) while leaving it out for Gemini 2.5; also when parts are
supported and jsonResponse is empty, ensure the JSON response emitted by
jsonResponse (used with functionName) includes proper references to attached
parts using the {"$ref":"<displayName>"} format so the model can resolve the
blobs (use multimodalParts and functionName/jsonResponse as the reference
points).
---
Outside diff comments:
In `@src/code.gs`:
- Around line 1594-1625: The verbose logging for OpenAI handler is placed after
blob/object processing, causing inconsistent trace ordering with the Gemini
handler; move the console.log(`[GenAIApp] - function ${functionName}() called by
OpenAI.`) so it runs immediately when the function is invoked (i.e., before the
block that inspects and converts functionResponse using isBlobLike and
blobToResponseInputFileContent), keeping the same verbose conditional and
message text to align log placement with the Gemini handler.
---
Duplicate comments:
In `@src/code.gs`:
- Around line 1781-1799: Both helpers are correctly implemented: keep
createGeminiInlinePart and blobToGeminiInlinePart as-is (they use inlineData,
mimeType and displayName correctly and properly base64-encode blob bytes via
Utilities.base64Encode(blob.getBytes())); no code changes required—leave the
functions createGeminiInlinePart and blobToGeminiInlinePart unchanged and mark
the change approved.
- Around line 196-202: addImage currently constructs Gemini content using
snake_case keys (inline_data, mime_type) instead of reusing the helper
createGeminiInlinePart like addFile does; update the addImage implementation to
call createGeminiInlinePart(fileInfo.mimeType, blobToBase64, fileInfo.fileName)
(or equivalent arg order used by createGeminiInlinePart) and remove the manual
inline_data/mime_type object so both addFile and addImage consistently use the
same camelCase helper.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
src/testFunctions.gs (1)
189-194:⚠️ Potential issue | 🟡 MinorFormat currency to fixed decimal places.
The
amountparameter is interpolated directly, which can produce floating-point artifacts or inconsistent decimal places in receipts (e.g.,42.5instead of42.50).💵 Suggested fix
function generateReceipt(customerName, amount) { + const formattedAmount = Number(amount).toFixed(2); return _createSimplePdf("receipt.pdf", "Receipt", [ `Customer: ${customerName}`, - `Amount: €${amount}` + `Amount: €${formattedAmount}` ]); }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/testFunctions.gs` around lines 189 - 194, The receipt prints raw numeric `amount` which can show floating artifacts or inconsistent decimals; update generateReceipt to format the amount to two decimal places (e.g., using amount.toFixed(2) or Intl.NumberFormat for Euro) before interpolating so the string passed to _createSimplePdf is like `Amount: €{formattedAmount}`; modify generateReceipt (and any callers if necessary) to compute formattedAmount and pass that into the array given to _createSimplePdf.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/testFunctions.gs`:
- Around line 160-165: The _escapeHtml helper currently escapes &, <, and > but
not quotes; update function _escapeHtml to also replace double quotes (") and
single quotes (') with " and ' respectively so it is safe for use in
attribute contexts; modify the replace chain on _escapeHtml (the
String(value).replace(...).replace(...)) to include .replace(/"/g,
""").replace(/'/g, "'") while preserving existing escapes and behavior.
---
Duplicate comments:
In `@src/testFunctions.gs`:
- Around line 189-194: The receipt prints raw numeric `amount` which can show
floating artifacts or inconsistent decimals; update generateReceipt to format
the amount to two decimal places (e.g., using amount.toFixed(2) or
Intl.NumberFormat for Euro) before interpolating so the string passed to
_createSimplePdf is like `Amount: €{formattedAmount}`; modify generateReceipt
(and any callers if necessary) to compute formattedAmount and pass that into the
array given to _createSimplePdf.
ℹ️ Review info
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
src/testFunctions.gs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.