Skip to content

Commit d54baf9

Browse files
AbhiPrasadAbhiPrasad
committed
fix(google_genai): align usage metadata metrics
Include tool-use prompts and reasoning in normalized token totals, map audio and image modality usage, and preserve provider-specific cache and tool breakdowns.\n\nAdd sanitized VCR coverage for audio input/output, generated images, cached content, and streaming usage metadata.
1 parent f41c859 commit d54baf9

7 files changed

Lines changed: 757 additions & 61 deletions

File tree

‎.agents/skills/sdk-integrations/SKILL.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,8 @@ Only emit spec-listed keys:
202202
- `prompt_audio_tokens`, `completion_audio_tokens`, `completion_image_tokens` — audio/image models when provider reports
203203
- `start`, `end` — standard span timing
204204

205+
Map provider modality breakdowns only to standardized metrics: input audio → `prompt_audio_tokens`, output audio → `completion_audio_tokens`, and output image → `completion_image_tokens`. These are subsets of prompt/completion usage, not additional tokens. Preserve unstandardized modality breakdowns as deliberately named provider metadata rather than inventing metric keys.
206+
205207
For streaming, still produce one span per API call with accumulated `input`/`output`. Capture usage from stream metadata (e.g. OpenAI `stream_options.include_usage`) when surfaced.
206208

207209
For reasoning models, capture the full output structure (reasoning summaries + message blocks) and include prior reasoning in `input` for multi-turn calls.
@@ -261,7 +263,7 @@ Assert on emitted spans (not just provider return values):
261263

262264
For streaming, assert both the provider iterator/async-iterator still works AND the final span has aggregated `output` + stream-specific `metrics`.
263265

264-
Cassettes live in `integrations/<provider>/cassettes/<version>/` (e.g. `cassettes/latest/`, `cassettes/0.48.0/`). Nox sets `BRAINTRUST_TEST_PACKAGE_VERSION` so cassettes land correctly. Do not add per-test `vcr_cassette_dir` / `cassette_library_dir` fixtures — `integrations/conftest.py` handles it. Re-record only when behavior intentionally changed. Sanitize cassettes when the provider returns binary bodies.
266+
Cassettes live in `integrations/<provider>/cassettes/<version>/` (e.g. `cassettes/latest/`, `cassettes/0.48.0/`). Nox sets `BRAINTRUST_TEST_PACKAGE_VERSION` so cassettes land correctly. Do not add per-test `vcr_cassette_dir` / `cassette_library_dir` fixtures — `integrations/conftest.py` handles it. Re-record only when behavior intentionally changed. Sanitize binary media in both request and response bodies so checked-in cassettes do not retain large base64 payloads.
265267

266268
Confirm the exact session name from `noxfile.py` — don't assume it matches the folder.
267269

‎py/src/braintrust/integrations/google_genai/cassettes/latest/test_audio_input_usage.yaml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
interactions:
2+
- request:
3+
body: '{"contents": [{"parts": [{"inlineData": {"data": "UklGRiQAAABXQVZFZm10IBAAAAABAAEAgD4AAAB9AAACABAAZGF0YQAAAAA=",
4+
"mimeType": "audio/wav"}}, {"text": "Describe this audio in five words or less."}],
5+
"role": "user"}], "generationConfig": {"maxOutputTokens": 100}}'
6+
headers:
7+
accept:
8+
- '*/*'
9+
accept-encoding:
10+
- gzip, deflate
11+
connection:
12+
- keep-alive
13+
content-length:
14+
- '42928'
15+
content-type:
16+
- application/json
17+
host:
18+
- generativelanguage.googleapis.com
19+
user-agent:
20+
- google-genai-sdk/2.14.0 gl-python/3.14.6
21+
x-goog-api-client:
22+
- google-genai-sdk/2.14.0 gl-python/3.14.6
23+
method: POST
24+
uri: https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent
25+
response:
26+
body:
27+
string: "{\n \"candidates\": [\n {\n \"content\": {\n \"parts\":
28+
[\n {\n \"text\": \"Relaxing acoustic guitar music.\"\n
29+
\ }\n ],\n \"role\": \"model\"\n },\n \"finishReason\":
30+
\"STOP\",\n \"index\": 0\n }\n ],\n \"usageMetadata\": {\n \"promptTokenCount\":
31+
43,\n \"candidatesTokenCount\": 6,\n \"totalTokenCount\": 49,\n \"promptTokensDetails\":
32+
[\n {\n \"modality\": \"TEXT\",\n \"tokenCount\": 10\n
33+
\ },\n {\n \"modality\": \"AUDIO\",\n \"tokenCount\":
34+
33\n }\n ],\n \"serviceTier\": \"standard\"\n },\n \"modelVersion\":
35+
\"gemini-2.5-flash-lite\",\n \"responseId\": \"N1NqapHyDayQ-8YPkLLHgAo\"\n}\n"
36+
headers:
37+
alt-svc:
38+
- h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
39+
content-length:
40+
- '649'
41+
content-type:
42+
- application/json; charset=UTF-8
43+
date:
44+
- Wed, 29 Jul 2026 19:23:36 GMT
45+
server:
46+
- scaffolding on HTTPServer2
47+
server-timing:
48+
- gfet4t7; dur=1155
49+
transfer-encoding:
50+
- chunked
51+
vary:
52+
- Origin
53+
- X-Origin
54+
- Referer
55+
x-content-type-options:
56+
- nosniff
57+
x-frame-options:
58+
- SAMEORIGIN
59+
x-gemini-service-tier:
60+
- standard
61+
x-xss-protection:
62+
- '0'
63+
status:
64+
code: 200
65+
message: OK
66+
version: 1

‎py/src/braintrust/integrations/google_genai/cassettes/latest/test_cached_content_usage_metadata.yaml‎

Lines changed: 277 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,277 @@
1+
interactions:
2+
- request:
3+
body: '{"model": "models/gemini-2.5-flash", "ttl": "300s", "displayName": "braintrust-google-genai-vcr-test",
4+
"contents": [{"parts": [{"text": "The sky is blue because molecules scatter
5+
blue light. The sky is blue because molecules scatter blue light. The sky is
6+
blue because molecules scatter blue light. The sky is blue because molecules
7+
scatter blue light. The sky is blue because molecules scatter blue light. The
8+
sky is blue because molecules scatter blue light. The sky is blue because molecules
9+
scatter blue light. The sky is blue because molecules scatter blue light. The
10+
sky is blue because molecules scatter blue light. The sky is blue because molecules
11+
scatter blue light. The sky is blue because molecules scatter blue light. The
12+
sky is blue because molecules scatter blue light. The sky is blue because molecules
13+
scatter blue light. The sky is blue because molecules scatter blue light. The
14+
sky is blue because molecules scatter blue light. The sky is blue because molecules
15+
scatter blue light. The sky is blue because molecules scatter blue light. The
16+
sky is blue because molecules scatter blue light. The sky is blue because molecules
17+
scatter blue light. The sky is blue because molecules scatter blue light. The
18+
sky is blue because molecules scatter blue light. The sky is blue because molecules
19+
scatter blue light. The sky is blue because molecules scatter blue light. The
20+
sky is blue because molecules scatter blue light. The sky is blue because molecules
21+
scatter blue light. The sky is blue because molecules scatter blue light. The
22+
sky is blue because molecules scatter blue light. The sky is blue because molecules
23+
scatter blue light. The sky is blue because molecules scatter blue light. The
24+
sky is blue because molecules scatter blue light. The sky is blue because molecules
25+
scatter blue light. The sky is blue because molecules scatter blue light. The
26+
sky is blue because molecules scatter blue light. The sky is blue because molecules
27+
scatter blue light. The sky is blue because molecules scatter blue light. The
28+
sky is blue because molecules scatter blue light. The sky is blue because molecules
29+
scatter blue light. The sky is blue because molecules scatter blue light. The
30+
sky is blue because molecules scatter blue light. The sky is blue because molecules
31+
scatter blue light. The sky is blue because molecules scatter blue light. The
32+
sky is blue because molecules scatter blue light. The sky is blue because molecules
33+
scatter blue light. The sky is blue because molecules scatter blue light. The
34+
sky is blue because molecules scatter blue light. The sky is blue because molecules
35+
scatter blue light. The sky is blue because molecules scatter blue light. The
36+
sky is blue because molecules scatter blue light. The sky is blue because molecules
37+
scatter blue light. The sky is blue because molecules scatter blue light. The
38+
sky is blue because molecules scatter blue light. The sky is blue because molecules
39+
scatter blue light. The sky is blue because molecules scatter blue light. The
40+
sky is blue because molecules scatter blue light. The sky is blue because molecules
41+
scatter blue light. The sky is blue because molecules scatter blue light. The
42+
sky is blue because molecules scatter blue light. The sky is blue because molecules
43+
scatter blue light. The sky is blue because molecules scatter blue light. The
44+
sky is blue because molecules scatter blue light. The sky is blue because molecules
45+
scatter blue light. The sky is blue because molecules scatter blue light. The
46+
sky is blue because molecules scatter blue light. The sky is blue because molecules
47+
scatter blue light. The sky is blue because molecules scatter blue light. The
48+
sky is blue because molecules scatter blue light. The sky is blue because molecules
49+
scatter blue light. The sky is blue because molecules scatter blue light. The
50+
sky is blue because molecules scatter blue light. The sky is blue because molecules
51+
scatter blue light. The sky is blue because molecules scatter blue light. The
52+
sky is blue because molecules scatter blue light. The sky is blue because molecules
53+
scatter blue light. The sky is blue because molecules scatter blue light. The
54+
sky is blue because molecules scatter blue light. The sky is blue because molecules
55+
scatter blue light. The sky is blue because molecules scatter blue light. The
56+
sky is blue because molecules scatter blue light. The sky is blue because molecules
57+
scatter blue light. The sky is blue because molecules scatter blue light. The
58+
sky is blue because molecules scatter blue light. The sky is blue because molecules
59+
scatter blue light. The sky is blue because molecules scatter blue light. The
60+
sky is blue because molecules scatter blue light. The sky is blue because molecules
61+
scatter blue light. The sky is blue because molecules scatter blue light. The
62+
sky is blue because molecules scatter blue light. The sky is blue because molecules
63+
scatter blue light. The sky is blue because molecules scatter blue light. The
64+
sky is blue because molecules scatter blue light. The sky is blue because molecules
65+
scatter blue light. The sky is blue because molecules scatter blue light. The
66+
sky is blue because molecules scatter blue light. The sky is blue because molecules
67+
scatter blue light. The sky is blue because molecules scatter blue light. The
68+
sky is blue because molecules scatter blue light. The sky is blue because molecules
69+
scatter blue light. The sky is blue because molecules scatter blue light. The
70+
sky is blue because molecules scatter blue light. The sky is blue because molecules
71+
scatter blue light. The sky is blue because molecules scatter blue light. The
72+
sky is blue because molecules scatter blue light. The sky is blue because molecules
73+
scatter blue light. The sky is blue because molecules scatter blue light. The
74+
sky is blue because molecules scatter blue light. The sky is blue because molecules
75+
scatter blue light. The sky is blue because molecules scatter blue light. The
76+
sky is blue because molecules scatter blue light. The sky is blue because molecules
77+
scatter blue light. The sky is blue because molecules scatter blue light. The
78+
sky is blue because molecules scatter blue light. The sky is blue because molecules
79+
scatter blue light. The sky is blue because molecules scatter blue light. The
80+
sky is blue because molecules scatter blue light. The sky is blue because molecules
81+
scatter blue light. The sky is blue because molecules scatter blue light. The
82+
sky is blue because molecules scatter blue light. The sky is blue because molecules
83+
scatter blue light. The sky is blue because molecules scatter blue light. The
84+
sky is blue because molecules scatter blue light. The sky is blue because molecules
85+
scatter blue light. The sky is blue because molecules scatter blue light. The
86+
sky is blue because molecules scatter blue light. The sky is blue because molecules
87+
scatter blue light. The sky is blue because molecules scatter blue light. The
88+
sky is blue because molecules scatter blue light. The sky is blue because molecules
89+
scatter blue light. The sky is blue because molecules scatter blue light. The
90+
sky is blue because molecules scatter blue light. The sky is blue because molecules
91+
scatter blue light. The sky is blue because molecules scatter blue light. The
92+
sky is blue because molecules scatter blue light. The sky is blue because molecules
93+
scatter blue light. The sky is blue because molecules scatter blue light. The
94+
sky is blue because molecules scatter blue light. The sky is blue because molecules
95+
scatter blue light. The sky is blue because molecules scatter blue light. The
96+
sky is blue because molecules scatter blue light. The sky is blue because molecules
97+
scatter blue light. The sky is blue because molecules scatter blue light. The
98+
sky is blue because molecules scatter blue light. The sky is blue because molecules
99+
scatter blue light. The sky is blue because molecules scatter blue light. The
100+
sky is blue because molecules scatter blue light. The sky is blue because molecules
101+
scatter blue light. The sky is blue because molecules scatter blue light. The
102+
sky is blue because molecules scatter blue light. The sky is blue because molecules
103+
scatter blue light. The sky is blue because molecules scatter blue light. The
104+
sky is blue because molecules scatter blue light. "}], "role": "user"}]}'
105+
headers:
106+
accept:
107+
- '*/*'
108+
accept-encoding:
109+
- gzip, deflate
110+
connection:
111+
- keep-alive
112+
content-length:
113+
- '8259'
114+
content-type:
115+
- application/json
116+
host:
117+
- generativelanguage.googleapis.com
118+
user-agent:
119+
- google-genai-sdk/2.14.0 gl-python/3.14.6
120+
x-goog-api-client:
121+
- google-genai-sdk/2.14.0 gl-python/3.14.6
122+
method: POST
123+
uri: https://generativelanguage.googleapis.com/v1beta/cachedContents
124+
response:
125+
body:
126+
string: "{\n \"name\": \"cachedContents/aixcv88yts91zirycnmno31aitdexjt7uc6b19mw\",\n
127+
\ \"model\": \"models/gemini-2.5-flash\",\n \"createTime\": \"2026-07-29T19:15:22.193758Z\",\n
128+
\ \"updateTime\": \"2026-07-29T19:15:22.193758Z\",\n \"expireTime\": \"2026-07-29T19:20:21.884570801Z\",\n
129+
\ \"displayName\": \"braintrust-google-genai-vcr-test\",\n \"usageMetadata\":
130+
{\n \"totalTokenCount\": 1502\n }\n}\n"
131+
headers:
132+
alt-svc:
133+
- h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
134+
content-length:
135+
- '361'
136+
content-type:
137+
- application/json; charset=UTF-8
138+
date:
139+
- Wed, 29 Jul 2026 19:15:22 GMT
140+
server:
141+
- scaffolding on HTTPServer2
142+
server-timing:
143+
- gfet4t7; dur=591
144+
transfer-encoding:
145+
- chunked
146+
vary:
147+
- Origin
148+
- X-Origin
149+
- Referer
150+
x-content-type-options:
151+
- nosniff
152+
x-frame-options:
153+
- SAMEORIGIN
154+
x-xss-protection:
155+
- '0'
156+
status:
157+
code: 200
158+
message: OK
159+
- request:
160+
body: '{"contents": [{"parts": [{"text": "In one sentence, why is the sky blue?"}],
161+
"role": "user"}], "cachedContent": "cachedContents/aixcv88yts91zirycnmno31aitdexjt7uc6b19mw",
162+
"generationConfig": {"maxOutputTokens": 100}}'
163+
headers:
164+
accept:
165+
- '*/*'
166+
accept-encoding:
167+
- gzip, deflate
168+
connection:
169+
- keep-alive
170+
content-length:
171+
- '216'
172+
content-type:
173+
- application/json
174+
host:
175+
- generativelanguage.googleapis.com
176+
user-agent:
177+
- google-genai-sdk/2.14.0 gl-python/3.14.6
178+
x-goog-api-client:
179+
- google-genai-sdk/2.14.0 gl-python/3.14.6
180+
method: POST
181+
uri: https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent
182+
response:
183+
body:
184+
string: "{\n \"candidates\": [\n {\n \"content\": {\n \"parts\":
185+
[\n {\n \"text\": \"The sky is blue because molecules
186+
scatter blue light.\"\n }\n ],\n \"role\": \"model\"\n
187+
\ },\n \"finishReason\": \"STOP\",\n \"index\": 0\n }\n ],\n
188+
\ \"usageMetadata\": {\n \"promptTokenCount\": 1513,\n \"candidatesTokenCount\":
189+
10,\n \"totalTokenCount\": 1595,\n \"cachedContentTokenCount\": 1502,\n
190+
\ \"promptTokensDetails\": [\n {\n \"modality\": \"TEXT\",\n
191+
\ \"tokenCount\": 1513\n }\n ],\n \"cacheTokensDetails\":
192+
[\n {\n \"modality\": \"TEXT\",\n \"tokenCount\": 1502\n
193+
\ }\n ],\n \"thoughtsTokenCount\": 72,\n \"serviceTier\": \"standard\"\n
194+
\ },\n \"modelVersion\": \"gemini-2.5-flash\",\n \"responseId\": \"SlFqaqzJGYGr1MkP4cu4yQ0\"\n}\n"
195+
headers:
196+
alt-svc:
197+
- h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
198+
content-length:
199+
- '775'
200+
content-type:
201+
- application/json; charset=UTF-8
202+
date:
203+
- Wed, 29 Jul 2026 19:15:23 GMT
204+
server:
205+
- scaffolding on HTTPServer2
206+
server-timing:
207+
- gfet4t7; dur=888
208+
transfer-encoding:
209+
- chunked
210+
vary:
211+
- Origin
212+
- X-Origin
213+
- Referer
214+
x-content-type-options:
215+
- nosniff
216+
x-frame-options:
217+
- SAMEORIGIN
218+
x-gemini-service-tier:
219+
- standard
220+
x-xss-protection:
221+
- '0'
222+
status:
223+
code: 200
224+
message: OK
225+
- request:
226+
body: ''
227+
headers:
228+
accept:
229+
- '*/*'
230+
accept-encoding:
231+
- gzip, deflate
232+
connection:
233+
- keep-alive
234+
content-type:
235+
- application/json
236+
host:
237+
- generativelanguage.googleapis.com
238+
user-agent:
239+
- google-genai-sdk/2.14.0 gl-python/3.14.6
240+
x-goog-api-client:
241+
- google-genai-sdk/2.14.0 gl-python/3.14.6
242+
method: DELETE
243+
uri: https://generativelanguage.googleapis.com/v1beta/cachedContents/aixcv88yts91zirycnmno31aitdexjt7uc6b19mw
244+
response:
245+
body:
246+
string: '{}
247+
248+
'
249+
headers:
250+
alt-svc:
251+
- h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
252+
content-length:
253+
- '3'
254+
content-type:
255+
- application/json; charset=UTF-8
256+
date:
257+
- Wed, 29 Jul 2026 19:15:23 GMT
258+
server:
259+
- scaffolding on HTTPServer2
260+
server-timing:
261+
- gfet4t7; dur=120
262+
transfer-encoding:
263+
- chunked
264+
vary:
265+
- Origin
266+
- X-Origin
267+
- Referer
268+
x-content-type-options:
269+
- nosniff
270+
x-frame-options:
271+
- SAMEORIGIN
272+
x-xss-protection:
273+
- '0'
274+
status:
275+
code: 200
276+
message: OK
277+
version: 1

0 commit comments

Comments
 (0)