Conversation
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed everything up to 0b71b49 in 11 seconds. Click for details.
- Reviewed
675lines of code in2files - Skipped
0files when reviewing. - Skipped posting
0draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
Workflow ID: wflow_7RWo3HNeOSMczM3N
You can customize 
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughRenames auto-monitor-setups endpoints to use Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
openapi.json (1)
298-309: Creation endpoint returns200instead of201.The auto-monitor-setups POST (line 82) returns
201 Created, which is the standard for resource creation. This custom-evaluator POST returns200 OK. Consider using201for consistency if this endpoint creates a new resource.Proposed fix
"responses": { - "200": { - "description": "OK", + "201": { + "description": "Created", "content": {🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@openapi.json` around lines 298 - 309, The POST for the "custom-evaluator" path currently returns a 200 response; change it to return HTTP 201 Created to match the resource-creation semantics used by the auto-monitor-setups POST and to be consistent; update the response status key from "200" to "201" and the description to "Created", keeping the same content schema reference to "#/components/schemas/response.CreateEvaluatorResponse" so the OpenAPI operation (custom-evaluator POST) still returns the same payload but with a 201 status.openapi-whitelist.yaml (1)
20-28: Duplicate/v2/metrics_hwmentry.Lines 21 and 26 both include
/v2/metrics_hwm. While the new addition on line 24 is correct, the duplicate pre-existing entry should be cleaned up to avoid confusion.Proposed fix
include_paths: - - /v2/metrics_hwm - /v2/organizations - /v2/evaluators/execute/* - /v2/evaluators/custom-evaluator - /v2/metrics - /v2/metrics_hwm - /v2/auto-monitor-setups - /v2/auto-monitor-setups/*🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@openapi-whitelist.yaml` around lines 20 - 28, The include_paths list contains a duplicate entry for "/v2/metrics_hwm"; remove the redundant occurrence so that "/v2/metrics_hwm" appears only once under include_paths (keep the correct/new entry and delete the duplicate pre-existing one) to avoid confusion and potential processing issues when reading the include_paths array.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@openapi.json`:
- Around line 4596-4605: Fix the typo in the OpenAPI enum varnames: locate the
x-enum-varnames array for the enum that includes "INTENT_CHANGE" and replace
"IdentIntentChange" with the correct PascalCase "IntentChange" so the mapping
matches the enum value INTENT_CHANGE; update the x-enum-varnames entry that
currently reads "IdentIntentChange" to "IntentChange" (verify adjacent entries
like "AgentGoalCompleteness" and "ContextRelevance" to ensure consistency).
---
Nitpick comments:
In `@openapi-whitelist.yaml`:
- Around line 20-28: The include_paths list contains a duplicate entry for
"/v2/metrics_hwm"; remove the redundant occurrence so that "/v2/metrics_hwm"
appears only once under include_paths (keep the correct/new entry and delete the
duplicate pre-existing one) to avoid confusion and potential processing issues
when reading the include_paths array.
In `@openapi.json`:
- Around line 298-309: The POST for the "custom-evaluator" path currently
returns a 200 response; change it to return HTTP 201 Created to match the
resource-creation semantics used by the auto-monitor-setups POST and to be
consistent; update the response status key from "200" to "201" and the
description to "Created", keeping the same content schema reference to
"#/components/schemas/response.CreateEvaluatorResponse" so the OpenAPI operation
(custom-evaluator POST) still returns the same payload but with a 201 status.
ℹ️ Review info
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (7)
api-reference/auto-monitor-setups/delete-an-auto-monitor-setup-by-external-id.mdxapi-reference/auto-monitor-setups/get-an-auto-monitor-setup-by-external-id.mdxapi-reference/auto-monitor-setups/update-an-auto-monitor-setup-by-external-id.mdxapi-reference/evaluators/create-a-custom-llm-as-a-judge-evaluator.mdxmint.jsonopenapi-whitelist.yamlopenapi.json
| "AgentEfficiency", | ||
| "AgentGoalCompleteness", | ||
| "ConversationQuality", | ||
| "IdentIntentChange", | ||
| "ContextRelevance", | ||
| "AgentToolErrorDetector", | ||
| "AgentToolTrajectory", | ||
| "HtmlComparison", | ||
| "LLMAsAJudge" | ||
| ] |
There was a problem hiding this comment.
Typo in x-enum-varnames: "IdentIntentChange" should be "IntentChange".
All other varnames follow PascalCase conversion of their enum values (e.g., "AGENT_GOAL_ACCURACY" → "AgentGoalAccuracy"). The enum value "INTENT_CHANGE" should map to "IntentChange", not "IdentIntentChange".
Proposed fix
- "IdentIntentChange",
+ "IntentChange",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "AgentEfficiency", | |
| "AgentGoalCompleteness", | |
| "ConversationQuality", | |
| "IdentIntentChange", | |
| "ContextRelevance", | |
| "AgentToolErrorDetector", | |
| "AgentToolTrajectory", | |
| "HtmlComparison", | |
| "LLMAsAJudge" | |
| ] | |
| "AgentEfficiency", | |
| "AgentGoalCompleteness", | |
| "ConversationQuality", | |
| "IntentChange", | |
| "ContextRelevance", | |
| "AgentToolErrorDetector", | |
| "AgentToolTrajectory", | |
| "HtmlComparison", | |
| "LLMAsAJudge" | |
| ] |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@openapi.json` around lines 4596 - 4605, Fix the typo in the OpenAPI enum
varnames: locate the x-enum-varnames array for the enum that includes
"INTENT_CHANGE" and replace "IdentIntentChange" with the correct PascalCase
"IntentChange" so the mapping matches the enum value INTENT_CHANGE; update the
x-enum-varnames entry that currently reads "IdentIntentChange" to "IntentChange"
(verify adjacent entries like "AgentGoalCompleteness" and "ContextRelevance" to
ensure consistency).
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
openapi.json (1)
277-327:⚠️ Potential issue | 🟠 MajorAdd auth requirement to custom evaluator creation endpoint.
POST /v2/evaluators/custom-evaluatoris missing asecurityrequirement while neighboring evaluator endpoints requireBearerAuth. In the spec, this exposes the operation as publicly callable and can lead to insecure client generation/usage.🔐 Proposed OpenAPI fix
"/v2/evaluators/custom-evaluator": { "post": { @@ "responses": { @@ }, + "security": [ + { + "BearerAuth": [] + } + ], "summary": "Create a custom LLM-as-a-judge evaluator", "tags": [ "evaluators" ] } },🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@openapi.json` around lines 277 - 327, The POST operation for the path "/v2/evaluators/custom-evaluator" is missing an OpenAPI security requirement; update the operation object for this path/method to include the same BearerAuth security declaration used by neighboring evaluator endpoints (add a "security": [{ "BearerAuth": [] }] entry on the POST), leaving the existing requestBody schema ($ref to request.CreateCustomEvaluatorInput) and responses (including response.CreateEvaluatorResponse and response.ErrorResponse) unchanged so the endpoint becomes protected by Bearer token authentication.
♻️ Duplicate comments (1)
openapi.json (1)
4583-4583:⚠️ Potential issue | 🟡 MinorFix enum varname typo:
IdentIntentChange→IntentChange.This still mismatches the enum value
INTENT_CHANGEand can generate incorrect SDK identifiers.✏️ Proposed fix
- "IdentIntentChange", + "IntentChange",🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@openapi.json` at line 4583, The enum entry "IdentIntentChange" is a typo and should be "IntentChange" to match the enum value INTENT_CHANGE; locate the enum array containing "IdentIntentChange" in openapi.json and replace that string with "IntentChange" so generated SDK identifiers align with the INTENT_CHANGE value.
🧹 Nitpick comments (1)
openapi.json (1)
3433-3435:placeholder_valueexample looks inverted.With
placeholder_name: "name",placeholder_valueshould likely be the substituted value (e.g.,"John Doe"), not a template string.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@openapi.json` around lines 3433 - 3435, The example for the schema property placeholder_value is inverted: instead of showing a template string it should show the actual substituted value corresponding to placeholder_name (e.g., "John Doe"). Update the example for placeholder_value (and any related examples in the same object) to use a concrete substituted value rather than "Hello ${name}", and ensure placeholder_name remains the template key (e.g., "name") so examples reflect template -> substituted value mapping.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@openapi.json`:
- Around line 3920-3923: The response schema renamed the field `result` to
`success`, which breaks existing clients; restore backward compatibility by
adding `result` back alongside `success` in the affected OpenAPI response
schemas (the same schema objects that currently define `"success": { "type":
"boolean", "example": true }`), set `result` to the identical schema and mark it
deprecated (e.g., add "deprecated": true or an "x-deprecated" vendor extension)
and update example objects to include both fields with the same value; apply
this change to all occurrences referenced (the schema blocks that currently
contain `"success"`) so both `result` and `success` are returned until clients
migrate.
---
Outside diff comments:
In `@openapi.json`:
- Around line 277-327: The POST operation for the path
"/v2/evaluators/custom-evaluator" is missing an OpenAPI security requirement;
update the operation object for this path/method to include the same BearerAuth
security declaration used by neighboring evaluator endpoints (add a "security":
[{ "BearerAuth": [] }] entry on the POST), leaving the existing requestBody
schema ($ref to request.CreateCustomEvaluatorInput) and responses (including
response.CreateEvaluatorResponse and response.ErrorResponse) unchanged so the
endpoint becomes protected by Bearer token authentication.
---
Duplicate comments:
In `@openapi.json`:
- Line 4583: The enum entry "IdentIntentChange" is a typo and should be
"IntentChange" to match the enum value INTENT_CHANGE; locate the enum array
containing "IdentIntentChange" in openapi.json and replace that string with
"IntentChange" so generated SDK identifiers align with the INTENT_CHANGE value.
---
Nitpick comments:
In `@openapi.json`:
- Around line 3433-3435: The example for the schema property placeholder_value
is inverted: instead of showing a template string it should show the actual
substituted value corresponding to placeholder_name (e.g., "John Doe"). Update
the example for placeholder_value (and any related examples in the same object)
to use a concrete substituted value rather than "Hello ${name}", and ensure
placeholder_name remains the template key (e.g., "name") so examples reflect
template -> substituted value mapping.
ℹ️ Review info
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
openapi.json
| "success": { | ||
| "example": true, | ||
| "type": "boolean" | ||
| } |
There was a problem hiding this comment.
result → success is a breaking response contract change in v2.
These schema renames can break existing clients deserializing result. Please either preserve backward compatibility (keep result as deprecated alias) or clearly version/broadcast this contract break.
Also applies to: 3946-3949, 3959-3962, 3976-3979, 4225-4228
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@openapi.json` around lines 3920 - 3923, The response schema renamed the field
`result` to `success`, which breaks existing clients; restore backward
compatibility by adding `result` back alongside `success` in the affected
OpenAPI response schemas (the same schema objects that currently define
`"success": { "type": "boolean", "example": true }`), set `result` to the
identical schema and mark it deprecated (e.g., add "deprecated": true or an
"x-deprecated" vendor extension) and update example objects to include both
fields with the same value; apply this change to all occurrences referenced (the
schema blocks that currently contain `"success"`) so both `result` and `success`
are returned until clients migrate.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@api-reference/auto-monitor-setups/create-an-auto-monitor-setup.mdx`:
- Around line 34-48: The ParamField for "selector" is unclear about optionality
and default behavior; update the documentation for ParamField body="selector" to
explicitly state that selector is optional (unlike required fields external_id
and evaluators) and describe the default: if omitted, the monitor applies to all
spans (i.e., no attribute filtering). Also mention accepted value types (string,
number, boolean) and that specifying key-value pairs requires all to match.
ℹ️ Review info
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
api-reference/auto-monitor-setups/create-an-auto-monitor-setup.mdx
| <ParamField body="selector" type="object"> | ||
| A map of span attribute key-value pairs used to filter which spans this monitor applies to. Only spans matching **all** provided key-value pairs will be evaluated. | ||
|
|
||
| Keys are span attribute names and values can be strings, numbers, or booleans. | ||
|
|
||
| **Common selector keys:** | ||
|
|
||
| | Key | Type | Example | | ||
| |-----|------|---------| | ||
| | `gen_ai.system` | string | `"openai"`, `"anthropic"` | | ||
| | `gen_ai.request.model` | string | `"gpt-4o"`, `"claude-3-5-sonnet"` | | ||
| | `gen_ai.request.max_tokens` | number | `1000` | | ||
|
|
||
| Any other span attribute from your traces can also be used as a selector key. | ||
| </ParamField> |
There was a problem hiding this comment.
Clarify whether selector is optional and document default behavior.
The selector parameter is not marked as required, unlike external_id and evaluators. The documentation should explicitly state whether this field is optional and what happens when it's omitted (e.g., does the monitor apply to all spans?).
📝 Suggested clarification
-<ParamField body="selector" type="object">
+<ParamField body="selector" type="object" required>
A map of span attribute key-value pairs used to filter which spans this monitor applies to. Only spans matching **all** provided key-value pairs will be evaluated.Or if it's truly optional, add clarifying text:
<ParamField body="selector" type="object">
- A map of span attribute key-value pairs used to filter which spans this monitor applies to. Only spans matching **all** provided key-value pairs will be evaluated.
+ A map of span attribute key-value pairs used to filter which spans this monitor applies to. Only spans matching **all** provided key-value pairs will be evaluated. If omitted, the monitor will apply to all incoming spans.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@api-reference/auto-monitor-setups/create-an-auto-monitor-setup.mdx` around
lines 34 - 48, The ParamField for "selector" is unclear about optionality and
default behavior; update the documentation for ParamField body="selector" to
explicitly state that selector is optional (unlike required fields external_id
and evaluators) and describe the default: if omitted, the monitor applies to all
spans (i.e., no attribute filtering). Also mention accepted value types (string,
number, boolean) and that specifying key-value pairs requires all to match.
There was a problem hiding this comment.
♻️ Duplicate comments (1)
api-reference/auto-monitor-setups/create-an-auto-monitor-setup.mdx (1)
28-31:⚠️ Potential issue | 🟡 MinorClarify default behavior when
selectoris omitted.The field semantics are clearer now, but it still doesn’t explicitly state the default behavior if
selectoris not provided.📝 Suggested wording
<ParamField body="selector" type="object"> - A map of span attribute key-value pairs used to filter which spans this monitor applies to. Only spans matching **all** provided key-value pairs will be evaluated. + Optional map of span attribute key-value pairs used to filter which spans this monitor applies to. Only spans matching **all** provided key-value pairs will be evaluated. If omitted, the monitor applies to all incoming spans (no attribute filtering).🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@api-reference/auto-monitor-setups/create-an-auto-monitor-setup.mdx` around lines 28 - 31, Clarify the default behavior for the ParamField named "selector": state that if `selector` is omitted (or an empty object) the monitor applies to all spans (no filtering), and explain that providing key-value pairs will restrict evaluation to only spans matching all pairs; update the ParamField description for `selector` to explicitly include this default behavior and an example sentence noting that omission equals "match all spans".
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@api-reference/auto-monitor-setups/create-an-auto-monitor-setup.mdx`:
- Around line 28-31: Clarify the default behavior for the ParamField named
"selector": state that if `selector` is omitted (or an empty object) the monitor
applies to all spans (no filtering), and explain that providing key-value pairs
will restrict evaluation to only spans matching all pairs; update the ParamField
description for `selector` to explicitly include this default behavior and an
example sentence noting that omission equals "match all spans".
ℹ️ Review info
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
api-reference/auto-monitor-setups/create-an-auto-monitor-setup.mdx
2 participants
Mintlify
0 threads from 0 users in Mintlify
Important
The PR updates documentation by restructuring evaluator routes, removing the custom evaluator endpoint, and modifying schemas in
mint.jsonandopenapi.json.mint.jsonby flattening the hierarchy and removing groupings./v2/evaluators/custom-evaluatorendpoint fromopenapi.json.evaluator.Propertyschema inopenapi.jsonto includeenum_values.completiontorequest.ContextRelevanceInputinopenapi.json.responseproperties fromresulttosuccessinopenapi.jsonfor several evaluators.context-relevanceandplaceholder-regexevaluators inopenapi.json.This description was created by
for 0b71b49. You can customize this summary. It will automatically update as commits are pushed.
Summary by CodeRabbit
Documentation
New Features