Skip to content

[Lens] Add AGENTS.md for the editor-to-render pipeline - #2

Closed
timductive wants to merge 1 commit into
markov00:2026_08_27-fix_multi_duration_unit_conversionfrom
timductive:claude/lens-agents-md-docs-f4effe
Closed

[Lens] Add AGENTS.md for the editor-to-render pipeline#2
timductive wants to merge 1 commit into
markov00:2026_08_27-fix_multi_duration_unit_conversionfrom
timductive:claude/lens-agents-md-docs-f4effe

Conversation

@timductive

@timductive timductive commented Sep 22, 2026

Copy link
Copy Markdown

Stacked on your elastic#289007 — targeting your branch so the diff stays to the four doc files and every link resolves. Merge it into your branch whenever suits and it rides along; happy to retarget at main instead if you'd rather keep it separate.

Summary

Adds agent-facing architecture notes for Lens and expressionXY, so future agents pick the right layer for a fix instead of patching the last stage before the symptom becomes visible.

The worked example is the duration-unit bug (elastic#240105) and the two attempts at it:

That pair is unusually good teaching material: same bug, same author-intent, two layers, and a written ADR explaining why one is wrong. These notes capture the reasoning so it isn't lost once both PRs scroll out of view.

What's in it

x-pack/platform/plugins/shared/lens/AGENTS.md — the entry point, since an agent handed a "Lens bug" starts there. A 7-stage pipeline table (editor state → buildExpression() → data fetch → lens_format_column → chart expression fn → renderer → React) with what each stage can and cannot see, plus the layer-selection rule and the generalizable heuristics:

  • find the earliest stage at which every consumer of a value can be made to agree
  • data semantics belong in common/ expression functions, never in React render
  • effective formats only exist after the datatable is decorated at runtime, so never reconstruct a format from editor state
  • duplicating a helper in order to reach it is a wrong-layer signal — move it down instead
  • conflicting user configs are a product decision needing a deterministic rule, UI surfacing and an ADR, not a silent default
  • tests written at the wrong seam cannot detect the wrong seam

src/platform/plugins/shared/chart_expressions/expression_xy/AGENTS.md — the specifics where fixes actually land. A common/ vs public/ ownership table naming the expression functions as the normalization seam, the policy modules from your PR, the invariants (log before transform, immutable one-pass copies, both entry points share one policy, the anchor converts too, unformatted values stay axis-relative), and a table of every consumer of a Y axis with how each resolves its formatter.

It defers to your CONTEXT.md and ADR as canonical rather than restating them — the glossary lives in one place and the rejected-options list is deep-linked, so there's nothing to drift.

READMEs — one pointer line each, into your existing ## Design section and at the top of the Lens readme.

Notes for review

  • Docs only; no code, no tests, no CI surface.
  • The consumer table was written against your merged state and verified against source — reference_line.tsx:56, tooltip.tsx:88, axes_configuration.ts:176. If the formatter precedence shifts before [Lens] Normalize data based on duration formats for multi-series xy chart elastic/kibana#289007 lands, that table needs a matching update.
  • I deliberately did not merge main in; the base is ~1242 commits behind, and for a docs-only change that merge is pure conflict risk.

🤖 Generated with Claude Code

@timductive @claude
[Lens] Add AGENTS.md for the editor-to-render pipeline
a0f433d 
Adds agent-facing architecture notes for Lens and expressionXY, so future
agents pick the right layer for a fix instead of patching the last stage
before the symptom.

The worked example is the duration-unit bug (elastic#240105): PR elastic#275682 was closed
because it normalized values during React render in `xy_chart.tsx`, which the
axis-owned format policy ADR rejects by name. PR elastic#289007 resolves the policy in
the expression functions instead. These notes record why, plus the generalizable
heuristics behind it:

- find the earliest stage where every consumer of a value can agree
- data semantics belong in `common/` expression functions, never in render
- effective formats only exist after the datatable is decorated at runtime
- duplicating a helper to reach it is a wrong-layer signal
- conflicting user configs are a product decision, not a silent default
- tests written at the wrong seam cannot detect the wrong seam

`expression_xy/AGENTS.md` defers to the existing CONTEXT.md and ADR as canonical
rather than restating them, and enumerates every consumer of a Y axis with how
each one resolves its formatter.

Stacked on elastic#289007, which adds the policy modules, CONTEXT.md and the ADR these
docs link to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@timductive timductive closed this Sep 25, 2026
markov00 pushed a commit that referenced this pull request Oct 2, 2026
@flash1293 @claude
@CoenWarmer @kibanamachine
[Connectors] Add Bitbucket Cloud connector (elastic#288824)
4a5bbfc 
## Summary

Adds a **Bitbucket Cloud** connector spec (`.bitbucket`) so workflows
and agents can drive the pull-request, branch, commit-status, and
pipeline lifecycle on a Bitbucket Cloud workspace without opening the
Bitbucket UI or hand-writing REST calls.

Closes elastic/security-team#18429

### Actions

| Group | Actions |
|---|---|
| Discovery | `listRepositories` |
| Pull requests | `createPullRequest`, `getPullRequest`,
`listPullRequests`, `updatePullRequest`, `mergePullRequest`,
`approvePullRequest`, `declinePullRequest`, `addPullRequestComment` |
| Branches | `createBranch`, `getBranch`, `deleteBranch` |
| Commits | `getCommit`, `listCommits`, `createCommitBuildStatus`,
`listCommitBuildStatuses` |
| Pipelines | `triggerPipeline`, `getPipeline`, `stopPipeline` |

Every action returns a curated camelCase shape (no `links`/avatar noise)
and surfaces Bitbucket API errors as `Bitbucket <action> failed (status
N): <message> - <detail>`.

### Authentication

- **Basic (recommended)**: Atlassian account email + API token created
**with scopes** for the Bitbucket app. Bitbucket rejects unscoped API
tokens (`API Token provided has no Bitbucket scopes.`), so the
in-product help text, the docs page's Authentication line, and the docs'
setup steps all list the same six scopes.
- **Bearer**: Bitbucket repository/project/workspace access token (or a
scoped API token). A repository-scoped access token cannot list the
whole workspace, so the connectivity test recognizes a 401/403 from that
call under bearer auth and reports the scope limitation instead of
reporting valid credentials as broken; every repository-scoped action
still works with that token.

The connector config holds the **workspace** slug; repositories are
addressed by slug in every action.

#### Deviation from the issue: no app password option

The issue's first acceptance criterion asks for "a workspace and an app
password (or an Atlassian email and API token)". This PR deliberately
does **not** offer app passwords:

- Atlassian [deprecated Bitbucket Cloud app
passwords](https://community.atlassian.com/forums/Bitbucket-articles/Deprecation-notice-Bitbucket-Cloud-shifts-to-API-tokens-from-app/ba-p/3040975)
in favour of API tokens. Creating new app passwords was disabled on 9
September 2025, and existing app passwords stopped authenticating on 9
June 2026, so at the time of this PR there is no working app password a
user could enter.
- The replacement, an Atlassian API token **with Bitbucket scopes**, is
what the Basic auth option takes (email + token). Technically the same
Basic auth fields would have accepted a Bitbucket username + app
password while those still worked, but documenting that path would send
users to a UI that no longer exists.
- For non-user (service) identities, the Bearer option covers Bitbucket
repository, project, and workspace access tokens, which is Atlassian's
recommended replacement for automation that previously used app
passwords.

### Notes for reviewers

- `supportedFeatureIds` is `['agentBuilder']` only, per the two-step
release rule for new connector types. A follow-up PR will add
`'workflows'` once this type is registered in every Production-NonCanary
version. The docs page carries a note stating this connector is
currently Agent Builder only.
- Vendor quirks verified against the live API and handled in code:
- Bitbucket ignores the `state` query params **and** its OPEN-only
default whenever a `q` filter is present, so `listPullRequests` folds
the state filter into `q` when a query is given. `state` also requires
`.min(1)` so an empty array cannot produce a malformed `() AND (...)`
query or silently fall back to the OPEN default.
- Repeated-key form (`state=OPEN&state=MERGED`) is required for
multi-state filters; the axios default bracket form is silently ignored.
- `PUT /pullrequests/{id}` preserves omitted `title`/`description`, but
reviewer handling on omission is undocumented, so `updatePullRequest`
reads the current PR and always sends the full field set.
- UUIDs (users, pipelines) must be brace-wrapped; the handlers normalise
inputs with or without braces.
- Bitbucket's commits endpoint paginates with an opaque cursor embedded
in the `next` link, not sequential page numbers - its own docs say to
follow that link rather than construct one. `listCommits` now returns
`nextCursor`/`hasMore` and accepts the cursor back as input, requesting
it directly instead of rebuilding it. The cursor is caller-supplied, so
it is rejected unless it starts with the Bitbucket API base URL, to keep
a workflow author from redirecting the connector's credentials to an
arbitrary host.
- `createCommitBuildStatus` is `scope: 'destroy'`, not `write`:
reposting an existing key overwrites the previous status, which the
`ActionScope` contract reserves for updates/overwrites.
`listCommitBuildStatuses` (new) is the read counterpart, since
`getPullRequest` has no status fields and nothing else could verify a
build status before merging.
- Icon is the Bitbucket mark from Simple Icons in Atlassian blue
(`#0052CC`).
- Docs quality skills (`docs-check-style` etc.) from elastic-docs-skills
were not installed in the authoring environment; the page follows the
Buildkite/Jira Cloud connector docs structure.

## Validated

Every action was exercised end-to-end through the Kibana Actions
`_execute` API against a live Bitbucket Cloud workspace (`flash1293`,
private repo `test`, and separately `semla`/`foo`) using a scoped
Atlassian API token. Two Agent Builder chat tests (agent wired to
`sml_search`/`sml_attach`/`execute_connector_sub_action`) drove
`listRepositories`, `listPullRequests` (multi-state), `getBranch`,
`triggerPipeline` (custom pipeline + variables), `stopPipeline`,
`getPipeline`, and `createCommitBuildStatus` with all optional fields,
choosing correct sub-action names and parameters from the spec
descriptions alone. The first chat also exposed that list responses omit
participants, so `approvalCount`/`reviewers`/`participants` are now
`undefined` rather than `0`/`[]` when Bitbucket does not return them.

| Action | What was tested | Result |
| --- | --- | --- |
| test (connectivity) | `_test` sub-action with email + scoped API
token; returned `Connected to Bitbucket workspace "flash1293"` /
`"semla"`. Also confirmed a repository-scoped bearer token's 403 on this
workspace-wide call is reported as a scope limitation, not a generic
failure | ✅ Pass |
| `listRepositories` | Listed workspace repos; returned the private
`test`/`foo` repo with slug, main branch, project key, URL, `hasMore:
false` | ✅ Pass |
| `createPullRequest` | Opened PR #1 from `kibana-e2e/remediation-*` →
`main` with description and `closeSourceBranch: true`; opened PR #2 as
`draft: true` with default destination; also confirmed Bitbucket's "no
changes to be pulled" and "author cannot be a reviewer" errors surface
verbatim | ✅ Pass |
| `getPullRequest` | Read PR #1 before/after approval; `approvalCount`
went 0 → 1, `commentCount` 3, `sourceCommit`/`destinationBranch`
populated | ✅ Pass |
| `listPullRequests` | Default (OPEN only), `state:
["MERGED","DECLINED"]` with `sort`, `state` + `query: title ~
"Remediate"`, and an empty `state: []` rejected at schema validation
before any request | ✅ Pass |
| `updatePullRequest` | Title-only update preserved description;
`reviewers: []` + multi-line description update; rejected with no fields
set | ✅ Pass |
| `mergePullRequest` | Squash merge of a non-draft PR with custom
message and `closeSourceBranch: true`; state `MERGED`, `mergeCommit`
returned, source branch gone (`getBranch` → 404); merging a draft PR
correctly rejected | ✅ Pass |
| `approvePullRequest` | Approved a PR; returned participant with
`approved: true`, `state: "approved"` | ✅ Pass |
| `declinePullRequest` | Declined a PR → `DECLINED`; second decline
surfaced `status 400: This pull request is already closed.` | ✅ Pass |
| `addPullRequestComment` | General comment with markdown link, inline
comment on a file line (`inline.to`), threaded reply via
`parentCommentId` | ✅ Pass |
| `createBranch` | From a full commit hash and from the branch name
`main`; branch names containing `/` are URL-encoded | ✅ Pass |
| `getBranch` | `main` and a `kibana-e2e/...` branch; tip commit,
parents, merge strategies returned; nonexistent branch 404s cleanly | ✅
Pass |
| `deleteBranch` | Deleted a merged/declined PR's branch (`{deleted:
true}`); deleting `main` surfaced `status 400: Cannot delete the main
branch` | ✅ Pass |
| `getCommit` | Read a commit by full hash; author user, parents, URL
returned; non-hex hash rejected by schema | ✅ Pass |
| `listCommits` | `revision` + `path` filter, no revision with
`pageSize`, and multi-page traversal via the new opaque
`cursor`/`nextCursor` following Bitbucket's documented `next`-link
pagination; an off-origin cursor rejected before any request | ✅ Pass |
| `createCommitBuildStatus` | `INPROGRESS` then `SUCCESSFUL` on the same
key (overwrite confirmed, hence `scope: 'destroy'`) with `refname` = PR
source branch | ✅ Pass |
| `listCommitBuildStatuses` | Listed statuses on a commit that had a
status posted by `createCommitBuildStatus`, confirming the value
round-trips | ✅ Pass |
| `triggerPipeline` | Branch-only (default pipeline, completed
`SUCCESSFUL`), branch + commit + `customPipeline` + secured/unsecured
`variables`, and bare commit + custom (`pipeline_commit_target`) | ✅
Pass |
| `getPipeline` | Polled with and without braces; observed `PARSING` →
`PENDING` → `COMPLETED`/`SUCCESSFUL` and `COMPLETED`/`STOPPED` | ✅ Pass
|
| `stopPipeline` | Stopped a running pipeline; second stop surfaced
`status 400: Cannot stop pipeline result that is already complete with
status STOPPED` | ✅ Pass |

## Testing this yourself

The Elastic Atlassian organization does not have Bitbucket enabled, so
you cannot test this connector with your Elastic Atlassian account. To
try it:

1. Create a **personal Atlassian account** (any email) at
https://id.atlassian.com/ and create a free Bitbucket Cloud workspace
with a private repository.
2. Create an API token **with scopes** at
https://id.atlassian.com/manage-profile/security/api-tokens, select the
Bitbucket app, and grant `read/write:repository`,
`read/write:pullrequest`, and `read/write:pipeline`. Tokens without
scopes are rejected by the Bitbucket API.
3. In Kibana, create a Bitbucket connector with your workspace slug, the
account email, and the token, then use the Test tab or run sub-actions
via `POST /api/actions/connector/{id}/_execute`.

## Test plan

- [x] `node scripts/check.js --scope=branch` (lint, Jest, tsc)
- [x] `node scripts/jest
src/platform/packages/shared/kbn-connector-specs/src/specs/bitbucket`
(52 tests)
- [x] `node scripts/type_check --project
src/platform/packages/shared/kbn-connector-specs/tsconfig.json`
- [x] `node scripts/i18n_check --path
src/platform/packages/shared/kbn-connector-specs/src/specs/bitbucket`
- [x] Live end-to-end run against Bitbucket Cloud (see Validated table)

## Follow-ups

- Add `'workflows'` to `supportedFeatureIds` once the type is in every
Production-NonCanary version.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01MDWdsgup91oM8zLp1m3Bos

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Coen Warmer <coen.warmer@gmail.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
markov00 pushed a commit that referenced this pull request Oct 2, 2026
@flash1293
[Elastic Console] Forward session id for prompt caching in OpenAI-com…
038cf22 
…patible endpoint (elastic#294736)

## Summary

The inference plugin already supports `sessionId` + `cacheControl` for
EIS prompt caching (elastic#284774), but the Elastic Console / Elastic Ramen
OpenAI-compatible endpoint
(`/internal/elastic_ramen/v1/chat/completions`) didn't forward them, so
external agents (Elastic Ramen, pi, …) never got prompt caching.

This PR adds that wiring:

- **Session id** is resolved from (first match wins):
1. `prompt_cache_key` body field (standard OpenAI field; used by Elastic
Ramen via `@ai-sdk/openai-compatible`)
2. `x-session-id` header (OpenRouter convention; pi sends this with
`sessionAffinityFormat: "openrouter"`)
  3. `x-session-affinity` header
- When a session id is present, `cacheControl: { type: 'ephemeral', ttl:
'5m' }` is set, matching the Agent Builder main loop.
`prompt_cache_retention: "24h"` maps to the longest EIS TTL (`1h`).
- Usage now includes `prompt_tokens_details.cached_tokens` whenever
inference reports cached tokens, so clients like ai-sdk and pi can show
cache reads.
- Session ids are bounded to 256 chars (validated for the body field;
oversized header values are ignored).
- **Request body limit raised to 20MB** for `/chat/completions` (from
the 1MB `server.maxPayload` default). Long agent conversations with
large tool results or base64 images quickly go over 1MB, and every turn
resends the full history.
- **Message limits raised**: up to 10,000 messages per request (was
1,000) and 2,000 content parts per message (was 100). Each tool call
adds an assistant message and a tool message, so long agent sessions hit
1,000 well before the context window fills. The total payload is still
capped by the 20MB body limit.
- README documents prompt caching and shows a verified pi `models.json`
setup.

Companion Elastic Ramen PR: elastic/elastic-ramen#122

### Verification (local stack with EIS)

Checked through Phoenix traces (`elastic.cache_control.*` span
attributes):
- `prompt_cache_key` in the body → `session_id` + `ttl: 5m` ✅
- `x-session-id` header → `session_id` + `ttl: 5m` ✅
- pi (`openai-completions` + compat config from README) → every turn of
a pi session carries the pi session id ✅. With `PI_CACHE_RETENTION=long`
→ `ttl: 1h` ✅

Body limit: checked live. A ~5MB request returns `200`; a ~21MB request
returns `413 Payload content length greater than maximum allowed:
20971520`.

**Prompt cache hits (Claude Sonnet 5 on EIS):** I sent the same
~422k-token request 3 times with one `prompt_cache_key`:

| Request | Time | `cached_tokens` |
|---|---|---|
| #1 | 8.3s | – (cache written) |
| #2 | 2.1s | 422,347 / 422,349 |
| elastic#3 | 2.0s | 422,347 / 422,349 |

So the session id reaches EIS, EIS caches the prompt, and the new
`prompt_tokens_details.cached_tokens` field reports the hit. Note:
Claude Sonnet 4.5 on EIS doesn't report cached tokens, so no hits show
up for that model.

**Large inputs** (one-word answer hidden in the middle of the input,
sent through the route):

| Model | Prompt tokens | Result |
|---|---|---|
| Claude Sonnet 5 | 422k / 814k | ✅ / ✅ |
| Gemini 2.5 Pro | 388k / 752k | ✅ / ✅ |
| GPT-5.4 | 291k | ✅ |
| Claude Sonnet 4.5 | 124k | ✅ (EIS context window looks like 200k) |

**Long conversations:** a request with 5,001 messages returns `200`.
That was over the old 1,000-message limit.

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials

## Release Notes

N/A (experimental feature)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@timductive