diff --git a/README.md b/README.md index 1713cef..71966ca 100644 --- a/README.md +++ b/README.md @@ -122,6 +122,7 @@ from the very first prompt — no priming, no setup. - **Vite HMR** — instant hot reload in development - **Tailwind** — preconfigured out of the box - **Production build** — `pyxle build` + `pyxle serve`; deploy anywhere Python runs +- **AI accessibility** — serve any page as clean Markdown (append `.md`, or `Accept: text/markdown`) plus an `llms.txt` index, so AI agents read your app as text — one flag, agent-friendly out of the box - **Editor tooling** — LSP, linter, and a [VS Code extension](https://marketplace.visualstudio.com/items?itemName=pyxle.pyxle-language-tools) ## Status diff --git a/docs/changelog.md b/docs/changelog.md index 739bf8b..9ac9ddc 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -2,6 +2,10 @@ Release notes for Pyxle. While we're in beta (`0.x`), minor versions may include breaking changes — those are called out explicitly here. To upgrade, run `pip install --upgrade pyxle-framework`. +## 0.6.0 — 2026-07-01 + +- **AI accessibility — serve your app as markdown, plus `llms.txt`.** A new opt-in `llms` config block makes every Pyxle app legible to AI assistants and coding agents. Enable it (`"llms": true`) and the framework serves a clean **markdown** rendition of each page at its URL with `.md` appended (`/docs/routing.md`), returns markdown from the same URL to requests that send `Accept: text/markdown` (browsers are unaffected), serves an **`/llms.txt`** index, and advertises it with `Link`/`X-Llms-Txt` discovery headers. A page's markdown is resolved from your project — a co-located `.md` file, a `to_markdown` handler in the page's server module, or a `to_markdown` in the nearest ancestor `llms.py` (which scopes to a whole route subtree; `pages/llms.py` is app-wide) — with an opt-in `autoConvert` HTML→markdown fallback and a redirect-to-the-page fallback when nothing resolves. `/llms.txt` comes from a static `public/llms.txt`, a `llms_txt` hook in `pages/llms.py`, or a generated index, and a `wrap_markdown` hook in `pages/llms.py` can frame every `.md` response with a header/footer (e.g. agent navigation hints). Off by default; adds nothing to the page hot path. See the [AI accessibility guide](guides/llms.md) and [Configuration → AI accessibility](reference/configuration.md#ai-accessibility). + ## 0.5.0 — 2026-06-18 The depth release: caching/SSG/ISR, streaming SSR, realtime (WebSockets + a diff --git a/docs/guides/for-ai-agents.md b/docs/guides/for-ai-agents.md index ba1af9d..00ef4b6 100644 --- a/docs/guides/for-ai-agents.md +++ b/docs/guides/for-ai-agents.md @@ -633,6 +633,20 @@ instead of scanning the full source tree. --- +## Serve your whole app as Markdown + +There's a second half to "agent-first": not just *writing* code an agent can reason about, but *serving* content an agent can read. Pyxle ships that too. Enable the [`llms`](llms.md) feature and every page of your app is available as clean Markdown — append `.md` to any URL, or send `Accept: text/markdown` — plus an [`/llms.txt`](https://llmstxt.org/) index and discovery headers: + +```json +{ "llms": true } +``` + +Now a developer can paste `yourapp.com/docs/anything.md` straight into Claude, Cursor, or ChatGPT and get clean, token-efficient context instead of HTML. You decide where each page's Markdown comes from — a co-located `.md` file, a `to_markdown` handler, or a directory-wide `llms.py` — and a `wrap_markdown` hook can frame every page with instructions for the agent reading it. This very handbook is served that way: append `.md` to this page's URL to see it. + +See the [AI accessibility guide](llms.md) for the full model. + +--- + ## Next steps - **New to Pyxle?** Start with the [quick start](../getting-started/quick-start.md). diff --git a/docs/guides/llms.md b/docs/guides/llms.md new file mode 100644 index 0000000..9c655f1 --- /dev/null +++ b/docs/guides/llms.md @@ -0,0 +1,341 @@ +# AI accessibility — your app in Markdown + +Every page of a Pyxle app can serve a clean **Markdown** rendition of itself — so AI assistants and coding agents (Claude, ChatGPT, Cursor, Copilot, Perplexity) read your app as text instead of scraping HTML. Turn it on with one flag; decide where the Markdown comes from with a few conventions in your project. + +```json +{ "llms": true } +``` + +That single line gives you: + +- **Per-page Markdown** at each URL with `.md` appended — `/docs/routing` → `/docs/routing.md`. +- **Content negotiation** — the *same* URL returns Markdown to any request that sends `Accept: text/markdown`. Browsers never send it, so humans are unaffected. +- **An `/llms.txt` index** — the [llms.txt](https://llmstxt.org/) convention: a Markdown map of your site. +- **Discovery headers** on every response — `Link: ; rel="llms-txt"` and `X-Llms-Txt: /llms.txt` — so an agent finds the index without parsing HTML. + +This very page proves it: append `.md` to its URL to read the Markdown you're looking at. + +--- + +## The mental model + +There are two moving parts, and they're cleanly separated: + +1. **Routing is the framework's job.** When `llms` is enabled, Pyxle registers a `.md` route for every page, wires up `Accept` negotiation, serves `/llms.txt`, and adds the discovery headers. You don't configure any of that. +2. **Content is your job — expressed as files, not config.** *Where* a page's Markdown comes from is resolved from your project on each request. The `llms` config block is just an on/off switch (plus one opt-in fallback); everything substantive lives in `.md` files and `llms.py` handlers next to your routes. + +The feature is **off by default** and adds nothing to the normal page render path — the `.md` routes are separate and only run for `.md` (or `Accept: text/markdown`) requests. + +--- + +## Enabling it + +In `pyxle.config.json`: + +```json +{ + "llms": { + "enabled": true, + "autoConvert": false + } +} +``` + +| Key | Default | What it does | +|-----|---------|--------------| +| `enabled` | `false` | Turns the whole feature on. `"llms": true` is shorthand for `{ "enabled": true }`. | +| `autoConvert` | `false` | A last-resort fallback: convert a page's rendered HTML to Markdown when no authored source exists. Off by default because it's lossy — see [autoConvert](#autoconvert-the-lossy-fallback). | + +That is the entire configuration surface. See [Configuration → AI accessibility](../reference/configuration.md#ai-accessibility). + +--- + +## How a page's Markdown is resolved + +For any `.md` request (or an `Accept: text/markdown` request to the page), Pyxle walks this ladder and uses the **first source that returns text**: + +| # | Source | Scope | Best for | +|---|--------|-------|----------| +| 1 | Co-located `.md` file | one page | static, hand-written pages | +| 2 | `to_markdown` in the page's own module | one page (a catch-all covers its subtree) | pages that already load their content | +| 3 | `to_markdown` in the nearest ancestor `llms.py` | a route subtree (`pages/llms.py` = app-wide) | one handler for many pages | +| 4 | `autoConvert` (only if enabled) | any page | a rough fallback when nothing else exists | +| 5 | **Redirect** `/.md` → `/` | — | so a guessed `.md` URL never 404s | + +Whatever a rung returns is then passed through the optional [`wrap_markdown`](#framing-every-page-wrap_markdown) hook before it's sent. + +### 1. A co-located `.md` file + +Drop a Markdown file next to the page. Simplest possible option — no code: + +``` +pages/ + about.pyxl + about.md ← served at /about.md + index.pyxl + index.md ← served at /index.md (i.e. the / page) +``` + +Best for static, content-heavy pages you'd rather write by hand than generate — a landing page, a manifesto, a pricing page. + +### 2. A page-local `to_markdown` handler + +Add a `to_markdown` function to a page's Python (server) section. It receives a [`MarkdownContext`](#the-markdowncontext-ctx) and returns a Markdown string — or `None` to defer to the next rung: + +```python +# pages/products/[id].pyxl + +@server +async def load(request): + return {"product": await get_product(request.path_params["id"])} + +async def to_markdown(ctx): + product = await get_product(ctx.request.path_params["id"]) + return f"# {product.name}\n\n{product.description}\n" +``` + +Because a **catch-all page** (`[[...slug]].pyxl`) is a single page that handles every sub-path, its `to_markdown` already covers the whole subtree — read `ctx.request.path_params["slug"]` to know which page was asked for. + +### 3. A directory `llms.py` handler (covers a route subtree) + +To serve many pages under a directory with **one** handler, put a `to_markdown` in an `llms.py` at that directory. Resolution walks **from the page's own directory up to `pages/`, nearest ancestor first** — exactly like `layout.pyxl`, `error.pyxl`, and `loading.pyxl`: + +``` +pages/ + llms.py ← app-wide: to_markdown for any page below + docs/ + llms.py ← handles everything under /docs + intro.pyxl + routing.pyxl +``` + +```python +# pages/docs/llms.py — one handler for the whole /docs subtree +import json +from pathlib import Path + +DOCS = Path("public/docs-data") + +def to_markdown(ctx): + slug = ctx.path.removeprefix("/docs/") # e.g. "guides/routing" + page = DOCS / f"{slug}.json" + if not page.is_file(): + return None # decline → try a broader handler + return json.loads(page.read_text())["markdown"] +``` + +Returning `None` **declines** and defers to the next ancestor (and ultimately to `autoConvert`/redirect). So a `/docs` handler can answer the slugs it knows and let everything else fall through — handlers compose down the tree. `llms.py` is also the intended home for any future per-directory AI hooks (it already hosts [`llms_txt`](#the-llmstxt-index) and [`wrap_markdown`](#framing-every-page-wrap_markdown) at the root). + +### autoConvert (the lossy fallback) + +If nothing above resolves and you've set `"autoConvert": true`, Pyxle renders the page and converts its HTML to Markdown with a small, **dependency-free** converter. It's **off by default** and deliberately best-effort: headings, paragraphs, lists, links, emphasis, and code survive; layout chrome, tables, and rich components may not. Treat it as "something is better than a redirect" — prefer an authored `.md` or a handler for anything you care about. + +### The redirect fallback + +With the feature on but no Markdown source and `autoConvert` off, `/.md` returns a **`307` redirect** to `/`. An agent that guesses a `.md` URL lands on the real page instead of a 404. + +--- + +## The `MarkdownContext` (`ctx`) + +Every `to_markdown` and `wrap_markdown` handler receives a single argument — a `MarkdownContext`. It's a small, read-only object with everything you need to produce a page's Markdown: + +| Member | Type | Description | +|--------|------|-------------| +| `ctx.request` | `starlette.requests.Request` | The incoming request. Use it for route params, query string, headers, and the body. | +| `ctx.path` | `str` | The **canonical page path**, always without `.md` — e.g. `/docs/routing`, or `/` for the home page. Use this, not `request.url.path`. | +| `await ctx.run_loader()` | `Any` | Runs *only* the page's `@server` loader and returns its data — the dict the page would receive — skipping the render. The cheap path when you just want the loaded data. Returns `{}` for a page with no loader. | +| `await ctx.render_html()` | `str` | Renders the *original* page — running its `@server` loader and full SSR — and returns the **body HTML** (the component output, without the document shell). Lazy: nothing renders unless you call it. | + +### `ctx.request` — the request object + +A standard Starlette [`Request`](https://www.starlette.io/requests/). The members you'll actually reach for: + +- **`ctx.request.path_params`** — the matched route parameters. For `pages/docs/[[...slug]].pyxl`, `ctx.request.path_params["slug"]` is `"guides/routing"`. This is how one handler serves many pages. +- **`ctx.request.query_params`** — the query string (`?q=…`), a multidict. +- **`ctx.request.headers`** — request headers. +- **`await ctx.request.body()`** / **`await ctx.request.json()`** — the request body, if any. + +### `ctx.path` vs `ctx.request.url.path` — an important distinction + +Use **`ctx.path`**. It is always the canonical page path with no `.md` suffix, whether the request arrived as `/docs/routing.md` or as `/docs/routing` with `Accept: text/markdown`. In contrast, `ctx.request.url.path` carries the *raw* request path — which **includes** `.md` on a `.md` request and **omits** it on an `Accept`-negotiated one. Reading `ctx.path` means your handler behaves identically on both entry points. + +```python +def to_markdown(ctx): + # ctx.path -> "/docs/routing" (always canonical) + # ctx.request.url.path -> "/docs/routing.md" OR "/docs/routing" + slug = ctx.path.removeprefix("/docs/") + ... +``` + +### `ctx.render_html()` — post-processing the rendered page + +When you want to *derive* Markdown from what the page actually renders — rather than from source data — call `await ctx.render_html()`. It runs the page's loader and server render and hands you the body HTML, which you can transform: + +```python +from pyxle.devserver.llms import html_to_markdown # the built-in converter + +async def to_markdown(ctx): + html = await ctx.render_html() + return html_to_markdown(html) # roughly what autoConvert does, but on your terms +``` + +It's lazy and potentially expensive (a full SSR pass), so only call it when you need it. When you want the page's *data* rather than its rendered HTML, reach for `ctx.run_loader()` instead — it's much cheaper. + +### `ctx.run_loader()` — the loader's data, without the render + +Often you don't want rendered HTML at all — you want the same data the page loads, to format as Markdown yourself. `await ctx.run_loader()` runs just the page's `@server` loader and returns its result (the dict the page would receive as `data`), skipping SSR entirely: + +```python +async def to_markdown(ctx): + data = await ctx.run_loader() # runs the @server loader, no render + post = data["post"] + return f"# {post['title']}\n\n{post['body']}\n" +``` + +This is the cheap path — a loader call, not a full render — and it reuses the exact data-loading your page already does. A page with no loader returns `{}`. + +### Handler contract + +- **Sync or async** — both work. An async handler is awaited. +- **Return `str`** to serve that Markdown. +- **Return `None`** to decline and fall through to the next rung. +- Returning anything else raises a `TypeError` (surfaced in logs); the `.md` request degrades gracefully to a redirect, and an `Accept`-negotiated request falls back to HTML. + +--- + +## Framing every page (`wrap_markdown`) + +To add a consistent **header/footer to every `.md` response** — agent instructions, navigation hints, a canonical-URL banner — define a `wrap_markdown(ctx, markdown)` function in the **root** `pages/llms.py`. Pyxle calls it with the `MarkdownContext` and the already-resolved Markdown, and serves whatever string it returns: + +```python +# pages/llms.py +BASE = "https://example.com" + +def wrap_markdown(ctx, markdown): + header = ( + f"> Markdown rendition of {BASE}{ctx.path}, served for AI agents.\n" + f"> Append `.md` to any URL for its Markdown. Index: {BASE}/llms.txt\n" + ) + return f"{header}\n{markdown}" +``` + +Because it runs on Markdown from *every* source (co-located files, `to_markdown` handlers, `autoConvert`), the framing is defined **once** and applied everywhere. Return `None` to leave the Markdown untouched. And because it's applied at serve time — not baked into your source — your `/llms-full.txt` corpus and the raw `.md` files stay clean. + +--- + +## The `/llms.txt` index + +[`/llms.txt`](https://llmstxt.org/) is a Markdown map of your site that agents (and humans) can read to discover what's available. Pyxle resolves it, first hit wins: + +1. a **static `public/llms.txt`** (served by the static-asset layer before anything else) — full manual control; +2. a **`llms_txt`** function in the root `pages/llms.py` — generate it dynamically; +3. a **generated** default: an `H1` plus a `## Pages` list linking every concrete (non-parameterised) page's `.md`. + +For a site with dynamic content — docs, a blog, a catalog — the generated default can't enumerate your dynamic routes, so provide a `llms_txt` hook: + +```python +# pages/llms.py +def llms_txt(ctx): + lines = ["# My App", "", "> One-line summary of the app.", "", "## Docs", ""] + for slug, title in load_doc_index(): + lines.append(f"- [{title}](https://example.com/docs/{slug}.md): short description") + return "\n".join(lines) + "\n" +``` + +The hook receives an **`LlmsTxtContext`**: + +| Member | Type | Description | +|--------|------|-------------| +| `ctx.request` | `Request` | The incoming request. | +| `ctx.pages` | `tuple[LlmsPageInfo, ...]` | Your app's concrete pages (see below). | +| `ctx.render_default()` | `str` | The framework's generated index — return it verbatim, extend it, or ignore it. | + +Each entry in `ctx.pages` is an **`LlmsPageInfo`** — `path` (e.g. `/about`), `md_url` (`/about.md`), and `title` (a humanized label). Return a string, or `None` to fall back to the generated default. + +> **`llms.txt` vs `llms-full.txt`.** `/llms.txt` is a *map* (links + descriptions) an agent reads to decide what to fetch. A `/llms-full.txt` is the *whole corpus* concatenated into one file for one-shot ingestion. Pyxle generates `/llms.txt`; if you want `/llms-full.txt`, produce it in your build (Pyxle serves it as a static file). Both are complementary — the index for discovery, the full file for bulk reading, the per-page `.md` for precise pulls. + +--- + +## Content negotiation and discovery headers + +Beyond the `.md` URLs, two things make the feature work for agents that don't append `.md`: + +- **`Accept: text/markdown` negotiation.** A request to the *canonical* URL (`/docs/routing`) that includes `text/markdown` in its `Accept` header gets the Markdown, resolved through the exact same ladder. Browsers never send that header, so this is invisible to human visitors. The response carries `Vary: Accept` so shared caches key on it correctly. +- **Discovery headers.** Every response advertises the index: `Link: ; rel="llms-txt"` and `X-Llms-Txt: /llms.txt`. An agent can find your `llms.txt` from any page without parsing the body. + +--- + +## Deployment + +- A page's **own `to_markdown`** is compiled into your build and works anywhere `pyxle serve` runs. +- **Co-located `.md` files and `llms.py` handlers are source files.** They must be present alongside `pages/` at runtime — which they are for the common case of deploying your whole project directory. If they're ever absent, resolution simply falls through to the next rung (and ultimately the redirect), so nothing breaks; the page just isn't available as Markdown. +- **Caching.** `.md` responses aren't run through the page edge-cache. If you serve heavy handlers under load, cache at your CDN/reverse proxy keyed on the path (and `Vary: Accept` for the negotiated route). + +--- + +## robots.txt and AI crawlers + +The `.md`/`llms.txt` endpoints *offer* clean content; they don't gate crawling. If you want reach, **don't block the AI bots** in `public/robots.txt` — being read and cited by them is free distribution: + +``` +User-agent: GPTBot +User-agent: OAI-SearchBot +User-agent: ChatGPT-User +User-agent: ClaudeBot +User-agent: Claude-SearchBot +User-agent: PerplexityBot +User-agent: CCBot +Allow: / + +Sitemap: https://example.com/sitemap.xml +``` + +Ship a normal XML sitemap alongside it. (OpenAI and Anthropic expose separate tokens for *training* vs *search* if you want to allow one and not the other — see their bot docs.) + +--- + +## Recipes + +**Serve docs from generated JSON** (as pyxle.dev does): one directory handler maps a slug to a pre-built Markdown field. See [directory handler](#3-a-directory-llmspy-handler-covers-a-route-subtree) above. + +**Rewrite links for portability.** Markdown that travels (pasted into a chat, saved to disk) should carry absolute links. Rewrite relative links to absolute `.md` URLs when you generate or store the Markdown, so `[Routing](../routing.md)` becomes `[Routing](https://example.com/docs/routing.md)`. + +**Add an agent search endpoint.** Agents can search by fetching `/llms-full.txt` and scanning it, but a dedicated endpoint is nicer. A plain `pages/api/*.py` route that ranks your content and returns Markdown links works well — then point to it from `wrap_markdown`: + +```python +# pages/api/search.py → GET /api/search?q=... +from starlette.responses import PlainTextResponse + +async def endpoint(request): + q = request.query_params.get("q", "") + hits = search_your_index(q) # your ranking + lines = [f"# Results for “{q}”", ""] + lines += [f"- [{h.title}]({h.md_url})" for h in hits] + return PlainTextResponse("\n".join(lines), media_type="text/markdown; charset=utf-8") +``` + +**Per-section handlers.** Give `pages/docs/llms.py` and `pages/blog/llms.py` different `to_markdown` handlers; each scopes to its subtree, and the root `pages/llms.py` catches anything else. + +--- + +## FAQ + +**Does this slow down my pages?** No. The `.md` routes are separate and only run for `.md`/`Accept: text/markdown` requests. The normal render path is untouched, and the whole feature is off unless you enable it. + +**Do I have to write Markdown for every page?** No. Enable the feature and pages with no source simply redirect their `.md` URL to the page. Add `.md` files or handlers only where clean Markdown is worth it (usually docs and content pages). + +**What about my interactive pages?** A dashboard or playground isn't meaningful as Markdown — either give it a short hand-written `.md` describing what it is, or let it redirect. `autoConvert` exists for a rough automatic version if you want one. + +**Is this the same as Mintlify's `.md`/llms.txt?** Same idea, but built into the framework and applied to your *whole app*, not just a hosted docs site — and you control exactly where each page's Markdown comes from. + +**Who reads all this?** Right now, the clearest win is direct: point any AI assistant — Claude, Cursor, ChatGPT — at a `.md` URL or your `llms.txt` and it gets clean, token-efficient context instead of scraped HTML. Beyond that, `.md` and [llms.txt](https://llmstxt.org/) are the conventions the AI ecosystem is standardizing on — so your app already speaks the format machine readers are moving toward, served to spec (per-page Markdown, `Accept` negotiation, discovery headers) with nothing more to do as adoption grows. + +--- + +## See also + +- [Configuration → AI accessibility](../reference/configuration.md#ai-accessibility) — the config block. +- [Runtime API → AI accessibility hooks](../reference/runtime-api.md#ai-accessibility-hooks) — `to_markdown`, `wrap_markdown`, `llms_txt`, and the context objects. +- [Pyxle for AI coding agents](for-ai-agents.md) — the broader case for Pyxle in the AI era. +- The [llms.txt specification](https://llmstxt.org/). diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 1e8c944..92ac4f3 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -66,6 +66,10 @@ Pyxle is configured via `pyxle.config.json` in the project root. All fields are "otelServiceName": "pyxle-app", "otelSampleRatio": 0.05 }, + "llms": { + "enabled": false, + "autoConvert": false + }, "plugins": [] } ``` @@ -320,6 +324,29 @@ Request correlation IDs and timing. Both are **on by default** -- generating an Shorthand to disable request-id and timing: `"observability": false`. The page-cache hit-ratio metric requires the [server-side page cache](../guides/caching.md) (active under `pyxle serve`). **Multi-worker note:** under `pyxle serve --workers N` each worker process exposes its own metrics (with a `worker` label), so aggregate across workers at the scraper. +## AI accessibility + +Serve a clean **markdown** rendition of every page — at its URL with `.md` appended, and to requests that send `Accept: text/markdown` — plus an `/llms.txt` index, so AI assistants and coding agents can read your app as text instead of parsing HTML. **Off by default.** See the [AI accessibility guide](../guides/llms.md) for the full model. + +```json +{ + "llms": { + "enabled": true, + "autoConvert": false + } +} +``` + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `llms` | `object` \| `boolean` | `{}` | AI-accessibility settings. `true` enables with defaults; an object enables unless `enabled: false`. | +| `llms.enabled` | `boolean` | `false` | Turn the feature on: register `.md` routes, negotiate `Accept: text/markdown`, serve `/llms.txt`, and add `Link`/`X-Llms-Txt` discovery headers. | +| `llms.autoConvert` | `boolean` | `false` | Last-resort fallback: when a page has no markdown source, convert its rendered HTML to markdown. **Off by default** — the conversion is lossy; prefer author-provided markdown. | + +**Where the markdown comes from** is *not* config — it lives in your project, resolved in this order (first hit wins): a co-located `.md` file → a `to_markdown` handler in the page's server module → a `to_markdown` in the nearest ancestor `llms.py` (covers a route subtree; `pages/llms.py` is app-wide) → `autoConvert` (if on) → otherwise the `.md` URL redirects to the page. A root `pages/llms.py` may also define `wrap_markdown(ctx, markdown)` to frame every `.md` response with a header/footer. Likewise `/llms.txt` comes from a static `public/llms.txt`, else a `llms_txt` function in the root `pages/llms.py`, else a generated index. See the [guide](../guides/llms.md). + +> **Deploying handlers.** Co-located `.md` files and `llms.py` handlers are part of your source, so they must be present alongside `pages/` when you `pyxle serve`. (A page's own `to_markdown`, compiled into the build, works regardless.) + ## Plugins ### `plugins` diff --git a/docs/reference/runtime-api.md b/docs/reference/runtime-api.md index 3d45424..a5a77ac 100644 --- a/docs/reference/runtime-api.md +++ b/docs/reference/runtime-api.md @@ -301,6 +301,49 @@ Whether the upgrade's `Origin` is permitted. CSRF doesn't apply to a WebSocket, so checking the origin is the equivalent guard. An empty `allowed_origins` allows all; a missing `Origin` header (same-origin / non-browser) is allowed. +## AI accessibility hooks + +Conventions Pyxle recognizes when the [`llms`](configuration.md#ai-accessibility) feature is enabled, so your app can serve a Markdown rendition of each page. None are imported — they're functions you define by name in your `.pyxl` server sections or in `llms.py` files. See the [AI accessibility guide](../guides/llms.md) for the full model. + +### `to_markdown(ctx)` + +Returns the Markdown for a page. Define it in a page's server section (scopes to that page) or in an `llms.py` file (scopes to that directory's subtree; nearest ancestor wins). Sync or async. + +```python +def to_markdown(ctx): # ctx: MarkdownContext + return f"# {title}\n\n{body}\n" # str -> served + # return None # -> decline, fall through to the next source +``` + +### `llms_txt(ctx)` + +Generates `/llms.txt`. Define it in the **root** `pages/llms.py`. Sync or async. Return a string, or `None` to use the generated default. `ctx` is an `LlmsTxtContext`. + +### `wrap_markdown(ctx, markdown)` + +Frames every resolved `.md` response with a header/footer. Define it in the **root** `pages/llms.py`. Sync or async. Return the wrapped string, or `None` to leave `markdown` unchanged. `ctx` is a `MarkdownContext`. + +### Context objects + +**`MarkdownContext`** — passed to `to_markdown` and `wrap_markdown`: + +| Member | Type | Description | +|--------|------|-------------| +| `ctx.request` | `starlette.requests.Request` | The request. `ctx.request.path_params` holds route params (e.g. `slug`). | +| `ctx.path` | `str` | Canonical page path, always without `.md` (e.g. `/docs/routing`, `/`). Prefer this over `request.url.path`. | +| `await ctx.run_loader()` | `Any` | Runs only the page's `@server` loader and returns its data (no render). `{}` if the page has no loader. | +| `await ctx.render_html()` | `str` | Renders the page (loader + SSR) and returns the body HTML. Lazy; only call if you need it. | + +**`LlmsTxtContext`** — passed to `llms_txt`: + +| Member | Type | Description | +|--------|------|-------------| +| `ctx.request` | `Request` | The request. | +| `ctx.pages` | `tuple[LlmsPageInfo, ...]` | The app's concrete pages. | +| `ctx.render_default()` | `str` | The framework's generated `/llms.txt`. | + +**`LlmsPageInfo`** — each entry in `ctx.pages`: `path` (e.g. `/about`), `md_url` (`/about.md`), `title` (humanized label). + ## Document `` elements Pyxle offers two ways to contribute elements to the document ``: the `` component (**recommended**) and the `HEAD` Python variable (lower-level alternative). Both are merged with automatic deduplication. diff --git a/pyproject.toml b/pyproject.toml index da202c3..dcb2b59 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "pyxle-framework" -version = "0.5.0" +version = "0.6.0" description = "Python-first full-stack framework with an intelligent CLI." readme = "README.md" authors = [ diff --git a/pyxle/config.py b/pyxle/config.py index f3627d0..2cb3743 100644 --- a/pyxle/config.py +++ b/pyxle/config.py @@ -162,6 +162,39 @@ def enabled(self) -> bool: return self.request_id or self.timing or self.metrics_endpoint or self.access_log +@dataclass(frozen=True, slots=True) +class LlmsConfig: + """AI-accessibility settings: per-page markdown and an ``llms.txt`` index. + + When ``enabled``, the framework serves a markdown rendition of each page at + the page's URL with ``.md`` appended (and when a request sends + ``Accept: text/markdown``), advertises the index via ``Link``/``X-Llms-Txt`` + discovery headers, and serves ``/llms.txt``. Everything here is **off by + default** — an opt-in feature for making an app legible to AI assistants and + coding agents. + + A page's markdown is resolved in this order, first hit wins: + + 1. A co-located ``.md`` file next to the ``.pyxl`` source. + 2. A ``to_markdown`` handler in the page's own server module (a function + ``fn(ctx) -> str | None``, sync or async). + 3. A ``to_markdown`` in the nearest ancestor ``llms.py`` — a per-directory + module covering a whole route subtree (closest ancestor wins, like + ``layout.pyxl``); ``pages/llms.py`` is the app-wide handler. + 4. Only if ``auto_convert`` is on: a best-effort HTML→markdown conversion of + the rendered page. + 5. If none apply: the ``.md`` URL redirects to the page itself. + + ``/llms.txt`` is served from a static ``public/llms.txt`` if present, else + from a ``llms_txt`` function in the root ``pages/llms.py``, else from a + generated index of the app's pages. ``auto_convert`` defaults off because the + conversion is lossy — prefer author-provided markdown or a handler. + """ + + enabled: bool = False + auto_convert: bool = False + + @dataclass(frozen=True, slots=True) class PyxleConfig: """Resolved configuration values for a Pyxle project.""" @@ -186,6 +219,7 @@ class PyxleConfig: navigation: NavigationConfig = NavigationConfig() rate_limit: RateLimitConfig = RateLimitConfig() observability: ObservabilityConfig = ObservabilityConfig() + llms: LlmsConfig = LlmsConfig() # Plugin entries as the raw payload from ``pyxle.config.json`` — # either a bare string (``"pyxle-auth"``) or an object # (``{"name": "pyxle-auth", "settings": {...}}``). Resolved into @@ -216,6 +250,7 @@ def to_devserver_kwargs(self) -> Dict[str, Any]: "navigation": self.navigation, "rate_limit": self.rate_limit, "observability": self.observability, + "llms": self.llms, "plugins": self.plugins, } @@ -324,6 +359,7 @@ def _parse_config_dict(data: Dict[str, Any], *, source: Path) -> PyxleConfig: "navigation", "rateLimit", "observability", + "llms", "plugins", } unknown_keys = set(data) - allowed_top_keys @@ -359,6 +395,7 @@ def _parse_config_dict(data: Dict[str, Any], *, source: Path) -> PyxleConfig: navigation_config = _parse_navigation_block(data.get("navigation"), source=source) rate_limit_config = _parse_rate_limit_block(data.get("rateLimit"), source=source) observability_config = _parse_observability_block(data.get("observability"), source=source) + llms_config = _parse_llms_block(data.get("llms"), source=source) plugins = _parse_plugins_block(data.get("plugins"), source=source) return PyxleConfig( @@ -382,6 +419,7 @@ def _parse_config_dict(data: Dict[str, Any], *, source: Path) -> PyxleConfig: navigation=navigation_config, rate_limit=rate_limit_config, observability=observability_config, + llms=llms_config, plugins=plugins, ) @@ -424,6 +462,48 @@ def _parse_plugins_block(value: Any, *, source: Path) -> tuple[Any, ...]: return tuple(entries) +def _parse_llms_block(value: Any, *, source: Path) -> LlmsConfig: + """Parse the ``llms`` block — AI/markdown accessibility settings. + + Accepts a boolean shorthand (``"llms": true`` enables the feature with + defaults) or an object with ``enabled`` and ``autoConvert`` keys. When the + object form is used, the feature is enabled unless ``enabled: false`` is set + explicitly. Markdown handlers and the ``llms.txt`` index live in ``llms.py`` + files, not in config. + """ + if value is None: + return LlmsConfig() + if isinstance(value, bool): + return LlmsConfig(enabled=value) + if not isinstance(value, Mapping): + raise ConfigError( + f"Invalid value for 'llms' in '{source}': expected an object or boolean." + ) + + _reject_unknown_keys( + value, + allowed={"enabled", "autoConvert"}, + block="llms", + source=source, + ) + + enabled = value.get("enabled", True) + if not isinstance(enabled, bool): + raise ConfigError( + f"Invalid 'llms.enabled' in '{source}': expected boolean, " + f"got {type(enabled).__name__}." + ) + + auto_convert = value.get("autoConvert", False) + if not isinstance(auto_convert, bool): + raise ConfigError( + f"Invalid 'llms.autoConvert' in '{source}': expected boolean, " + f"got {type(auto_convert).__name__}." + ) + + return LlmsConfig(enabled=enabled, auto_convert=auto_convert) + + def _parse_styling_block(value: Any, *, source: Path) -> tuple[tuple[str, ...], tuple[str, ...]]: if value is None: return ((), ()) diff --git a/pyxle/devserver/llms.py b/pyxle/devserver/llms.py new file mode 100644 index 0000000..7c1e578 --- /dev/null +++ b/pyxle/devserver/llms.py @@ -0,0 +1,765 @@ +"""AI accessibility: per-page markdown responses and an ``/llms.txt`` index. + +Enabled with the ``llms`` block in ``pyxle.config.json`` (see +:class:`pyxle.config.LlmsConfig`). When on, the framework serves a markdown +rendition of each page at its URL with ``.md`` appended — and to requests that +send ``Accept: text/markdown`` — advertises the index via ``Link`` / +``X-Llms-Txt`` discovery headers, and serves a generated ``/llms.txt`` +(overridable by a static ``public/llms.txt``, which the static-asset middleware +serves first). + +A page's markdown is resolved in order, first hit wins: + +1. a co-located ``.md`` file next to the ``.pyxl`` source, +2. a ``to_markdown`` handler in the page's own server module, +3. a ``to_markdown`` in the nearest ancestor ``llms.py`` — a per-directory + module that covers a whole route subtree (closest ancestor wins, like + ``layout.pyxl``); ``pages/llms.py`` at the root is the app-wide handler, +4. only if ``auto_convert`` is on, a best-effort HTML→markdown conversion, +5. otherwise the ``.md`` URL redirects to the page itself. + +``/llms.txt`` is served from a static ``public/llms.txt``, else a ``llms_txt`` +function in the root ``pages/llms.py``, else a generated index of the app's +pages. A root ``pages/llms.py`` may also define ``wrap_markdown(ctx, markdown)`` +to frame every ``.md`` response with a header/footer (e.g. agent navigation and +search hints). + +The whole feature is off by default and adds nothing to the page hot path — the +markdown routes are separate Starlette routes hit only for ``.md`` URLs. +""" + +from __future__ import annotations + +import asyncio +import inspect +import logging +from dataclasses import dataclass +from html.parser import HTMLParser +from pathlib import Path +from typing import Any, Awaitable, Callable, Optional + +from starlette.datastructures import MutableHeaders +from starlette.requests import Request +from starlette.responses import PlainTextResponse, RedirectResponse, Response +from starlette.routing import Route + +from .routes import PageRoute, RouteTable + +logger = logging.getLogger("pyxle.devserver.llms") + +#: Media type used for every markdown response the feature emits. +MARKDOWN_MEDIA_TYPE = "text/markdown; charset=utf-8" + +#: Conventional name of a page's / directory's markdown handler. +LOCAL_HANDLER_NAME = "to_markdown" + +#: Conventional name of the ``/llms.txt`` generator in the root ``pages/llms.py``. +LLMS_TXT_HOOK_NAME = "llms_txt" + +#: Conventional name of the markdown wrapper (header/footer) in root ``pages/llms.py``. +WRAP_HOOK_NAME = "wrap_markdown" + +#: Well-known path for the index. +LLMS_TXT_PATH = "/llms.txt" + + +def is_enabled(config: Any) -> bool: + """Return ``True`` when a resolved ``LlmsConfig`` has the feature enabled.""" + return bool(config is not None and getattr(config, "enabled", False)) + + +# --------------------------------------------------------------------------- +# Handler context +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class MarkdownContext: + """Context passed to a ``to_markdown`` or global markdown handler. + + ``request`` is the incoming Starlette request (a ``.md`` URL or an + ``Accept: text/markdown`` request). ``path`` is the canonical page path the + markdown represents (without the ``.md`` suffix). ``render_html`` renders the + original page — running its loader and SSR — and returns the body HTML; + ``run_loader`` runs only the page's ``@server`` loader and returns its data, + skipping the render. Call whichever you need — both are lazy. + """ + + request: Request + path: str + _render_html: Callable[[], Awaitable[str]] + _run_loader: Callable[[], Awaitable[Any]] + + async def render_html(self) -> str: + """Render the original page and return its body HTML.""" + return await self._render_html() + + async def run_loader(self) -> Any: + """Run the page's ``@server`` loader and return its data (no render). + + A cheaper alternative to :meth:`render_html` when you only need the + loader's return value. Returns ``{}`` for a page with no loader. + """ + return await self._run_loader() + + +async def _call_handler(handler: Callable[..., Any], ctx: MarkdownContext) -> Optional[str]: + """Invoke a markdown handler (sync or async) and validate its result.""" + result = handler(ctx) + if inspect.isawaitable(result): + result = await result + if result is None: + return None + if not isinstance(result, str): + raise TypeError( + f"Markdown handler {getattr(handler, '__qualname__', handler)!r} must " + f"return a string or None, got {type(result).__name__}." + ) + return result + + +# --------------------------------------------------------------------------- +# Resolution ladder +# --------------------------------------------------------------------------- + + +def colocated_markdown_path(page: PageRoute) -> Path: + """Return the co-located ``.md`` path next to a page's ``.pyxl`` source.""" + return page.source_absolute_path.with_suffix(".md") + + +def _read_text_if_file(path: Path) -> Optional[str]: + try: + if path.is_file(): + return path.read_text(encoding="utf-8") + except OSError: + return None + return None + + +async def _read_colocated_markdown(page: PageRoute) -> Optional[str]: + return await asyncio.to_thread(_read_text_if_file, colocated_markdown_path(page)) + + +def _load_local_handler(page: PageRoute, *, debug: bool) -> Optional[Callable[..., Any]]: + """Return the page's ``to_markdown`` server-module function, if defined.""" + from pyxle.ssr.view import _import_server_module # lazy: avoid import cycle + + try: + module = _import_server_module(page.module_key, page.server_module_path, debug=debug) + except Exception: # pragma: no cover - defensive; missing module falls through + logger.debug("Could not import server module for %s", page.path, exc_info=True) + return None + handler = getattr(module, LOCAL_HANDLER_NAME, None) + return handler if callable(handler) else None + + +#: Conventional per-directory module hosting AI hooks (currently ``to_markdown``) +#: for a whole route subtree. The nearest ancestor to the page wins. +DIRECTORY_MODULE_NAME = "llms.py" + + +def _directory_handler_candidates(page: PageRoute, settings: Any): + """Yield ``(llms.py path, module_key)`` from the page's dir up to the root.""" + pages_dir = Path(getattr(settings, "pages_dir")) + parts = page.source_relative_path.parent.parts + for depth in range(len(parts), -1, -1): + sub_parts = parts[:depth] + directory = pages_dir.joinpath(*sub_parts) if sub_parts else pages_dir + tag = "_".join(sub_parts) or "root" + yield directory / DIRECTORY_MODULE_NAME, f"_pyxle_llms_{tag}" + + +def _iter_directory_handlers(page: PageRoute, settings: Any, *, debug: bool): + """Yield ``llms.py`` ``to_markdown`` handlers, nearest ancestor first. + + Walks from the page's own directory up to ``pages/``. Each handler may + decline (return ``None``) to defer to a broader ancestor, so more than one + can participate — the closest one that returns a string wins. + """ + from pyxle.ssr.view import _import_server_module # lazy: avoid import cycle + + for path, module_key in _directory_handler_candidates(page, settings): + if not path.is_file(): + continue + try: + module = _import_server_module(module_key, path, debug=debug) + except Exception: # pragma: no cover - defensive; a broken handler module + logger.exception("Failed to import directory markdown handler %s", path) + continue + handler = getattr(module, LOCAL_HANDLER_NAME, None) + if callable(handler): + yield handler + + +def _load_root_llms_attr(settings: Any, attr: str, *, debug: bool) -> Optional[Callable[..., Any]]: + """Return a callable named ``attr`` from the root ``pages/llms.py``, if any.""" + from pyxle.ssr.view import _import_server_module # lazy: avoid import cycle + + path = Path(getattr(settings, "pages_dir")) / DIRECTORY_MODULE_NAME + if not path.is_file(): + return None + try: + module = _import_server_module("_pyxle_llms_root", path, debug=debug) + except Exception: # pragma: no cover - defensive; a broken module + logger.exception("Failed to import root %s", path) + return None + fn = getattr(module, attr, None) + return fn if callable(fn) else None + + +def strip_md_suffix(path: str) -> str: + """Map a ``.md`` request path back to its canonical page path. + + ``/about.md`` -> ``/about``; ``/index.md`` (the root) -> ``/``. + """ + base = path[:-3] if path.endswith(".md") else path + if base == "/index": + return "/" + if base.endswith("/index"): + base = base[: -len("index")] # keep the trailing slash + return base or "/" + + +async def resolve_page_markdown( + *, + request: Request, + page: PageRoute, + settings: Any, + renderer: Any, + config: Any, +) -> Optional[str]: + """Resolve a page's markdown, then apply the optional wrap hook. + + Returns the markdown string, or ``None`` when no source resolves (the caller + then redirects the ``.md`` URL to the page itself). + """ + debug = bool(getattr(settings, "debug", False)) + + async def _render() -> str: + from pyxle.ssr.view import render_page_body_html # lazy: avoid import cycle + + body_html, _status = await render_page_body_html( + request=request, settings=settings, page=page, renderer=renderer + ) + return body_html + + async def _loader() -> Any: + from pyxle.ssr.view import run_page_loader # lazy: avoid import cycle + + return await run_page_loader(request=request, settings=settings, page=page) + + ctx = MarkdownContext( + request=request, + path=strip_md_suffix(request.url.path), + _render_html=_render, + _run_loader=_loader, + ) + + markdown = await _resolve_source_markdown( + ctx, page=page, settings=settings, config=config, debug=debug + ) + if markdown is None: + return None + + wrapped = await _apply_wrap_hook(markdown, ctx, settings, debug=debug) + return wrapped if wrapped is not None else markdown + + +async def _resolve_source_markdown( + ctx: MarkdownContext, + *, + page: PageRoute, + settings: Any, + config: Any, + debug: bool, +) -> Optional[str]: + """Run the markdown resolution ladder (no wrap). First hit wins, else None.""" + # 1. Co-located .md + colocated = await _read_colocated_markdown(page) + if colocated is not None: + return colocated + + # 2. Page-local `to_markdown` in the page's own server module + local = _load_local_handler(page, debug=debug) + if local is not None: + result = await _call_handler(local, ctx) + if result is not None: + return result + + # 3. Ancestor `llms.py` handlers (nearest first), each covering a route + # subtree; a handler may return None to defer to a broader ancestor. + for handler in _iter_directory_handlers(page, settings, debug=debug): + result = await _call_handler(handler, ctx) + if result is not None: + return result + + # 4. Best-effort HTML -> markdown (opt-in only) + if getattr(config, "auto_convert", False): + return html_to_markdown(await ctx.render_html()) + + # 5. Nothing resolved. + return None + + +async def _apply_wrap_hook( + markdown: str, ctx: MarkdownContext, settings: Any, *, debug: bool +) -> Optional[str]: + """Apply the root ``pages/llms.py`` ``wrap_markdown`` hook, if defined. + + Lets an app frame every ``.md`` response with a header/footer (e.g. agent + navigation/search hints). Returns the wrapped markdown, or ``None`` to keep + the resolved markdown unchanged. + """ + hook = _load_root_llms_attr(settings, WRAP_HOOK_NAME, debug=debug) + if hook is None: + return None + result = hook(ctx, markdown) + if inspect.isawaitable(result): + result = await result + if result is None: + return None + if not isinstance(result, str): + raise TypeError( + f"{WRAP_HOOK_NAME} must return a string or None, got {type(result).__name__}." + ) + return result + + +# --------------------------------------------------------------------------- +# Best-effort HTML -> markdown (dependency-free, opt-in via auto_convert) +# --------------------------------------------------------------------------- + +_SKIP_TAGS = frozenset({"script", "style", "head", "noscript", "svg", "template"}) +_HEADING_TAGS = {f"h{n}": "#" * n for n in range(1, 7)} +_BLOCK_TAGS = frozenset( + { + "p", "div", "section", "article", "header", "footer", "main", "nav", + "aside", "figure", "figcaption", "table", "thead", "tbody", "tr", + "blockquote", "form", "details", "summary", + } +) + + +class _MarkdownExtractor(HTMLParser): + """Convert a fragment of rendered HTML into approximate markdown. + + Deliberately small and dependency-free: it covers the common structural + tags (headings, paragraphs, lists, links, emphasis, code, blockquotes, + rules) and drops everything it doesn't understand to plain text. This backs + the opt-in ``auto_convert`` fallback only — author-provided markdown or a + handler always produces cleaner output. + """ + + def __init__(self) -> None: + super().__init__(convert_charrefs=True) + self._out: list[str] = [] + self._skip_depth = 0 + self._pre_depth = 0 + self._list_stack: list[dict[str, Any]] = [] # {"ordered": bool, "n": int} + self._quote_depth = 0 + self._pending_prefix: str | None = None + + # -- helpers ---------------------------------------------------------- + def _emit(self, text: str) -> None: + self._out.append(text) + + def _newline(self, count: int = 1) -> None: + self._out.append("\n" * count) + + # -- tag handling ----------------------------------------------------- + def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + if self._skip_depth: + if tag in _SKIP_TAGS: + self._skip_depth += 1 + return + if tag in _SKIP_TAGS: + self._skip_depth = 1 + return + if tag in _HEADING_TAGS: + self._newline(2) + self._emit(_HEADING_TAGS[tag] + " ") + elif tag == "br": + self._newline() + elif tag == "hr": + self._newline(2) + self._emit("---") + self._newline(2) + elif tag in ("strong", "b"): + self._emit("**") + elif tag in ("em", "i"): + self._emit("*") + elif tag == "code" and not self._pre_depth: + self._emit("`") + elif tag == "pre": + self._pre_depth += 1 + self._newline(2) + self._emit("```\n") + elif tag == "a": + href = _attr(attrs, "href") + self._emit("[") + self._pending_prefix = href or "" + elif tag in ("ul", "ol"): + self._list_stack.append({"ordered": tag == "ol", "n": 0}) + elif tag == "li": + self._newline() + marker = "- " + if self._list_stack: + top = self._list_stack[-1] + if top["ordered"]: + top["n"] += 1 + marker = f"{top['n']}. " + self._emit(" " * (len(self._list_stack) - 1) + marker) + else: + self._emit(marker) + elif tag == "blockquote": + self._quote_depth += 1 + self._newline(2) + elif tag in _BLOCK_TAGS: + self._newline(2) + + def handle_endtag(self, tag: str) -> None: + if self._skip_depth: + if tag in _SKIP_TAGS: + self._skip_depth -= 1 + return + if tag in _HEADING_TAGS: + self._newline(2) + elif tag in ("strong", "b"): + self._emit("**") + elif tag in ("em", "i"): + self._emit("*") + elif tag == "code" and not self._pre_depth: + self._emit("`") + elif tag == "pre": + self._pre_depth = max(0, self._pre_depth - 1) + self._emit("\n```") + self._newline(2) + elif tag == "a" and self._pending_prefix is not None: + href = self._pending_prefix + self._pending_prefix = None + self._emit(f"]({href})" if href else "]") + elif tag in ("ul", "ol"): + if self._list_stack: + self._list_stack.pop() + if not self._list_stack: + self._newline(2) + elif tag == "blockquote": + self._quote_depth = max(0, self._quote_depth - 1) + self._newline(2) + elif tag in _BLOCK_TAGS: + self._newline(2) + + def handle_data(self, data: str) -> None: + if self._skip_depth: + return + if self._pre_depth: + self._emit(data) + return + text = " ".join(data.split()) + if not text: + return + # Preserve a single leading/trailing space when the source had one, so + # inline elements don't get glued to adjacent words. + if data[:1].isspace(): + text = " " + text + if data[-1:].isspace(): + text = text + " " + self._emit(text) + + def result(self) -> str: + text = "".join(self._out) + # Collapse 3+ blank lines and trim. + lines = [line.rstrip() for line in text.split("\n")] + collapsed: list[str] = [] + blanks = 0 + for line in lines: + if line: + blanks = 0 + collapsed.append(line) + else: + blanks += 1 + if blanks <= 1: + collapsed.append("") + return "\n".join(collapsed).strip() + "\n" + + +def _attr(attrs: list[tuple[str, str | None]], name: str) -> str | None: + for key, value in attrs: + if key == name: + return value + return None + + +def html_to_markdown(html: str) -> str: + """Convert rendered HTML to approximate markdown (best-effort, lossy).""" + parser = _MarkdownExtractor() + parser.feed(html) + parser.close() + return parser.result() + + +# --------------------------------------------------------------------------- +# /llms.txt index +# --------------------------------------------------------------------------- + + +def _humanize_segment(segment: str) -> str: + words = segment.replace("-", " ").replace("_", " ").split() + return " ".join(word[:1].upper() + word[1:] for word in words) if words else segment + + +def _label_for(page: PageRoute) -> str: + path = page.path.rstrip("/") + if not path: + return "Home" + return _humanize_segment(path.rsplit("/", 1)[-1]) + + +def _md_url_for(page_path: str) -> str: + return "/index.md" if page_path == "/" else page_path.rstrip("/") + ".md" + + +def _listable_pages(routes: RouteTable) -> list[PageRoute]: + """Concrete (non-parameterised) pages, de-duplicated and sorted by path.""" + seen: set[str] = set() + pages: list[PageRoute] = [] + for page in routes.pages: + if "{" in page.path or page.path in seen: + continue + seen.add(page.path) + pages.append(page) + pages.sort(key=lambda p: p.path) + return pages + + +def _default_title(settings: Any) -> str: + root = getattr(settings, "project_root", None) + if root is not None: + return _humanize_segment(Path(root).name) or "Pyxle app" + return "Pyxle app" + + +@dataclass(frozen=True) +class LlmsPageInfo: + """A concrete (non-parameterised) page surfaced to a ``llms_txt`` hook.""" + + path: str + md_url: str + title: str + + +@dataclass(frozen=True) +class LlmsTxtContext: + """Context passed to a root ``pages/llms.py`` ``llms_txt`` hook. + + ``pages`` lists the app's concrete pages; ``render_default`` returns the + framework's generated index, so a hook can return it verbatim, tweak it, or + build a fully custom index (for example from a docs manifest). + """ + + request: Request + pages: tuple[LlmsPageInfo, ...] + _render_default: Callable[[], str] + + def render_default(self) -> str: + """Return the framework's generated ``/llms.txt``.""" + return self._render_default() + + +def _page_infos(routes: RouteTable) -> tuple[LlmsPageInfo, ...]: + return tuple( + LlmsPageInfo(path=page.path, md_url=_md_url_for(page.path), title=_label_for(page)) + for page in _listable_pages(routes) + ) + + +def build_llms_txt(*, routes: RouteTable, settings: Any) -> str: + """Generate a spec-shaped ``/llms.txt`` index from the route table. + + Produces an H1 title (the project directory name) and a ``## Pages`` list + linking each concrete page's ``.md`` rendition. Dynamic (parameterised) + routes are omitted since they have no single URL to list — apps with dynamic + content should provide a ``llms_txt`` hook or a static ``public/llms.txt``. + """ + lines = [f"# {_default_title(settings)}", "", "## Pages", ""] + for info in _page_infos(routes): + lines.append(f"- [{info.title}]({info.md_url})") + lines.append("") + return "\n".join(lines) + + +# --------------------------------------------------------------------------- +# Accept-header negotiation + discovery headers +# --------------------------------------------------------------------------- + + +def wants_markdown(request: Request) -> bool: + """Return ``True`` when a request explicitly accepts ``text/markdown``. + + Browsers never send ``text/markdown`` in ``Accept``, so this only fires for + agents that opt in — the canonical HTML URL is unchanged for humans. + """ + return "text/markdown" in request.headers.get("accept", "") + + +class LlmsDiscoveryMiddleware: + """Advertise the ``/llms.txt`` index via ``Link`` and ``X-Llms-Txt`` headers. + + Pure ASGI (never buffers the body) so it is safe in front of streaming SSR. + """ + + def __init__(self, app: Any, *, index_path: str = LLMS_TXT_PATH) -> None: + self.app = app + self._index = index_path + self._link = f'<{index_path}>; rel="llms-txt"' + + async def __call__(self, scope: Any, receive: Any, send: Any) -> None: + if scope.get("type") != "http": + await self.app(scope, receive, send) + return + + async def send_wrapper(message: Any) -> None: + if message["type"] == "http.response.start": + headers = MutableHeaders(raw=message.setdefault("headers", [])) + existing = headers.get("link") + headers["link"] = f"{existing}, {self._link}" if existing else self._link + headers["x-llms-txt"] = self._index + await send(message) + + await self.app(scope, receive, send_wrapper) + + +# --------------------------------------------------------------------------- +# Route construction +# --------------------------------------------------------------------------- + + +def markdown_route_path(page_path: str) -> str: + """Return the ``.md`` route pattern for a page route path. + + ``/`` -> ``/index.md``; ``/about`` -> ``/about.md``; + ``/docs/{slug:path}`` -> ``/docs/{slug:path}.md``. + """ + if page_path == "/": + return "/index.md" + return page_path.rstrip("/") + ".md" + + +def make_markdown_route_handler( + page: PageRoute, + *, + settings: Any, + renderer: Any, + config: Any, +) -> Callable[[Request], Awaitable[Response]]: + """Build the Starlette handler serving ``.md``.""" + + async def handler(request: Request) -> Response: + try: + markdown = await resolve_page_markdown( + request=request, + page=page, + settings=settings, + renderer=renderer, + config=config, + ) + except Exception: + # Never surface internals on the .md channel — log and gracefully + # fall back to the HTML page, which carries the same content. + logger.exception("Markdown rendering failed for %s", request.url.path) + markdown = None + if markdown is None: + return RedirectResponse(strip_md_suffix(request.url.path), status_code=307) + return PlainTextResponse(markdown, media_type=MARKDOWN_MEDIA_TYPE) + + handler.__name__ = f"markdown_{page.module_key.replace('.', '_')}" + return handler + + +def build_markdown_routes( + routes: RouteTable, + *, + settings: Any, + renderer: Any, + config: Any, +) -> list[Route]: + """Build one ``.md`` route per page route (registered before page routes).""" + built: list[Route] = [] + seen: set[str] = set() + for page in routes.pages: + md_path = markdown_route_path(page.path) + if md_path in seen: + continue + seen.add(md_path) + built.append( + Route( + md_path, + make_markdown_route_handler( + page, settings=settings, renderer=renderer, config=config + ), + methods=["GET", "HEAD"], + ) + ) + return built + + +def _load_llms_txt_hook(settings: Any, *, debug: bool) -> Optional[Callable[..., Any]]: + """Return the ``llms_txt`` function from the root ``pages/llms.py``, if any.""" + return _load_root_llms_attr(settings, LLMS_TXT_HOOK_NAME, debug=debug) + + +def make_llms_txt_route(routes: RouteTable, *, settings: Any) -> Route: + """Build the ``/llms.txt`` route. + + Resolution order: a static ``public/llms.txt`` (served by the static-asset + middleware before this route ever runs) → a ``llms_txt`` hook in the root + ``pages/llms.py`` → a generated index of the app's pages. + """ + + async def handler(request: Request) -> Response: + debug = bool(getattr(settings, "debug", False)) + hook = _load_llms_txt_hook(settings, debug=debug) + if hook is not None: + ctx = LlmsTxtContext( + request=request, + pages=_page_infos(routes), + _render_default=lambda: build_llms_txt(routes=routes, settings=settings), + ) + try: + result = hook(ctx) + if inspect.isawaitable(result): + result = await result + except Exception: + logger.exception("Root llms.py 'llms_txt' hook failed") + result = None + if isinstance(result, str): + return PlainTextResponse(result, media_type=MARKDOWN_MEDIA_TYPE) + return PlainTextResponse( + build_llms_txt(routes=routes, settings=settings), + media_type=MARKDOWN_MEDIA_TYPE, + ) + + handler.__name__ = "llms_txt" + return Route(LLMS_TXT_PATH, handler, methods=["GET", "HEAD"]) + + +__all__ = [ + "LLMS_TXT_PATH", + "MARKDOWN_MEDIA_TYPE", + "LlmsDiscoveryMiddleware", + "LlmsPageInfo", + "LlmsTxtContext", + "MarkdownContext", + "build_llms_txt", + "build_markdown_routes", + "colocated_markdown_path", + "html_to_markdown", + "is_enabled", + "make_llms_txt_route", + "make_markdown_route_handler", + "markdown_route_path", + "resolve_page_markdown", + "strip_md_suffix", + "wants_markdown", +] diff --git a/pyxle/devserver/settings.py b/pyxle/devserver/settings.py index 12558c0..4817daa 100644 --- a/pyxle/devserver/settings.py +++ b/pyxle/devserver/settings.py @@ -56,6 +56,9 @@ class DevServerSettings: # Request observability (ObservabilityConfig). None = framework defaults # (request-id + timing on). observability: Any = None + # AI accessibility (LlmsConfig): per-page ``.md`` + ``/llms.txt``. None / + # disabled = feature off. + llms: Any = None # Plugin entries from pyxle.config.json::plugins — raw payload # (strings or dicts), resolved to PluginSpec/PyxlePlugin instances # by the starlette app at startup. Empty tuple = no plugins. @@ -88,6 +91,7 @@ def from_project_root( navigation: Any = None, rate_limit: Any = None, observability: Any = None, + llms: Any = None, plugins: Sequence[Any] | None = None, ) -> "DevServerSettings": """Create settings derived from a project root directory.""" @@ -150,6 +154,7 @@ def from_project_root( navigation=navigation, rate_limit=rate_limit, observability=observability, + llms=llms, plugins=tuple(plugins) if plugins else (), ) diff --git a/pyxle/devserver/starlette_app.py b/pyxle/devserver/starlette_app.py index fbf12f5..1509a91 100644 --- a/pyxle/devserver/starlette_app.py +++ b/pyxle/devserver/starlette_app.py @@ -23,7 +23,7 @@ from starlette.middleware import Middleware from starlette.middleware.base import BaseHTTPMiddleware from starlette.requests import Request -from starlette.responses import JSONResponse, Response +from starlette.responses import JSONResponse, PlainTextResponse, Response from starlette.routing import Mount, Route, Router, WebSocketRoute from starlette.staticfiles import NotModifiedResponse, StaticFiles from starlette.types import ASGIApp, Message, Receive, Scope, Send @@ -65,6 +65,7 @@ load_route_hooks, wrap_with_route_hooks, ) +from . import llms from .routes import ActionRoute, ApiRoute, PageRoute, RouteTable, select_static_pages from .settings import DevServerSettings @@ -1086,6 +1087,51 @@ async def _build_cached_page_response( return response +def _merge_vary(response: Response, value: str) -> None: + """Add ``value`` to a response's ``Vary`` header without dropping existing tokens.""" + existing = response.headers.get("vary") + if not existing: + response.headers["vary"] = value + return + tokens = {token.strip().lower() for token in existing.split(",")} + if value.lower() not in tokens: + response.headers["vary"] = f"{existing}, {value}" + + +async def _maybe_markdown_response( + request: Request, + route: PageRoute, + *, + settings: DevServerSettings, + renderer: ComponentRenderer, + llms_cfg: Any, +) -> Response | None: + """Return a markdown response for an ``Accept: text/markdown`` request, else None. + + Content negotiation on the canonical page URL: agents that opt into markdown + get the page's ``.md`` rendition; browsers (which never send that Accept) + fall through to HTML. Any failure resolving markdown also falls through to + HTML, where a genuine render error still surfaces via the error boundary. + """ + if not (llms.is_enabled(llms_cfg) and llms.wants_markdown(request)): + return None + try: + markdown = await llms.resolve_page_markdown( + request=request, + page=route, + settings=settings, + renderer=renderer, + config=llms_cfg, + ) + except Exception: + return None + if markdown is None: + return None + response = PlainTextResponse(markdown, media_type=llms.MARKDOWN_MEDIA_TYPE) + response.headers["Vary"] = "Accept" + return response + + def _make_page_handler( route: PageRoute, *, @@ -1096,6 +1142,9 @@ def _make_page_handler( page_cache: PageCache | None = None, stream_render: Callable[..., Any] | None = None, ): + llms_cfg = getattr(settings, "llms", None) + llms_on = llms.is_enabled(llms_cfg) + async def handler(request: Request): # pragma: no cover - thin wrapper wants_navigation_payload = request.headers.get(_NAVIGATION_HEADER) == "1" if wants_navigation_payload: @@ -1118,7 +1167,14 @@ async def handler(request: Request): # pragma: no cover - thin wrapper response.headers["Cache-Control"] = "no-store" return response - return await _build_cached_page_response( + if llms_on: + md_response = await _maybe_markdown_response( + request, route, settings=settings, renderer=renderer, llms_cfg=llms_cfg + ) + if md_response is not None: + return md_response + + response = await _build_cached_page_response( request=request, route=route, settings=settings, @@ -1128,6 +1184,11 @@ async def handler(request: Request): # pragma: no cover - thin wrapper page_cache=page_cache, stream_render=stream_render, ) + if llms_on: + # The canonical URL now varies by Accept (HTML vs markdown), so a + # shared cache must key on it. + _merge_vary(response, "Accept") + return response handler.__name__ = f"page_{route.module_key.replace('.', '_')}" return handler @@ -1479,6 +1540,20 @@ def _build_app_routes( built: list[Any] = [] built.extend(api_router.routes) built.extend(action_router.routes) + # AI accessibility: per-page ``.md`` routes + ``/llms.txt``, registered + # BEFORE the page routes so ``/x.md`` resolves here rather than being + # captured by a dynamic page route (e.g. ``/docs/{slug:path}`` would + # otherwise match ``/docs/x.md`` with ``slug="x.md"``). Off unless the + # ``llms`` config block is enabled; a static ``public/llms.txt`` (served by + # the static middleware) still takes precedence over the generated index. + _llms_cfg = getattr(settings, "llms", None) + if llms.is_enabled(_llms_cfg): + built.extend( + llms.build_markdown_routes( + routes, settings=settings, renderer=renderer, config=_llms_cfg + ) + ) + built.append(llms.make_llms_txt_route(routes, settings=settings)) built.extend(page_router.routes) if overlay is not None: built.append(WebSocketRoute("/__pyxle__/overlay", overlay.websocket_endpoint)) @@ -1876,6 +1951,11 @@ async def lifespan(app: Starlette): # pragma: no cover - lifecycle orchestratio if not settings.debug: middleware_stack.append(Middleware(_SecurityHeadersMiddleware)) + # Advertise the /llms.txt index on every response via Link + X-Llms-Txt + # headers when AI accessibility is enabled (pure ASGI, streaming-safe). + if llms.is_enabled(getattr(settings, "llms", None)): + middleware_stack.append(Middleware(llms.LlmsDiscoveryMiddleware)) + if cors_middleware is not None: middleware_stack.append(cors_middleware) if csrf_middleware is not None: diff --git a/pyxle/ssr/view.py b/pyxle/ssr/view.py index 1ce2f65..97ca226 100644 --- a/pyxle/ssr/view.py +++ b/pyxle/ssr/view.py @@ -249,6 +249,54 @@ async def _document_stream(): ) +async def render_page_body_html( + *, + request: Request, + settings: DevServerSettings, + page: PageRoute, + renderer: ComponentRenderer, + suppress_per_user: bool = False, +) -> tuple[str, int]: + """Render a page and return ``(body_html, status_code)``. + + Runs the same loader + SSR path as :func:`build_page_response` but returns + the rendered component HTML *without* the surrounding document shell, so + callers (such as the ``.md`` markdown resolver) can post-process the + content. Propagates the same render-stage exceptions as the page pipeline. + """ + breadcrumb = _initial_loader_breadcrumb(page) + artifacts = await _create_page_artifacts( + request=request, + settings=settings, + page=page, + renderer=renderer, + loader_breadcrumb=breadcrumb, + suppress_per_user=suppress_per_user, + ) + return artifacts.body_html, artifacts.status_code + + +async def run_page_loader( + *, + request: Request, + settings: DevServerSettings, + page: PageRoute, +) -> Any: + """Run a page's ``@server`` loader and return its data — no SSR render. + + A lighter-weight counterpart to :func:`render_page_body_html` for callers + (such as the ``.md`` markdown resolver) that only need the loader's return + value. Returns the loader's data dict; a page with no loader returns ``{}``. + Propagates loader exceptions. + """ + if settings.debug: + _purge_page_modules(settings.pages_dir) + payload, _status, _revalidate, _module = await _execute_loader( + page, request, module=None, debug=settings.debug + ) + return payload + + async def _handle_render_exception( exc: BaseException, *, diff --git a/tests/devserver/test_llms.py b/tests/devserver/test_llms.py new file mode 100644 index 0000000..69394e1 --- /dev/null +++ b/tests/devserver/test_llms.py @@ -0,0 +1,558 @@ +"""Tests for the AI-accessibility feature (per-page ``.md`` + ``/llms.txt``).""" + +from __future__ import annotations + +from pathlib import Path +from types import SimpleNamespace + +import pytest +from starlette.applications import Starlette +from starlette.responses import PlainTextResponse +from starlette.testclient import TestClient + +from pyxle.devserver import llms +from pyxle.devserver.starlette_app import _maybe_markdown_response, _merge_vary + +pytestmark = pytest.mark.anyio("asyncio") + + +@pytest.fixture +def anyio_backend() -> str: # pragma: no cover - fixture wiring + return "asyncio" + + +# --------------------------------------------------------------------------- +# Pure helpers +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "path,expected", + [ + ("/about.md", "/about"), + ("/index.md", "/"), + ("/docs/routing.md", "/docs/routing"), + ("/docs/a/b.md", "/docs/a/b"), + ("/deep/index.md", "/deep/"), + ("/plain", "/plain"), + ], +) +def test_strip_md_suffix(path, expected): + assert llms.strip_md_suffix(path) == expected + + +@pytest.mark.parametrize( + "page_path,expected", + [ + ("/", "/index.md"), + ("/about", "/about.md"), + ("/docs/{slug:path}", "/docs/{slug:path}.md"), + ("/blog/", "/blog.md"), + ], +) +def test_markdown_route_path(page_path, expected): + assert llms.markdown_route_path(page_path) == expected + + +def test_wants_markdown(): + req_md = SimpleNamespace(headers={"accept": "text/markdown, text/html"}) + req_html = SimpleNamespace(headers={"accept": "text/html,application/xhtml+xml"}) + req_none = SimpleNamespace(headers={}) + assert llms.wants_markdown(req_md) is True + assert llms.wants_markdown(req_html) is False + assert llms.wants_markdown(req_none) is False + + +def test_is_enabled(): + assert llms.is_enabled(None) is False + assert llms.is_enabled(SimpleNamespace(enabled=False)) is False + assert llms.is_enabled(SimpleNamespace(enabled=True)) is True + + +# --------------------------------------------------------------------------- +# HTML -> markdown converter +# --------------------------------------------------------------------------- + + +def test_html_to_markdown_structure(): + html = ( + "

Title

" + "

Hello world and friends, see " + 'here.

' + "
  • one
  • two
" + "
  1. first
  2. second
" + "
x = 1\ny = 2
" + "

inline code() sample

" + "
quoted
" + "
" + ) + md = llms.html_to_markdown(html) + assert "# Title" in md + assert "**world**" in md + assert "*friends*" in md + assert "[here](/x)" in md + assert "- one" in md and "- two" in md + assert "1. first" in md and "2. second" in md + assert "x = 1\ny = 2" in md # code preserved verbatim + assert "`code()`" in md + assert "---" in md + # dropped content + assert "evil()" not in md + assert ".a{}" not in md + + +def test_html_to_markdown_collapses_blank_lines_and_trails_newline(): + md = llms.html_to_markdown("

a

b

") + assert md.endswith("\n") + assert "\n\n\n" not in md + + +def test_html_to_markdown_link_without_href(): + md = llms.html_to_markdown("bare") + assert "bare" in md + assert "](" not in md + + +# --------------------------------------------------------------------------- +# Resolution ladder +# --------------------------------------------------------------------------- + + +@pytest.fixture +def app_tree(tmp_path): + pages = tmp_path / "pages" + (pages / "docs").mkdir(parents=True) + settings = SimpleNamespace(pages_dir=pages, project_root=tmp_path, debug=True, llms=SimpleNamespace(enabled=True, auto_convert=False)) + return SimpleNamespace(root=tmp_path, pages=pages, settings=settings) + + +def _page_in(app_tree, rel: str, path: str, **over): + abs_src = app_tree.pages / rel + return SimpleNamespace( + path=path, + source_absolute_path=abs_src, + source_relative_path=Path(rel), + server_module_path=abs_src.with_suffix(".py"), + module_key=f"pages.{rel.replace('/', '.').removesuffix('.pyxl')}", + **over, + ) + + +async def _resolve(app_tree, page, request_path, config=None): + request = SimpleNamespace(url=SimpleNamespace(path=request_path), headers={}) + return await llms.resolve_page_markdown( + request=request, + page=page, + settings=app_tree.settings, + renderer=None, + config=config or app_tree.settings.llms, + ) + + +async def test_colocated_md_file_wins(app_tree): + (app_tree.pages / "about.pyxl").write_text("x") + (app_tree.pages / "about.md").write_text("# About (file)\n") + page = _page_in(app_tree, "about.pyxl", "/about") + assert await _resolve(app_tree, page, "/about.md") == "# About (file)\n" + + +async def test_page_local_to_markdown(app_tree): + (app_tree.pages / "p.pyxl").write_text("x") + (app_tree.pages / "p.py").write_text( + "async def to_markdown(ctx):\n return f'# {ctx.path}'\n" + ) + page = _page_in(app_tree, "p.pyxl", "/p") + assert await _resolve(app_tree, page, "/p.md") == "# /p" + + +async def test_directory_llms_py_covers_subtree(app_tree): + (app_tree.pages / "docs" / "intro.pyxl").write_text("x") + (app_tree.pages / "docs" / "llms.py").write_text( + "def to_markdown(ctx):\n return 'DIR: ' + ctx.path\n" + ) + page = _page_in(app_tree, "docs/intro.pyxl", "/docs/intro") + assert await _resolve(app_tree, page, "/docs/intro.md") == "DIR: /docs/intro" + + +async def test_root_llms_py_is_app_wide(app_tree): + (app_tree.pages / "contact.pyxl").write_text("x") + (app_tree.pages / "llms.py").write_text( + "def to_markdown(ctx):\n return 'ROOT'\n" + ) + page = _page_in(app_tree, "contact.pyxl", "/contact") + assert await _resolve(app_tree, page, "/contact.md") == "ROOT" + + +async def test_nearest_directory_handler_wins_over_root(app_tree): + (app_tree.pages / "docs" / "x.pyxl").write_text("x") + (app_tree.pages / "llms.py").write_text("def to_markdown(ctx):\n return 'ROOT'\n") + (app_tree.pages / "docs" / "llms.py").write_text("def to_markdown(ctx):\n return 'DOCS'\n") + page = _page_in(app_tree, "docs/x.pyxl", "/docs/x") + assert await _resolve(app_tree, page, "/docs/x.md") == "DOCS" + + +async def test_handler_returning_none_falls_through(app_tree): + (app_tree.pages / "docs" / "y.pyxl").write_text("x") + # directory handler declines -> root handler answers + (app_tree.pages / "docs" / "llms.py").write_text("def to_markdown(ctx):\n return None\n") + (app_tree.pages / "llms.py").write_text("def to_markdown(ctx):\n return 'ROOT'\n") + page = _page_in(app_tree, "docs/y.pyxl", "/docs/y") + assert await _resolve(app_tree, page, "/docs/y.md") == "ROOT" + + +async def test_auto_convert_fallback(app_tree, monkeypatch): + import pyxle.ssr.view as view + + async def fake_render(*, request, settings, page, renderer, suppress_per_user=False): + return "

Converted

", 200 + + monkeypatch.setattr(view, "render_page_body_html", fake_render) + (app_tree.pages / "z.pyxl").write_text("x") + page = _page_in(app_tree, "z.pyxl", "/z") + cfg = SimpleNamespace(enabled=True, auto_convert=True) + md = await _resolve(app_tree, page, "/z.md", config=cfg) + assert "# Converted" in md + + +async def test_no_source_returns_none(app_tree): + (app_tree.pages / "empty.pyxl").write_text("x") + page = _page_in(app_tree, "empty.pyxl", "/empty") + # no .md, no handlers, auto_convert off -> None (caller redirects) + assert await _resolve(app_tree, page, "/empty.md") is None + + +async def test_handler_must_return_str_or_none(app_tree): + (app_tree.pages / "bad.pyxl").write_text("x") + (app_tree.pages / "bad.py").write_text("def to_markdown(ctx):\n return 123\n") + page = _page_in(app_tree, "bad.pyxl", "/bad") + with pytest.raises(TypeError): + await _resolve(app_tree, page, "/bad.md") + + +# --------------------------------------------------------------------------- +# /llms.txt generation +# --------------------------------------------------------------------------- + + +def _routes(*paths): + return SimpleNamespace(pages=[SimpleNamespace(path=p) for p in paths]) + + +def test_build_llms_txt(tmp_path): + settings = SimpleNamespace(project_root=tmp_path / "acme-docs", pages_dir=tmp_path) + routes = _routes("/", "/about", "/docs/{slug:path}", "/benchmarks") + txt = llms.build_llms_txt(routes=routes, settings=settings) + assert txt.startswith("# Acme Docs") + assert "- [Home](/index.md)" in txt + assert "- [About](/about.md)" in txt + assert "- [Benchmarks](/benchmarks.md)" in txt + assert "slug" not in txt # dynamic route omitted + + +# --------------------------------------------------------------------------- +# Route + middleware integration (via TestClient) +# --------------------------------------------------------------------------- + + +def test_markdown_route_serves_and_redirects(app_tree): + (app_tree.pages / "about.pyxl").write_text("x") + (app_tree.pages / "about.md").write_text("# About\n") + (app_tree.pages / "missing.pyxl").write_text("x") + + routes = SimpleNamespace( + pages=[ + _page_in(app_tree, "about.pyxl", "/about"), + _page_in(app_tree, "missing.pyxl", "/missing"), + ] + ) + md_routes = llms.build_markdown_routes( + routes, settings=app_tree.settings, renderer=None, config=app_tree.settings.llms + ) + app = Starlette(routes=md_routes) + client = TestClient(app) + + resp = client.get("/about.md") + assert resp.status_code == 200 + assert resp.text == "# About\n" + assert resp.headers["content-type"].startswith("text/markdown") + + # No source resolves -> 307 redirect to the canonical page. + resp2 = client.get("/missing.md", follow_redirects=False) + assert resp2.status_code == 307 + assert resp2.headers["location"] == "/missing" + + +def test_markdown_route_dynamic_slug(app_tree): + (app_tree.pages / "docs").mkdir(exist_ok=True) + (app_tree.pages / "docs" / "llms.py").write_text( + "def to_markdown(ctx):\n return 'SLUG:' + ctx.request.path_params.get('slug', '')\n" + ) + (app_tree.pages / "docs" / "cat.pyxl").write_text("x") + routes = SimpleNamespace( + pages=[_page_in(app_tree, "docs/cat.pyxl", "/docs/{slug:path}")] + ) + md_routes = llms.build_markdown_routes( + routes, settings=app_tree.settings, renderer=None, config=app_tree.settings.llms + ) + client = TestClient(Starlette(routes=md_routes)) + resp = client.get("/docs/getting-started.md") + assert resp.status_code == 200 + assert resp.text == "SLUG:getting-started" + + +def test_llms_txt_route_default_and_hook(app_tree): + routes = _routes("/", "/about") + # default generated index + default_route = llms.make_llms_txt_route(routes, settings=app_tree.settings) + client = TestClient(Starlette(routes=[default_route])) + resp = client.get("/llms.txt") + assert resp.status_code == 200 + assert "## Pages" in resp.text + assert "- [About](/about.md)" in resp.text + + # hook override in root pages/llms.py + (app_tree.pages / "llms.py").write_text( + "def llms_txt(ctx):\n return '# Custom\\n' + ctx.render_default()\n" + ) + hook_route = llms.make_llms_txt_route(routes, settings=app_tree.settings) + client2 = TestClient(Starlette(routes=[hook_route])) + resp2 = client2.get("/llms.txt") + assert resp2.text.startswith("# Custom\n") + assert "## Pages" in resp2.text + + +def test_discovery_middleware_adds_headers(): + async def app(scope, receive, send): + await PlainTextResponse("ok")(scope, receive, send) + + wrapped = llms.LlmsDiscoveryMiddleware(app) + client = TestClient(wrapped) + resp = client.get("/anything") + assert resp.headers["x-llms-txt"] == "/llms.txt" + assert 'rel="llms-txt"' in resp.headers["link"] + + +def test_discovery_middleware_appends_to_existing_link(): + async def app(scope, receive, send): + resp = PlainTextResponse("ok") + resp.headers["link"] = "; rel=preload" + await resp(scope, receive, send) + + client = TestClient(llms.LlmsDiscoveryMiddleware(app)) + resp = client.get("/x") + link = resp.headers["link"] + assert "rel=preload" in link and 'rel="llms-txt"' in link + + +# --------------------------------------------------------------------------- +# starlette_app helpers: _merge_vary + _maybe_markdown_response (negotiation) +# --------------------------------------------------------------------------- + + +def test_merge_vary(): + r = PlainTextResponse("x") + _merge_vary(r, "Accept") + assert r.headers["vary"] == "Accept" + _merge_vary(r, "Accept") # idempotent + assert r.headers["vary"] == "Accept" + + r2 = PlainTextResponse("x") + r2.headers["vary"] = "X-Foo" + _merge_vary(r2, "Accept") + assert r2.headers["vary"] == "X-Foo, Accept" + + r3 = PlainTextResponse("x") + r3.headers["vary"] = "accept" # already present (case-insensitive) + _merge_vary(r3, "Accept") + assert r3.headers["vary"] == "accept" + + +def _req(path: str, accept: str = "text/markdown"): + return SimpleNamespace(url=SimpleNamespace(path=path), headers={"accept": accept}) + + +async def test_negotiation_disabled_returns_none(app_tree): + page = _page_in(app_tree, "about.pyxl", "/about") + resp = await _maybe_markdown_response( + _req("/about"), page, settings=app_tree.settings, + renderer=None, llms_cfg=SimpleNamespace(enabled=False), + ) + assert resp is None + + +async def test_negotiation_not_requested_returns_none(app_tree): + page = _page_in(app_tree, "about.pyxl", "/about") + resp = await _maybe_markdown_response( + _req("/about", accept="text/html"), page, settings=app_tree.settings, + renderer=None, llms_cfg=app_tree.settings.llms, + ) + assert resp is None + + +async def test_negotiation_serves_markdown(app_tree): + (app_tree.pages / "about.pyxl").write_text("x") + (app_tree.pages / "about.md").write_text("# About\n") + page = _page_in(app_tree, "about.pyxl", "/about") + resp = await _maybe_markdown_response( + _req("/about"), page, settings=app_tree.settings, + renderer=None, llms_cfg=app_tree.settings.llms, + ) + assert resp is not None + assert resp.body == b"# About\n" + assert resp.headers["vary"] == "Accept" + assert resp.headers["content-type"].startswith("text/markdown") + + +async def test_negotiation_none_when_unresolved(app_tree): + (app_tree.pages / "solo.pyxl").write_text("x") + page = _page_in(app_tree, "solo.pyxl", "/solo") + resp = await _maybe_markdown_response( + _req("/solo"), page, settings=app_tree.settings, + renderer=None, llms_cfg=app_tree.settings.llms, + ) + assert resp is None + + +async def test_negotiation_swallows_errors(app_tree, monkeypatch): + async def boom(**_kw): + raise RuntimeError("nope") + + monkeypatch.setattr(llms, "resolve_page_markdown", boom) + page = _page_in(app_tree, "about.pyxl", "/about") + resp = await _maybe_markdown_response( + _req("/about"), page, settings=app_tree.settings, + renderer=None, llms_cfg=app_tree.settings.llms, + ) + assert resp is None + + +# --------------------------------------------------------------------------- +# Error/fallback branch coverage +# --------------------------------------------------------------------------- + + +def test_markdown_route_redirects_on_handler_error(app_tree): + (app_tree.pages / "boom.pyxl").write_text("x") + (app_tree.pages / "boom.py").write_text("def to_markdown(ctx):\n raise RuntimeError('x')\n") + routes = SimpleNamespace(pages=[_page_in(app_tree, "boom.pyxl", "/boom")]) + md_routes = llms.build_markdown_routes( + routes, settings=app_tree.settings, renderer=None, config=app_tree.settings.llms + ) + client = TestClient(Starlette(routes=md_routes)) + resp = client.get("/boom.md", follow_redirects=False) + assert resp.status_code == 307 + assert resp.headers["location"] == "/boom" + + +async def test_local_handler_import_failure_falls_through(app_tree): + (app_tree.pages / "brk.pyxl").write_text("x") + (app_tree.pages / "brk.py").write_text("raise ValueError('boom')\n") + (app_tree.pages / "llms.py").write_text("def to_markdown(ctx):\n return 'ROOT'\n") + page = _page_in(app_tree, "brk.pyxl", "/brk") + # page-local server module fails to import -> fall through to root llms.py + assert await _resolve(app_tree, page, "/brk.md") == "ROOT" + + +def test_llms_txt_hook_returning_none_falls_back(app_tree): + (app_tree.pages / "llms.py").write_text("def llms_txt(ctx):\n return None\n") + route = llms.make_llms_txt_route(_routes("/", "/about"), settings=app_tree.settings) + resp = TestClient(Starlette(routes=[route])).get("/llms.txt") + assert resp.status_code == 200 and "## Pages" in resp.text + + +def test_llms_txt_hook_error_falls_back(app_tree): + (app_tree.pages / "llms.py").write_text("def llms_txt(ctx):\n raise RuntimeError('x')\n") + route = llms.make_llms_txt_route(_routes("/", "/about"), settings=app_tree.settings) + resp = TestClient(Starlette(routes=[route])).get("/llms.txt") + assert resp.status_code == 200 and "## Pages" in resp.text + + +async def test_discovery_middleware_passes_non_http(): + seen = [] + + async def downstream(scope, receive, send): + seen.append(scope["type"]) + + await llms.LlmsDiscoveryMiddleware(downstream)({"type": "lifespan"}, None, None) + assert seen == ["lifespan"] + + +def test_html_to_markdown_blockquote_and_nested_list(): + md = llms.html_to_markdown( + "
quote
  • a
    • b
" + ) + assert "quote" in md + assert "- a" in md and "b" in md + + +# --------------------------------------------------------------------------- +# wrap_markdown hook (root pages/llms.py frames every .md response) +# --------------------------------------------------------------------------- + + +async def test_wrap_markdown_hook_frames_output(app_tree): + (app_tree.pages / "about.pyxl").write_text("x") + (app_tree.pages / "about.md").write_text("# About\n") + (app_tree.pages / "llms.py").write_text( + "def wrap_markdown(ctx, md):\n return f'\\n' + md + '\\n---\\nfooter'\n" + ) + page = _page_in(app_tree, "about.pyxl", "/about") + out = await _resolve(app_tree, page, "/about.md") + assert out.startswith("") + assert "# About" in out + assert out.rstrip().endswith("footer") + + +async def test_wrap_markdown_hook_none_keeps_markdown(app_tree): + (app_tree.pages / "about.pyxl").write_text("x") + (app_tree.pages / "about.md").write_text("# About\n") + (app_tree.pages / "llms.py").write_text("def wrap_markdown(ctx, md):\n return None\n") + page = _page_in(app_tree, "about.pyxl", "/about") + assert await _resolve(app_tree, page, "/about.md") == "# About\n" + + +async def test_wrap_markdown_not_applied_on_redirect(app_tree): + # No source resolves -> None (redirect); wrap must not run / must stay None. + (app_tree.pages / "solo.pyxl").write_text("x") + (app_tree.pages / "llms.py").write_text("def wrap_markdown(ctx, md):\n return 'WRAPPED'\n") + page = _page_in(app_tree, "solo.pyxl", "/solo") + assert await _resolve(app_tree, page, "/solo.md") is None + + +async def test_wrap_markdown_must_return_str_or_none(app_tree): + (app_tree.pages / "about.pyxl").write_text("x") + (app_tree.pages / "about.md").write_text("# About\n") + (app_tree.pages / "llms.py").write_text("def wrap_markdown(ctx, md):\n return 42\n") + page = _page_in(app_tree, "about.pyxl", "/about") + with pytest.raises(TypeError): + await _resolve(app_tree, page, "/about.md") + + +# --------------------------------------------------------------------------- +# ctx.run_loader() — loader data without the SSR render +# --------------------------------------------------------------------------- + + +async def test_run_loader_returns_loader_data(app_tree): + (app_tree.pages / "p.pyxl").write_text("x") + (app_tree.pages / "p.py").write_text( + "def load(request):\n" + " return {'n': 42}\n" + "\n" + "async def to_markdown(ctx):\n" + " data = await ctx.run_loader()\n" + " return f\"# {data['n']}\"\n" + ) + page = _page_in(app_tree, "p.pyxl", "/p", has_loader=True, loader_name="load") + assert await _resolve(app_tree, page, "/p.md") == "# 42" + + +async def test_run_loader_no_loader_returns_empty(app_tree): + (app_tree.pages / "q.pyxl").write_text("x") + (app_tree.pages / "q.py").write_text( + "async def to_markdown(ctx):\n" + " data = await ctx.run_loader()\n" + " return 'EMPTY' if data == {} else 'HAS'\n" + ) + page = _page_in(app_tree, "q.pyxl", "/q", has_loader=False, loader_name=None) + assert await _resolve(app_tree, page, "/q.md") == "EMPTY" diff --git a/tests/devserver/test_starlette_app.py b/tests/devserver/test_starlette_app.py index 7c3b055..82d1715 100644 --- a/tests/devserver/test_starlette_app.py +++ b/tests/devserver/test_starlette_app.py @@ -2929,3 +2929,43 @@ async def _stream_render(*args, **kwargs): # pragma: no cover - sentinel ) assert called.get("yes") is True assert response.headers["Cache-Control"] == "private, no-cache" + + +def test_build_app_routes_registers_llms_routes_when_enabled( + project: DevServerSettings, +) -> None: + from dataclasses import replace + + from pyxle.config import LlmsConfig + from pyxle.devserver.starlette_app import _build_app_routes + + build_once(project) + registry = load_metadata_registry(project) + table = build_route_table(registry) + + enabled = replace(project, llms=LlmsConfig(enabled=True)) + built, _eb = _build_app_routes( + settings=enabled, + routes=table, + renderer=object(), # type: ignore[arg-type] + overlay=None, + api_route_hooks=[], + page_route_hooks=[], + ) + paths = {getattr(route, "path", None) for route in built} + assert "/index.md" in paths + assert "/posts/{id}.md" in paths + assert "/llms.txt" in paths + + # Off by default: no markdown routes or index. + built_off, _ = _build_app_routes( + settings=project, + routes=table, + renderer=object(), # type: ignore[arg-type] + overlay=None, + api_route_hooks=[], + page_route_hooks=[], + ) + paths_off = {getattr(route, "path", None) for route in built_off} + assert "/index.md" not in paths_off + assert "/llms.txt" not in paths_off diff --git a/tests/test_config_security.py b/tests/test_config_security.py index 93f7e20..9dc936d 100644 --- a/tests/test_config_security.py +++ b/tests/test_config_security.py @@ -12,6 +12,7 @@ ConfigError, CorsConfig, CsrfConfig, + LlmsConfig, ObservabilityConfig, PyxleConfig, load_config, @@ -590,3 +591,68 @@ def test_boolean_sample_ratio_rejected(self, tmp_path: Path): # bool is a subclass of int — must not be accepted as a ratio. with pytest.raises(ConfigError, match="otelSampleRatio"): self._load(tmp_path, {"otelSampleRatio": True}) + + +# --------------------------------------------------------------------------- +# LlmsConfig — AI accessibility (per-page markdown + /llms.txt) +# --------------------------------------------------------------------------- + + +class TestLlmsConfigDefaults: + def test_default_disabled(self): + cfg = LlmsConfig() + assert cfg.enabled is False + assert cfg.auto_convert is False + + def test_default_in_pyxle_config(self): + assert PyxleConfig().llms == LlmsConfig() + + +class TestLlmsConfigParsing: + def _load(self, tmp_path: Path, llms_data) -> LlmsConfig: + config_file = tmp_path / "pyxle.config.json" + config_file.write_text(json.dumps({"llms": llms_data})) + return load_config(tmp_path, config_path=config_file).llms + + def test_no_block_returns_defaults(self, tmp_path: Path): + config_file = tmp_path / "pyxle.config.json" + config_file.write_text(json.dumps({})) + assert load_config(tmp_path, config_path=config_file).llms == LlmsConfig() + + def test_boolean_true_enables(self, tmp_path: Path): + assert self._load(tmp_path, True).enabled is True + + def test_boolean_false_disables(self, tmp_path: Path): + assert self._load(tmp_path, False).enabled is False + + def test_object_enables_by_default(self, tmp_path: Path): + assert self._load(tmp_path, {"autoConvert": True}).enabled is True + + def test_object_explicit_disable(self, tmp_path: Path): + cfg = self._load(tmp_path, {"enabled": False, "autoConvert": True}) + assert cfg.enabled is False + assert cfg.auto_convert is True + + def test_auto_convert_flag(self, tmp_path: Path): + assert self._load(tmp_path, {"enabled": True}).auto_convert is False + assert self._load(tmp_path, {"autoConvert": True}).auto_convert is True + + def test_unknown_key_rejected(self, tmp_path: Path): + with pytest.raises(ConfigError, match="llms"): + self._load(tmp_path, {"handler": "app:conv"}) + + def test_invalid_type_rejected(self, tmp_path: Path): + with pytest.raises(ConfigError, match="llms"): + self._load(tmp_path, ["not", "an", "object"]) + + def test_invalid_enabled_type_rejected(self, tmp_path: Path): + with pytest.raises(ConfigError, match="llms.enabled"): + self._load(tmp_path, {"enabled": "yes"}) + + def test_invalid_auto_convert_type_rejected(self, tmp_path: Path): + with pytest.raises(ConfigError, match="llms.autoConvert"): + self._load(tmp_path, {"autoConvert": "sure"}) + + def test_carried_into_devserver_kwargs(self): + kwargs = PyxleConfig(llms=LlmsConfig(enabled=True)).to_devserver_kwargs() + assert kwargs["llms"] == LlmsConfig(enabled=True)