Kitty graphics protocol - #37
Conversation
Kitty images (ESC _ G <control> ; <base64> ESC \) are decoded and placed in the buffer alongside Sixel, so icat, chafa -f kitty, timg -pk, yazi and image.nvim draw pictures against a host that renders tiles. Kitty was unreachable for the same reason Sixel had been: the parser collapsed SOS, PM and APC into one state that hunted for the terminator and discarded every byte. APC now has a real streaming path mirroring the DCS one, routed from ESC _ only, leaving ESC ^ and ESC X on the discard path they belong on. Cells reference an ImagePlacement rather than a TerminalImage. Sixel decodes a picture and shows it once, so the image identified the appearance; Kitty transmits once under an id and places as often as it likes, so it no longer does. BufferCell.Image stays as a computed Placement?.Image, and the struct does not grow. Tiles have two geometries, which are not the same formula. Natural keeps a fixed cell pitch and clips edge tiles; Stretched divides the source proportionally across the cell box, which is what c/r ask for. A 1160px image at a 14px cell needs 83 cells and 83x14 is 1162, so the two disagree on every tile rather than only the last. Sixel builds itself in natural mode, and a test pins the equivalence so the migration cannot drift. t=f, t=t and t=s are refused with ENOTSUP by design, not left unimplemented. They would have the terminal open a path named by the program it hosts, and a host generally holds more privilege than its guest. t=d is the only medium accepted. Animation, interlaced PNG, z-index and overlapping placements are also refused rather than silently ignored. Verified end to end through a real ConPTY: chafa --format kitty --size 4x2 produces a control-only opening sequence, 13 payload chunks and an empty terminator, landing 4x2 stretched tiles with nothing scrolled. ucs-detect reports "Graphics? Kitty, Sixel" from the real control -- its probe is byte-for-byte the a=q query StringSequenceTests already guarded, which also confirms a query places nothing. 938 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrundhvU51mvU7QHTwvnfY
Delete matrix. Only d=a and d=i were implemented; the rest were refused. Now by image number (d=n), at the cursor (d=c), at a cell (d=p), at a cell with a z-index (d=q), by column (d=x), by row (d=y), and by z-index (d=z), with a placement id narrowing any of the identity forms to a single appearance. Two keys change meaning on a delete: x and y are screen cells there rather than a crop origin, and they are one-based where the buffer is zero-based. Both conversions are pinned by tests that go red when either is dropped. A positional target finds a placement through one of its cells and then removes all of it -- deleting only the cells in the named row would leave a picture with a hole through it. The scrollback is deliberately not searched: a picture scrolled out of view is not "at row 3" however many rows above it happen to be. Image numbers. A client may send I=<number> instead of i=<id> and let the terminal choose the id; the reply carries both halves so the client can match the response and then use the image. Sending a number again makes a new image and the number follows the newest. Placeholder diacritics. The combining marks that state a tile's row and column were consumed and ignored, which only works for a rectangle written in reading order. They are decoded now, so a client may write tiles in any order. The table is kitty's own rowcolumn-diacritics.txt, taken verbatim rather than regenerated -- it was frozen against Unicode 6.0.0, and deriving it afresh would silently renumber every tile. Also fixed while there: a placeholder run built a fresh placement per cell, which rendered identically and cost a blit per cell instead of one per strip. One placement per run now. X and Y pixel offsets, and interlaced PNG. The offset shifts a picture inside its first cell and, per the spec, "is not added to the number of rows/columns" -- so the box is unchanged and the overflow is clipped. That case cannot be expressed by a tile's size alone, since the leading tile is both narrower and shifted, so TryGetTileLayout returns the source rectangle and the destination offset together. TryGetTileSource and GetTileCoverage still work and still return the same numbers. The tile arithmetic is now one uniform intersection -- the cell against the picture's span within the box, mapped back onto the source -- covering both scalings, cropping and the offsets. Scaling numerator and denominator by the same amount leaves the floor unchanged, so it reproduces the previous results exactly; the 1160x870-over-14x15 guard confirms it tile for tile. Adam7 is decoded rather than refused. Each pass is filtered independently against its own neighbours, so it cannot be read as one strided image, and an empty pass contributes no bytes at all -- counting one would shift every later pass and turn the rest of the picture into noise. 968 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrundhvU51mvU7QHTwvnfY
Ordering between two pictures: a placement no longer displaces one with a higher z-index, and at equal z the newer wins. A cell holds one placement, so "in front" means "instead of" rather than "over the top of" -- exact for opaque pictures, and losing only the blend where a translucent one overlaps another. Recorded as a test rather than left implicit. A negative z-index means something different in kind: behind the TEXT. There the cell keeps both. A picture placed under existing text leaves it readable, and text typed onto a background picture does not erase it -- which needed the one exception to the rule that printing rebuilds a cell from scratch, since that rule is what gives every other image its overwrite semantics for free. The exception is narrow: negative z only, carried tile and all, and the spacer cell of a wide glyph as well, or a CJK character punches a hole through the picture behind it. Erasing still clears a background picture. A picture showing through a cleared screen would be a leak, not a feature. 985 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrundhvU51mvU7QHTwvnfY
Frame transmission (a=f), animation control (a=a) and frame composition (a=c). Frames may carry only the rectangle that changed, composed onto a previous frame or onto a flat colour, blended or replaced. Both driving styles work: a client can make a frame current itself, or set gaps and hand the timing to the terminal. The emulator still owns no timer. It is driven entirely by Write, and starting a thread inside a library that has none -- to repaint a host that already has a render loop -- would be the wrong place for it. So the host calls AdvanceAnimations with however long its last frame took and is told whether anything moved. That also makes the timing exactly testable: no sleeping, no tolerance windows, no flake. Advancing loops rather than stepping once. Several gaps can fall inside one slice when the gaps are short or a repaint was late, and stepping once per call would silently run an animation at the host's frame rate instead of its own. An image's own pixels never change. They are documented immutable and a host may already have uploaded them, so the root frame starts as a reference to them and is copied away the moment a client edits it. What moves is CurrentPixels, with FrameSerial changing alongside so a cached texture can be spotted as stale without comparing pixels. Animated images are tracked in a weakly-held list of their own. The host asks whether anything is moving on every frame, so for a terminal showing text the answer has to cost nothing -- scanning both buffers and the registry is the length of the scrollback, sixty times a second. Weak, or the list would keep every animation's pixels alive for the life of the terminal. Two bugs the tests caught rather than review: v unspecified was read as "no loops" and stopped every animation after one pass, where the protocol makes it infinite; and Y carries a 32-bit RGBA background on a frame, which does not fit the signed int the same key uses for a pixel offset -- opaque red is 4278190335 and saturated into a different colour entirely. 1016 tests pass. Every guard was checked by breaking the code it guards; three tests that stayed green were rewritten rather than kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrundhvU51mvU7QHTwvnfY
A cell kept one placement, so ordering between two pictures was expressed by which of them the cell kept: the front one won and the other was gone from those cells. That is exact for opaque pictures and was documented as the limit, but it cost two things. A translucent picture could not blend over what it covered, because what it covered was no longer there. And deleting the front picture left the one behind with a hole punched through it -- which bit opaque pictures too, so it was a bug rather than a missing feature. A cell now keeps every picture covering it, ordered by z-index and, at equal z, by age. The frontmost stays inline on the cell exactly as before, plus a Below chain that is null for every cell not actually overlapped, so the common case pays the field and nothing else: no allocation, no indirection, and one reference comparison to find a tile. Only genuinely overlapping cells build a chain, and it is bounded at eight deep -- nothing in the protocol stops a client stacking pictures over one spot forever, and every layer is retained pixels that cannot be seen. The chain is immutable. Cells are structs copied by value all over the buffer -- scrolling, resizing, the render cache -- and a chain that could be edited in place would be shared by every copy of a cell that was ever made. Inserting and removing rebuild the part above the change and share the tail, which is a few nodes on a structure that is rarely two deep. Placements carry a Sequence from a counter that only goes up. Kitty breaks a z-index tie by age and nothing else on a placement records that: reference identity says whether two placements differ, not which came first. It has to be a total order because the buffer stacks per cell while a renderer sorts per line, and if those disagreed then which picture is in front would depend on which cell you looked at. Deleting every placement took the text with it. d=a reached the cells through the helper a resize uses, which blanks them -- right for a picture in front of the text, whose character was only ever the placeholder space it wrote when it landed, and destructive for a background one, whose character is whatever the user typed onto it. Every other delete target already got this right, so the two disagreed. The rule now lives on BufferCell.RemoveImages and all three paths go through it. It was invisible until something was drawn behind text, which is how it survived: every other image cell holds a space, and blanking a space changes nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzGenEw6pqZGGS2cwqb9AS
Eleven steps, each drawing one feature and pausing so the screen can be looked at, emitted as raw escape sequences so it depends on nothing but a terminal. Run it inside the one under test. Three of the steps cover overlapping placements: a translucent panel that the picture shows through, a picture deleted off the top of another to prove the one behind comes back whole, and an animation layered on a still so the per-frame texture re-upload and the layering are exercised together. Photographs rather than flat rectangles. A solid colour hides a shuffled tile, a strip drawn from the wrong row and a picture stretched by a couple of pixels, and it hides a blend entirely, since a tint over one flat colour is just another flat colour. The one place a featureless picture is right is the tint itself, which is built in the script rather than stored. The pictures are PNG and go over the wire as f=100, so the terminal does the decoding and the script does no image work at all -- and being 8-bit colormap they walk the indexed and PLTE branches of the decoder, which the raw RGBA the script builds itself never reach. All four assets are CC0 or public domain from Wikimedia Commons, recorded in tools/README.md. The animation is Muybridge's trotting cat: a locomotion study is a sequence whose whole point is that every frame differs from its neighbours in a way the eye checks without being asked, so a dropped or reordered frame is obvious rather than plausible. Everything that can be is placed with c and r, naming a box in cells, because the terminal may not answer the CSI 16 t cell-size query -- it did not while this was being written -- and then the pixel arithmetic is guesswork. The placeholder step cannot do that: a virtual placement's c and r are parsed and dropped, so its grid has to be computed from the picture and the cell size, which is why it gets an asset sized for the job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzGenEw6pqZGGS2cwqb9AS
There was a problem hiding this comment.
🟡 Changes recommended
TerminalImage.ByteCount can overflow and break image budget accounting, which should be corrected before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR adds full support for the Kitty graphics protocol to XTerm.NET, extending the emulator’s graphics pipeline beyond Sixel by introducing APC parsing, Kitty command handling (transmit/place/delete/query), Unicode placeholder placement, overlap/z-ordering, and animation support, plus documentation and demo tooling.
Changes:
- Add APC (ESC
_) streaming support in the parser so Kitty payloads can be delivered chunk-by-chunk. - Implement Kitty graphics core features: transmissions (RGB/RGBA/PNG + zlib), placements (crop/scale/offset), deletion matrix, Unicode placeholders, z-index overlap stacking, and animation (frames/control/compose).
- Update docs/tests/tools to cover rendering guidance, host integration for animation timing, and an end-to-end PowerShell walkthrough script.
File summaries
| File | Description |
|---|---|
| tools/README.md | Documents the kitty demo script and bundled CC0/public-domain assets. |
| tools/kitty-demo.ps1 | Adds a step-by-step Kitty protocol demo tool emitting raw escape sequences. |
| src/XTerm.NET/XTerm.NET.csproj | Bumps version and package description to include Sixel + Kitty graphics. |
| src/XTerm.NET/Terminal.cs | Wires APC events, adds placement/image drop helpers, and adds animation-driving APIs. |
| src/XTerm.NET/Parser/EscapeSequenceParser.cs | Introduces APC streaming state/events and correct termination/abort behavior. |
| src/XTerm.NET/Options/TerminalOptions.cs | Adds Kitty enable/disable option and registry budget for unplaced transmitted images. |
| src/XTerm.NET/Graphics/TerminalImage.cs | Adds animation support hooks and “current frame” pixels/serial tracking. |
| src/XTerm.NET/Graphics/PngDecoder.cs | Adds internal PNG decoder (incl. palette + Adam7) to support Kitty PNG payloads. |
| src/XTerm.NET/Graphics/PlaceholderDiacritics.cs | Adds Kitty’s fixed combining-mark table for explicit placeholder tile addressing. |
| src/XTerm.NET/Graphics/KittyTransmission.cs | Implements chunk reassembly, base64 decode, optional zlib inflate, and format decode. |
| src/XTerm.NET/Graphics/KittyCommand.cs | Parses Kitty control keys into a structured command model. |
| src/XTerm.NET/Graphics/ImageRegistry.cs | Adds id/number-based storage with eviction for transmitted-but-unplaced images. |
| src/XTerm.NET/Graphics/ImagePlacement.cs | Introduces placement model (crop/scale/offset/z/sequence) and tile layout APIs. |
| src/XTerm.NET/Graphics/ImageAnimation.cs | Adds frame storage, blend/compose helper, and host-driven time advancement. |
| src/XTerm.NET/Graphics/CellImageLayer.cs | Adds immutable per-cell overlap stacking chain + ordering rule. |
| src/XTerm.NET/Events/ParserEvents.cs | Adds APC event args types (hook/put/unhook). |
| src/XTerm.NET/Common/Types.cs | Adds a dedicated parser state for APC payload capture. |
| src/XTerm.NET/Buffer/BufferLine.cs | Updates image-clearing and “has images” detection for placement/layer model. |
| src/XTerm.NET/Buffer/BufferCell.cs | Migrates from per-cell image refs to placements + layered overlap stack operations. |
| src/XTerm.NET/Assembly.cs | Exposes internals to test project for decoder-focused tests. |
| src/XTerm.NET.Tests/Parser/ApcSequenceTests.cs | Adds regression tests for APC parsing/termination/interaction with DCS and text. |
| src/XTerm.NET.Tests/Graphics/KittyZIndexTests.cs | Tests z-index ordering and negative-z “behind text” semantics. |
| src/XTerm.NET.Tests/Graphics/KittyPlaceholderTests.cs | Tests U+10EEEE placeholders, tile diacritics, and id-in-foreground-color behavior. |
| src/XTerm.NET.Tests/Graphics/KittyOverlapTests.cs | Tests overlap stack retention and “reveal” behavior on deletes/typing/erase. |
| src/XTerm.NET.Tests/Graphics/KittyGraphicsTests.cs | End-to-end Kitty protocol tests: transmit/place/query/delete and malformed input survival. |
| src/XTerm.NET.Tests/Graphics/KittyDeleteTests.cs | Tests the full Kitty delete target matrix and id/number/positional semantics. |
| src/XTerm.NET.Tests/Graphics/ImagePlacementTests.cs | Tests placement geometry across natural vs stretched scaling and pixel offsets. |
| src/XTerm.NET.Tests/Esc.cs | Adds test helpers for generating CSI/APC sequences consistently. |
| README.md | Updates public documentation for Kitty support, placement-based rendering, and animation driving. |
| FIXES.md | Adds a detailed implementation note/changelog for Kitty protocol support and follow-on features. |
Review details
- Files reviewed: 33/56 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - **Interlaced PNG is refused rather than decoded.** Adam7 is rare from these tools, and a wrong | ||
| picture is worse than a reported failure. |
There was a problem hiding this comment.
Correct — the document contradicted itself. That bullet was written when the first Kitty commit landed and refusal was the plan; a later commit in this branch decoded Adam7 instead, and the Adam7 note two sections down already described how. The bullet was simply never updated.
Fixed in 12ada0c: it now records that this began as a refusal and says why decoding won, with a pointer to the note that covers the passes.
|
@tomlm thank you sir, I won't let you down! 🫡 |
Rebuilds the protocol work on top of #34, where a picture is a run held by the line rather than tiles scattered through cells. The protocol layer -- transmission, the delete matrix, placeholders, animation -- is unchanged and storage-independent. What changed is everything that touched a cell. PlaceImage emits one LinePlacement per row, each with its own slice of the source, taken from the placement rather than the image because a Kitty placement may be cropped and scaled: row 3 of a stretched box is not row 3 of the picture at its natural size. Overlap stopped needing a mechanism. Two pictures over the same columns are two runs, and covering one has no way to modify it -- so a translucent picture blends over what it covers and deleting the front one reveals the back one whole, both for free. The BufferCell.Below chain this branch used to carry is gone, along with the layer inserts, the removal rules and the character-preservation rule that went with them. What runs DO need is an identity. A placement spanning eight rows is eight structs on eight lines, and a positional delete finds it through one cell of one of them, so LinePlacement.Serial says which placement a run belongs to. That is the terminal's own identity, not Kitty's p=, which is the client's, may be zero and may repeat. ImagePlacement.Sequence went the other way: it existed to break a z-index tie between two cell layers, and age now comes from the order runs were added to the line. Two behaviours changed, both toward the protocol. A Kitty placement is an OVERLAY, so printing over one no longer destroys it -- the character lands, the z-index decides which is drawn on top, and deleting the picture gives the character back. Sixel keeps replace-on-write, which is what PlacementKind is for. And erasing now takes overlays as well as content: printing splits only a Sixel, but a cleared cell is blank and a picture showing through one would be a leak whichever protocol placed it. Two bugs found on the way over. A combining mark could attach to a cell showing a picture, because the guard against it had tested a cell field that no longer exists -- which also swallowed the placeholder diacritics that state a tile. And re-tiling a placeholder read its current tile back from the cell; recomputing it from the origin instead made each mark undo the last, so a row-then-column pair kept only the column. 1034 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzGenEw6pqZGGS2cwqb9AS
ByteCount clamped the animation to int.MaxValue and then added the root picture in int arithmetic, so the total wrapped negative at exactly the size the clamp existed to guard against. Everything that reads it accumulates into a long already -- the registry's running total, the sweep's, and the bytes counted since the last sweep -- so the property is now a long too and the sum happens before anything is narrowed. A negative byte count is worse than a wrong one: the eviction sweep would read an image as free and keep it while dropping others. The overflow itself needs gigabytes of frames to reach, so what the test pins is the arithmetic rather than the wrap -- that the two halves are summed and that the frames reach the budget at all. Also corrects a FIXES.md bullet that still said interlaced PNG was refused. A later commit in this branch decoded Adam7 instead, and the same file already described it two sections down, so the document contradicted itself. Both found by Copilot's review on #37. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzGenEw6pqZGGS2cwqb9AS
Implements the Kitty graphics protocol on top of #34's placements-as-runs: transmission, placement, deletion, Unicode placeholders, z-index, animation, and overlapping placements.
Rebuilt on
mainafter #34 landed. The protocol layer is storage-independent and carried over unchanged; everything that touched a cell was rewritten, and most of it got shorter.What works
Transmission —
a=tanda=T, chunked payloads, RGB, RGBA and PNG (including Adam7 and indexed/PLTE), zlib, and image numbers as well as ids.Placement —
a=pwith cropping, cell-box scaling, theX/Ypixel offsets, cursor policy and quiet levels. OneLinePlacementper row, each carrying its own slice of the source — taken from the placement rather than the image, because row 3 of a stretched box is not row 3 of the picture at its natural size.Deletion — the full
d=matrix: by id, by number, at the cursor, at a cell, by column, by row, by z-index, and with a placement id narrowing any identity form to a single appearance.Unicode placeholders — U+10EEEE with Kitty's 297-mark diacritic table, the way
image.nvimplaces pictures.Animation — frames (
a=f), control (a=a) and composition (a=c), both client-driven and terminal-driven.Overlap — two pictures over the same columns are two runs, ordered by z and, at equal z, by age.
What the run model changed
Overlap stopped needing a mechanism. Covering a picture has no way to modify it, so a translucent picture blends over what it covers and deleting the front one reveals the back one whole — both for free. An earlier version of this branch carried a per-cell layer chain with inserts, removals and a character-preservation rule to get the same two properties; all of it is gone.
Runs needed an identity instead. A placement spanning eight rows is eight structs on eight lines, and a positional delete finds it through one cell of one of them.
LinePlacement.Serialsays which placement a run belongs to. That is the terminal's own identity, not Kitty'sp=, which is the client's, may be zero, and may repeat.Two behaviours moved toward the protocol. A Kitty placement is an overlay: printing over one no longer destroys it, the z-index decides which is drawn on top, and deleting the picture gives the character back. Sixel keeps replace-on-write — that is what
PlacementKindis for. And erasing now takes overlays as well as content: printing splits only a Sixel, but a cleared cell is blank and a picture showing through one would be a leak whichever protocol placed it.Two bugs found on the way over
Decisions worth knowing about
The emulator still owns no timer. It is driven entirely by
Write, and starting a thread inside a library that has none — to repaint a host that already has a render loop — would be the wrong place for it. A host callsAdvanceAnimations(delta)and is told whether anything moved;HasRunningAnimations()says whether a clock is needed at all. It also makes the timing exactly testable: no sleeping, no tolerance windows, no flake.File, temp-file and shared-memory transmission are refused with
ENOTSUPby design. The terminal would be opening a path chosen by the program it hosts, and the host usually holds more privilege than that program does. Clients are told so and fall back to sending the pixels directly.For hosts
Draw
image.CurrentPixelsrather thanimage.Pixels— the latter stays the root frame and never changes, which is what makes it safe to hold and hand to another thread. Cache a texture against the image and re-upload whenimage.FrameSerialchanges. Draw a line's runs back to front, and paint each cell's own background once, from the bottom-most run covering it — a nearer one repainting it erases what is behind instead of blending over it.A run keeps its natural width even when the window is too narrow to show it, so clip the destination and narrow the source by the same proportion. That is what makes narrowing show less of a picture rather than squeeze all of it into fewer cells.
The consumer side exists
Iciclecreek's renderer has been rewritten against this and is working — one blit per run, no coalescing and no tile arithmetic, since the line's run list is the draw list. Verified in the running app as well as its suite:
place,crop,behind,overlapandrevealall render correctly. 294 tests pass there. It cannot be opened as a PR until this ships in a package, since its build falls back to the published XTerm.NET.Testing
1034 tests, all passing.
tools/kitty-demo.ps1walks the protocol feature by feature against a real terminal — eleven steps, raw escape sequences, no dependencies. It uses photographs rather than flat rectangles because a solid colour hides a shuffled tile, a strip drawn from the wrong row, or a picture stretched by a couple of pixels, and hides a blend entirely. All of its assets are CC0 or public domain from Wikimedia Commons.It has already earned its keep twice: the
d=atext-destruction bug in an earlier revision of this branch was found by running it, not by a test, and so was aGet-PngSizebug in the script itself.Known gaps
candr(U=1) are parsed and then dropped, so a placeholder picture always covers its natural number of cells and cannot be scaled.🤖 Generated with Claude Code
https://claude.ai/code/session_01TzGenEw6pqZGGS2cwqb9AS